The story is long and quite complicated. Here is a very short and necessarily incomplete sketch.
It begins with experimental observations about atomic spectra. The main fact is that each atom has some fixed discrete sequence of frequencies
of light that it emits or absorbs. This was known from the early 19th century (Fraunhofer) and constitutes the basis of spectral analysis. In 1885 Balmer discovered a simple empiric formula for the spectrum of hydrogen atom. Then Planck discovered the discreteness of black body radiation and relation between energy and frequency.
This enabled Einstein to explain photoelectric effect. Both Planck and Einstein were explaining experimental facts
about radiation.
Next important step was made by Bohr, who proposed a model
of hydrogen atom which explained Balmer's formula. This was the basis of the so-called Old quantum theory. It explained many experimental facts but was based on some ad hoc "quantization rules".
Linear operators came to the theory in the work of Heisenberg (matrices) and Schrodinger (differential operators), followed very shortly shortly by the work of Born, Jordan and Dirac.
From their work it became clear that the discrete values of
energy (related to the frequency by Plank's formula) are actually eigenvalues of certain Hermitian operator. This final formulation is due to Dirac, and rigorous mathematical background was developed by von Neumann.
So the first quantized physical quantity were energy and action. And this was a result of complicated theoretical development whose goal was to explain experimental facts, mainly about emission and absorption of radiation.
Chronologically, the first hint that measured values of physical quantities are eigenvalues of certain operators comes from the work of Heisenberg. But Heisenberg did not know the relevant mathematics and had to invent matrices himself. The connection with known mathematical theories was established by Born, Jordan and Dirac. And necessary further development of these mathematical theories is due to von Neumann.
Recommended literature:
For the first period (from Fraunhofer to Bohr): S. Sternberg, History of 19th century
spectroscopy. Appendix F to his book Group theory and physics Cambridge UP, 1994.
For the second period (from Bohr to Dirac) B. L. van der Waerden, Sources of quantum mechanics. It includes English translations of the main original articles with short comments by van der Waerden.
Also A. Sommerfeld, Atombau und Spektrallinien (1921) is a very good source for the transition period between old and modern quantum mechanics. Later editions of this book were expanded to include the work of Heisenberg and Schrodinger.