What are the definitive experiments/phenomena which motivate quantum mechanics?

Question

The double slit experiment is usually given as the foremost example of a physical experiment that requires quantum mechanics to satisfactorily explain. However, every account i've seen of it (such as in, eg, Feynman's famed QED book) can be perfectly justified using classical wave theory.

So what actual example of physical phenomena can be given to undergraduate students that conclusively demonstrates the insufficiency of classical theory and thus motivates the introduction of quantum mechanics?

Answer 1

I will describe the most important experimental data which led to creation of quantum mechanics, in the chronologic order of their explanations, not the order of experiments.

The idea of quanta was initially motivated by the theory of black body radiation. Plank derived his formula generalizing other formulas which came from experiments. To explain his empiric formula he had to invent the energy quanta (1900).

Another phenomenon which was unexplained by classical physics was photoelectric effect. (This is probably the simplest experiment to explain to beginning students. Understanding this work of Einstein requires almost no background in physics or mathematics, unlike other work mentioned here). Einstein explained what we observe in photoelectric effect by using Plank's idea of quanta, and extending it to electromagnetic radiation (1905).

Another experimental data which defied classical explanation were spectra of atoms. Bohr used the idea of quanta and explained the Balmer lines of hydrogen. Balmer discovered his empiric formula for the spectral lines in 1885 and Bohr "explained" it in 1913.

These are three kinds of experimental data from which quantum mechanic was born, historically. All three named physicists (Planck, Einstein and Bohr) were eventually awarded Nobel prizes for these discoveries, but it took some time, until 1930s before quantum mechanics obtained its modern form (Heisenberg, Born, Jordan, Dirac, Schrodinger and von Neumann).

Slit experiments played a role later than the three pieces of experimental data mentioned above. There was nothing unusual for physicists, since 19th century in the slit experiments with light. But discovery of electron diffraction in 1924 confirmed quantum mechanics.

Even more important was the Stern-Gerlach experiment (1922) which led to the discovery of spin. This eventually led to the explanation of the most important experimental fact of all mentioned: the Periodic Table.

Added on 3.4.2021. An outstanding exposition of the history of spectral lines is the article by S. Sternberg, A history of 19th century spectroscopy (one of the very best articles on history of science that I know). It is published as Appendix F to his book "Group theory and physics".

On Stern-Gerlach (and Einstein-Podolsky-Rosen) there is a nice book by Jim Baggott, "The meaning of quantum theory".

Unfortunately I don't know any good exposition of Planck's discovery.

Answer 2

There's no easy answer of course, things moved quite quickly at the turn of the century. Below, I'll give an overview of what was going on and what experents were being performed, but I'll have to defer to more substantial sources for any details.

Experimental evidence begins, I would argue, with Hertz and Wallachs in the 1890s. (Experiments on black body radiation would of course be a more sound foundation, but the results of those experiments were less startling, although they may have been, if investigated more carefully.) Hertz discovered an effect of the ultraviolet light from a spark upon the intensity of a second spark near by, which his lab assistant (and successor of sorts) Wallachs further investigated. This phenomenon was originally called the Wallachs effect, but today we recognize it as early evidence for the photoelectric effect. Lenard would push the experimental evidence supporting this phenomenon further, and from there came the behavior which Einstein was attempting to explain in his 1905 paper.

Spectroscopy, the investigation of the spectral lines of elements, was one of the most exciting areas of research. By taking a sample of e.g. hydrogen, excited to emitting visible light, we have our source; the light from the source is then targeted at a diffraction grating, and the resulting spectrum is viewed on a color scale of sorts. Such experiments were popular, and they were the source of much theoretical work. Particularly notable is the development of the Balmer series for the locations of spectral lines of hydrogen, but other phenomena like spectral line splitting (Zeeman effect) would be highly influential as well.

Experiments on cathode rays were the basis of many important discoveries, but most significant is J. J. Thompson's experiment demonstrating the fact that the "rays" in fact contained electric charge, and thus were the flow of matter (electrons). The study of x-rays also developed from cathode ray tube experiments, and combined with the newly discovered phenomena of radioactivity, this opened the door to far more sophosticated models of atomic structure. Such models are essential to the development of quantum theory.

At around the same time (late 1800s to 1900-1901) experimental data regarding black body radiation revealed limits to the models of the period, particularly Wein's law, and in response to this, Planck developed his famous theory of quanta. This is an instance where rash assumptions had to be made to obtain a suitable model of reality, but Planck made no claims of the physical 'truth' of his quanta, he only demonstrated a derivation which was far more successful than existing theories, with semi-heuristic reasoning. Even still, Einstein's 1905 paper on the photoelectric effect drew a lot from Planck's ideas.

Experiments targeting the structure of matter allowed Bohr (among many others) to formulate his theories of the atom. Chief among them was Rutherford's gold foil experiment, an experiment made possible by the studies of radioactivity performed in the decade leading up. Spectroscopic experiments were also most influential for Bohr and his contemporaries. But early efforts to explain the behavior of the atom suffered from paradoxes. For example, the laws of electromagnetism forbade a charged particle to revolve in orbit around a nucleus without constantly radiating energy. The so-called 'old quantum theory' was built by embracing quantization in energy to explain this fact, working both from Planck's theory, and from experimental and theoretical work in spectroscopy.

As the 1900s turned into the 1910s, very careful experiments on the photoelectric effect were verifying key aspects of the corpuscule-wave debate. Applications of classical mechanics, modified to allow for quantization, revealed some inadequacies in the working theories of Bohr et al. Efforts to combine Einstein's (highly successful but still developing) theory of the interaction between radiation and matter, with the known wave-like phenomena associated with light (particularly dispersion) inspired more abstract work, including the famous BKS theory, as well as important theoretical work from Kramers, Heisenberg, Born, Jordan, Dirac, and Schrodinger. But I digress, we've departed from the experiments.

Many further experiments shaped the concepts of quantum mechanics. Some eliminated classical ideas, like Michelson-Morely; others demonstrated examples of quantization beyond energy, like the Stern-Gerlach experiment. A key piece of supporting evidence for Einstein's theory of photons was Compton's work on x-ray diffraction. Discoveries of protons, muons and pions, and eventually neutrons, began the development of the standard model of particle physics. There are really endless paths to follow, I'm sure I'm forgetting things, but this provides a rough chronology.

References and Suggested Reading

Jammer, The Conceptual Development of Quantum Mechanics

Waerden, Sources of Quantum Mechanics

Segre, From X-Rays to Quarks