I am sure that there are some good and fairly complete histories of the thinking over time of what light is, and specifically the colors. The following site is a simple and quick summary of the highlights: http://photonterrace.net/en/photon/history/
It goes from Aristotle (white and black combinations for the colors), to Newton identifying refraction and how it splits light into its colors (his theory: particles with different refractivity), to Hyguins and Young determining that it is a wave and explaining refraction due to it and wavelengths, to Maxwell proving that it is electromagnetic and color was freq and equivalently wavelength, to Einstein who identified it as photons in the photoelectric effect - the start of anything quantum.
Of course there was also the study and theories of how the eye saw colors, that different colors affected different retina parts -- that took some time, and it actually was Goethe who first theorized that each person may see colors somewhat differently. Since that's less about physics than about biology I won't add more on that.
I saw elsewhere a spectrum from different substances back before Maxwell, with sharp lines. See it in the link at http://library.si.edu/exhibition/color-in-a-new-light/science
It's from the Smithsonian, see the spectrum shown for Spectrum Analysis, Six Lectures, in 1868. Don't know what the thinking was on sharp lines in the spectrum, maybe you can get that from the Smithsonian and read. What they said in 1868 on the spectral lines. Somewhere there it says it allows one to identify different materials. Not too shabby for 1868.
Only after Bohr atom was there some explanation of discrete lines and jumps and orbitals, leading of course to QM.
So, the theories changed over time, and became more scientific really after Newton.