I've just finished reading Quantum: Einstein, Bohr, and the Great Debate About the Nature of Reality by Manjit Kumar, which deals with the history of QM from Plank's quantisation for black body problem to Bell's inequality focusing especially on the debate between Bohr and Einstein as indicated in the title. Studying QM with mathematical background the book provides a perfect way to follow the advancements of QM step by step, though a layman could also understand the book since he explains very briefly the physical concepts aside. I'm looking for such books, with which I can study the history of physics and not the physics of some subject, which I already study in a more rigorous way.
If you're interested in reading more about the history of quantum mechanics, one of the most comprehensive works is Mehra's six-volume epic. Other than that, I can personally recommend reading scientific (rather than "informal") biographies of some of the individually involved scientists. I thoroughly enjoyed Pais' accounts of both Bohr's and Einstein's (which naturally also discusses relativity at length) life, as well as Moore's book on Schrödinger.
For the slightly more modern topic of quantum field theory and QED more specifically, I've found Mehra's biography of Feynman quite good, though stylistically inferior to the above-mentioned books. I'm also currently reading his book on Schwinger, and found it to be quite good so far.
The great Dirac falls somewhere in between these two (as he mastered both!), and the recent book by Farmelo on his life was decent, though a bit short and less in-depth than the other books I mentioned.
All of these go (as most scientific biographies) into considerable detail, both historically and scientifically, and I think it's exactly this type of books that you need to look out for.
You will find history of development of solid state physics in Out of crystal maze. It is a very well researched book.
I'm currently reading "The History of the theories of aether and electricity" by Whittaker. A truly illuminating book. Found it in recommended readings section in Jackson, where he described it as one of the most authoritative sources of the history of electromagnetism and science as a whole. It actually includes some history of nuclear physics, relativity and QM, with many amusing examples of historical accidents that lead to great discoveries. Like Becquerel believing radiation came only after the exposure of his crystals to the Sun, since he couldn't believe something could radiate without energy, and energy should have come from somewhere, right? So it went until one cloudy day, when it turned out he was wrong.