Discovery of Neptune was indeed a significant step which increased the prestige of science but it was only one of a long chain of such events starting in 17th century. Let me give a short sample: discovery of Jupiter satellites with the newly discovered telescope (unlike the discovery of Neptune, this had a great practical application!), Newton's gravitation theory and celestial mechanics,
it predicted the correct shape of the Earth, verified by the French expeditions,
it gave the first working theory of tides, and it permitted Halley to predict the return if the Halley comet. Mathematical theory of Moon's motion solved the centuries old practical problem of finding longitude at sea.
The discovery of spectral lines made it possible to determine chemical composition of stars and the Sun.
This short list only mentions some discoveries in physics and astronomy. But remarkable discoveries were made in other areas as well, let me only mention smallpox vaccination which saved millions of lives.
So by 1846, science had already sufficiently high prestige among educated public. All discoveries that I listed had more influence on people's lives than the discovery of Neptune. By the way, discovery of new planets (large asteroids,
Ceres, Pallas, Juno and Vesta)
before Neptune, also had enormous public resonance, now forgotten.
EDIT. To address the comments, let me explain more about Jupiter satellites for example. The main thing about this discovery was not that they just exist, but that they obey Kepler laws. Kepler laws had to be discovered in advance to determine this. Now the fact that they obey Kepler's laws permitted to predict their motion. And this gave the first practical method of determining longitude, one of the most pressing scientific problems of 17th century. Jupiter satellites made possible determination of longitude only on land, because you need a telescope to see them, and a telescope cannot be installed on a ship.
Solution of longitude problem at sea had to wait for another century, and was
obtained in two ways: a) mathematicians managed to predict Moon's motion, using Newton's laws and advances in mathematics, and b) chronometer was invented.
(A lot of mathematics was used in both cases, especially for the Moon motion).
Now about asteroids. You look at the sky through a telescope, and you see a new star. But you do not know what is this (a Nova, a comet, etc.) When you look again and again, you notice that it moves. So a Nova is excluded. But then you have to compute its orbit, to be sure that when you look in a year from now you see the same object. If the orbit turns to be elliptic with small excentricity, this is a new planet. In the case of Ceres, computation was performed by Gauss,
the greatest mathematician of that time. He had to develop new methods which enriched mathematics. This discovery has an enormous public resonance.
So in no way these milestones in the development of science can be called "stamp collecting".