I heard in wave optics and electromagnetism that Hamilton could have discovered the Schrödinger equation, or that he was the first man who used the expression
$$ \Psi(x)= \exp(i S(x)/\hbar)\,. $$
I also heard that Hamilton got the idea from the eikonal equation
$$ (\nabla S)^{2}=n(x)\,, $$
but that he couldn't complete it.
He was trying to show that light particles could also be described as a wave, but had no way to prove it. Is this what happened?
It is more accurate to say that Hamilton anticipated some of the ideas of mathematics and heuristics of quantum mechanics, that would later inspire Schrödinger to produce his formulation of wave mechanics. The reason he was able to anticipate those ideas is that the quantum wave-particle duality had a classical predecessor, the optico-mechanical analogy. Indeed, it was optics, not mechanics, that originally inspired the Hamiltonian formalism. But no, Hamilton was not about to discover the Schrödinger equation. There is a qualitative difference between the first order Hamilton-Jacobi equation, and the second order Schrödinger equation, the connection is only recovered in the quasi-classical limit. And it is only for the latter that the stationary phase approximation for integrals of $\exp(i S(x)/\hbar)$ is used. The approximation itself was only introduced by Kelvin in 1887, Hamilton's approach was more geometric.
In a simple form the analogy was discovered by Huygens around 1670 (published in Traitė de la Lumiere, 1678), who noticed that propagation of waves could be dually described in terms of wavefronts and rays ("characteristics") perpendicular to them. The latter can be considered as trajectories of particles, and massive amounts of particles spreading along characteristics can create the appearence of a continuous wave. But it also works vise versa, and Huygens suggested that light may well be a wave, with geometric optics of rays being only the first approximation. In particular, Huygens showed how Fermat's least time principle follows from the analogy, Johann Bernoulli used it to solve the famous brachistochrone problem in 1696, and in 1818 Fresnel showed how not only geometric optics but also diffraction and interference can be explained by wave optics, which led to its wide acceptance in the 19th century.
But it was Hamilton who explored the analogy in its full generality. As Guillemin writes in Geometric Asymptotics:
"In 1828 Hamilton published his fundamental paper on geometrical optics, introducing his "characteristics", as a key tool in the study of optical instruments. It wasn't until substantially later that Hamilton realized that his method applied equally well to the study of mechanics. Hamilton's method was developed by Jacobi and has been a cornerstone of theoretical mechanics ever since... It is interesting to note that although Hamilton was aware of the work of Fresnel, he chose to ignore it completely in his fundamental papers on geometrical optics."
The eikonal equation is only the simplest case of the Hamilton-Jacobi equation, when the Hamiltonian of the mechanical system only has the standard kinetic energy term. In general, the familiar equations of Hamilton dynamics are solved by the characteristics of the corresponding Hamilton-Jacobi equation. Although the method of stationary phase is often used to derive geometric asymptotics today, it was not available to Hamilton.
When the conflict between the wave optics and the newly introduced light quanta Schrödinger, a big fan of Hamiltonian dynamics, was reminded of the optico-mechanical analogy, and started thinking about classical mechanics being the limit of a new mechanics along the lines of geometric optics being a limit of wave optics. This led him to his celebrated equation. In the quasi-classical limit the surfaces of equal phase are the "wave fronts", and the trajectories of the particles are the characteristics. A systematic development of this approach is known as the WKB (Wentzel–Kramers–Brillouin) method.
What Hamilton discovered is a mathematical "Hamiltonian formalism". It was applied to those parts of physics which were known at the time of Hamilton: classical mechanics and optics. There was no slightest reasons at the time of Hamilton to suspect that matter on small scale does not obey the laws of classical mechanics.
That this is so, is a late 19s century discovery. However it turned out that Hamiltonian formalism is so general that it applies to quantum mechanics as well. Thus many equations written by Hamilton (and not only by Hamilton, but by other late 18s and early 19s century researchers in mechanics, like Lagrange) actually apply, if they are correctly interpreted.
Similarly, Calculus, a mathematical tool invented in 17s century serves not only mechanics, but all physics discovered later. But nobody claims on this ground that Newton and Leibniz discovered all physics which was discovered later.
EDIT. Of course this is a miracle that the same mathematical tool applies to a very wide class of phenomena in the universe, including those which were not known when the tool was invented/discovered, but this is the way our world is created.
