The notion of Hilbert space comes from Hilbert's theory of integral equations. Of course, it was partially motivated by physics, by the theory of oscillations in classical mechanics, but this theory was developed much earlier than quantum mechanics:
the main work Grundzuge einer allgemeinen Theorie der linearen Intergalgleichungen (Foundations of the general theory of linear integral equations), in 6 parts was published in 1904-1910.
Hilbert's own exposition (besides original papers) is in Courant Hilbert, Methods of mathematical physics. The very title shows that he definitely was motivated by physics, while quantum mechanics was not discovered yet.
There are many anecdotes showing that Hilbert was surprised by applications to
quantum mechanics and by the very term "Hilbert space". Once in a conference, he asked the speaker: "What is a Hilbert space?"
W. Feller was fond of repeating: "I could stare all my life at a symmetric matrix, I will never get Hilbert space out of it!" (Cited from G-C Rota, Indiscrete thoughts p. 223).
There are many other examples when mathematicians develop new tools just in time, shortly before they find new applications in physics (Riemannian geometry and tensor calculus, Hamiltonian formalism (wave-particle duality) etc.) On the other hand, this mathematical research is motivated by already existing physics,
at least partially.