From the rough understanding of the physics of the era I have, I would say there are at least two main reasons as to why it is actually quite expected that finding a relativistic theory of gravity wouldn't be the priority of the majority of physicists.
The holy grail of physics was quantum mechanics since the mid-1900s if not from the beginning of the century itself. Planck proposes the light-quanta based solution to the blackbody radiation problem in 1900, kicking off the quantum era. Thompson proposes the so-called plum-pudding model of the atom in 1904. Einstein comes up with his explanation of the photoelectric effect in 1905, elevating the status of photons from a mathematical trick in Planck's idea to a real physical thing. Milikan shows the quantization of charge in 1907 with the famous oil-drop experiment. Rutherford does the gold-foil experiment showing that the atom has a nucleus in 1911. Finally, Bohr introduces the historic Bohr model of the atom in 1913. As you can see, the world of quantum mechanics is continuously action-packed in the era with which we are concerned.
On the other hand, special relativity wasn't noticed at all in 1905. The acceptance and awareness of special relativity began around 1909 when Planck compared it to be a revolutionary step of the order of the Copernican revolution. A major step in the acceptance of special relativity was the development of the geometric formulation of special relativity by Minkowski in 1907 which also took some time to get acceptance (funnily enough, Einstein himself wasn't quite kind to his old teacher Minkowski about his funny four-dimensional diagrams). While most physicists accepted special relativity by around 1911-1912, it still was a controversial subject, to say the least. Einstein was nominated (along with Lorentz) for the Nobel prize in physics for special relativity but the Nobel committee had decided against it, in part because the old guard of physics still believed in the aether theory.
In addition to it being unlikely that finding a relativistic theory of gravity is the top priority of many physicists, the first physical insight which led to general relativity was put in its furnished form only in 1911 when Einstein laid out the equivalence principle in detail which he had introduced in 1907 earlier. In any case, just like Einstein, all other physicists were completely unaware of the machinery of differential geometry which was needed to mathematically formulate a theory that utilized the equivalence principle. So, it makes sense that it was a mathematician who actually tried to do it, namely, Hilbert. Also, given how much of an esoteric and a purely theoretical subject general relativity was considered even after it was "completed" in 1915 (until the 60s and the 70s when the likes of John Wheeler revived it), one can imagine how esoteric it would have been considered before 1915. People had much more tangible work to do in quantum mechanics.
Despite all these reasons, Hilbert was by no means the only person other than Einstein who worked on it. As is famous, Emmy Noether (another mathematician) worked on it, helping Hilbert. In fact, Poincare tried to create his own relativistic theory of gravity by trying to modify Newton's law of gravity in analogy with the relativistic Maxwell's equations as early as 1905. In around 1912, Abraham and Mile formulated a scalar theory of gravity which was Lorentz invariant but failed to satisfy the equivalence principle. The most impressive effort was by Nordstrom who actually formulated a self-consistent scalar theory of gravity in 1912-1913, consistent with both Lorentz invariance and the equivalence principle, but it violated the requirement of general covariance which Einstein thought was crucial. In the end, as we know, Einstein's general theory of relativity which was a tensor theory was consistent with Lorentz invariance, equivalence principle, as well as general covariance--and most importantly, has matched in its predictions with all the experiments so far.
