Why did Einstein develop General Relativity?

I know that ultimately GR was validated by experiment. I'd like to know however, if there was any experimental discrepancy at the time that pointed to a conclusive flaw in Newtonian gravity, and which led Einstein to say "Hmm...you know what? Newtonian gravity can't be correct. Let me see what I can come up with".

I've read that SR contradicted Newtonian gravity since the latter postulated instantaneous action at a distance. But surely we don't need to throw the whole thing over right? I mean, I can simply modify Newtonian theory to say that the action of gravity travels at the speed of light - introduce a time component, create your equations to include that and everyone's happy.

Now if you say that the time delay approach to Newtonian gravity is contradicted by experiments, would you happen to know which experiments exactly? And were those experiments conducted before Einstein concocted GR?

I know that the Advance of Mercury's perihelion was a problem, but I've not heard that physicists were ready to throw away Newtonian mechanics based on that alone. Certainly I've never read that Einstein was inspired by Mercury to develop GR...

Because I'm guessing that the instinctive and first approach to resolving a contradiction in a long standing and established theory, would be to try and tweak what works. Not just happily throw over the entire table and start from scratch.

Or did Einstein not have a solid reason to develop GR at all? Was he just doing his own thing for kicks, and not looking to actually solve any existing experimental contradiction between Newtonian gravity and experimentation?

"Simply modifying" Newtonian gravity to have it spread at finite speed does not work if the finite speed is the speed of light. It was attempted by Laplace in his Celestial Mechanics (1799), who found that the planets will promptly fly off their orbits and the Solar system will disintegrate in seconds, unless the propagation speed is $7×10^6$ times greater than the the speed of light. This is because of the aberration of the direction of attractive force due to delay in transmission, see Resolving General relativity and Newtonian mechanics on a computer.

A more sophisticated modification follows from Mossotti's electromagnetic gravity hypothesis: electric attraction and repulsion do not balance each other exactly, and the difference is gravity. In 1864-72 Seegers, Scheibner and Tisserand experimented with applying the velocity and acceleration dependent correction to Newton's law imported from Weber's electrodynamics to the precession of the perihelion of Mercury. Around 1900 Lorentz, Einstein's precursor on special relativity, showed that under the Maxwell electrodynamics the Laplacian aberration problem is eliminated because the correction is of the order $v^2/c^2$ rather than $v/c$ that Laplace assumed, so the attraction between masses moving with constant relative velocity is always toward the instantaneous position of the other mass. It is the Lorentz invariance of the Maxwell electrodynamics that cancels the effects of transmission delay to the first order, as Poincare pointed out in 1905. See What 19th century developments contributed to the General theory of Relativity?

However, Lorentz's theory did not work either, and this time exactly because of the perihelion of Mercury. Poincare mentioned in Science and Method (1908) that it gave the advance of 7" for Mercury’s perihelion (the value accepted at the time was 38", the modern value is 43"), but wrote:"This cannot be regarded as an argument in favor of the new dynamics, since we still have to seek another explanation of the greater part of the anomaly connected with Mercury, but still less can it be regarded as an argument against it." Seeliger offered an ad hoc solar corona explanation for the rest in 1906, which some astronomers accepted, but neither Einstein, nor Lorentz and Poincare, took it seriously.

Einstein first mentions Mercury in a letter to Habicht in 1907:"At the moment I am working on a relativistic analysis of the law of gravitation by means of which I hope to explain the still unexplained secular changes in the perihelion of Mercury." Some of Mach's remarks in Science of Mechanics possibly led him to believe that the effect was relativistic. The original versions of his new gravity theory, the so-called "Entwurf" theory Einstein and Grossman developed in 1913, first predicted precession in the wrong direction, and then 18" instead of 45". This was one of the reasons Einstein cited for abandoning it in favor of what we now call general relativity. Alternative relativistic theories of gravity proposed by Mie and Nordström in 1912 did not fare much better with the perihelion, nor did they meet Einstein's more philosophical invariance requirements. See more in What attracted Einstein to the anomalous precession of Mercury?

The correction to the Newton's gravity force is of the order $v^4/c^4$ in general relativity, which in hindsight explains why Newton's gravity worked so well. And in the end it was the perihelion of Mercury, along with philosophical considerations related to the Mach's principle, which Einstein morphed into the general covariance requirement, that took it down. See more in Concquering the Perihelion chapter in Kevin Brown's book Reflections on Relativity.

• Thank you for that well thought out and comprehensive reply with links! – Bhagwad Jal Park Mar 16 '16 at 23:24

Not really an answer, but a few relevant remarks:

Newtonian gravity is inconsistent with special relativity in various ways (e.g., it describes an instantaneous action at a distance and "instantaneous" can only make sense with respect to a given reference frame). There are, also, physical reasons to believe that gravity should produce a redshift on light when it loses energy in leaving a gravitational field. So a number of people tried to formulate relativistic theories of gravitation.

Nordstrøm was the first to come up with theories of gravitation that are compatible with relativity. Nordström's second theory can be reformulated in a way that is very similar to GR (and also satisfies the equivalence principle, background independence, etc.), but it does not have any effect on light.

Riemann had already tried, in his time, to describe gravitation "geometrically" by curving space. So it probably made sense, given Minkowski's reformulation of special relativity by using an indefinite quadratic form, to try to curve this structure.

Einstein developed General Relativity because his initial analysis of motion was only able to produce Special Relativity. That is, in his 'first cut' analysis of relativity of motion, he had to limit himself to translating motion between inertial reference frames. Since not all reference frames are inertial, Relativity was 'incomplete' (hence the term 'Special') and needed to be 'General'-ized.

From that perspective, yes, he was doing this 'just for kicks', except that he was more driven than the word 'kicks' suggests. He was always compelled to find a theory that removed what he saw as inconsistencies of existing theory; that is what led him to develop Special Relativity in the first place.