I guess it is an experimental law, so what was the experiment?
It was not "the experiment". First, Newton considered "his" laws to be "common knowledge" already "abundantly" confirmed and accepted by experts (he names Galileo, Wallis, Wren, and Huygens). In Axioms, or Laws of Motion section of Principia, where they are laid down, he writes in particular:"Hitherto I have laid down such principles as have been received by mathematicians, and are confirmed by abundance of experiments". Nonetheless, he describes plenty of experiments of his own used to reconfirm the laws.
But second, Newton also understood that it is the theory as a whole that is being confirmed, not each law by itself. He usually needed all of them to derive any one prediction, so no one experiment could prove them one at a time. He writes in the Scholium:"And thus the third Law, so far as it regards percussions and reflexions, is proved by a theory exactly agreeing with experience." Indeed, the third law is first introduced with appeal to common experiences:
"To every action there is always opposed an equal reaction: or the mutual actions of two bodies upon each other are always equal, and directed to contrary parts.
Whatever draws or presses another is as much drawn or pressed by that other. If you press a stone with your finger, the finger is also pressed by the stone. If a horse draws a stone tied to a rope, the horse (if I may so say) will be equally drawn back towards the stone: for the distended rope, by the same endeavour to relax or unbend itself, will draw the horse as much towards the stone, as it does the stone towards the horse, and will obstruct the progress of the one as much as it advances that of the other. If a body impinge upon another, and by its force change the motion of the other, that body also (because of the equality of the mutual pressure) will undergo an equal change, in its own motion, towards the contrary part. The changes made by these actions are equal, not in the velocities but in the motions of bodies; that is to say, if the bodies are not hindered by any other impediments. For, because the motions are equally changed, the changes of the velocities made towards contrary parts are reciprocally proportional to the bodies. This law takes place also in attractions, as will be proved in the next scholium."
The "proofs" in the Scholium refer to more formally staged experiments, which include colliding two pendula of various masses, sizes and densities, and measuring the rebound, bringing a system of iron and loadstone into equilibrium, and measurements on simple machines, like balances, wedges and pulleys. They are not proofs in the literal sense (as in deriving a law from the experiment), but rather confirmations that all of the above are predicted to behave as observed when the third law (along with the first and second) is used to make the predictions:
"Thus trying the thing with pendulums of ten feet, in unequal as well as equal bodies, and making the bodies to concur after a descent through large spaces, as of 8, 12, or 16 feet, I found always, without an error of 3 inches, that when the bodies concurred together directly, equal changes towards the contrary parts were produced in their motions, and, of consequence, that the action and reaction were always equal... I must add, that the experiments we have been describing, by no means depending upon that quality of hardness, do succeed as well in soft as in hard bodies. For if the rule is to be tried in bodies not perfectly hard, we are only to diminish the reflexion in such a certain proportion as the quantity of the elastic force requires.
[...] In attractions, I briefly demonstrate the thing after this manner... I made the experiment on the loadstone and iron. If these, placed apart in proper vessels, are made to float by one another in standing water, neither of them will propel the other; but, by being equally attracted, they will sustain each other's pressure, and rest at last in an equilibrium."
[...] The power and use of machines consist only in this, that by diminishing the velocity we may augment the force, and the contrary... I was only willing to show by those examples the great extent and certainty of the third Law of motion. For if we estimate the action of the agent from its force and velocity conjunctly, and likewise the reaction of the impediment conjunctly from the velocities of its several parts, and from the forces of resistance arising from the attrition, cohesion, weight, and acceleration of those parts, the action and reaction in the use of all sorts of machines will be found always equal to one another. And so far as the action is propagated by the intervening instruments, and at last impressed upon the resisting body, the ultimate determination of the action will be always contrary to the determination of the reaction."
A good question.
Julian Barbour, the British physicist, asks the same question in one of his books and he didn't answer it. The real answer is that we don't know. Newton didn't write out how he came to his laws. But then again he didn't write the Principia in the language of the calculus because he knew just how conservative mathematicians were - he wrote it in the language of the day - Euclids geometry (which kind of suggests just how much mathematics/physics today goes under the radar because mathematicians/physicists can't invest the time to learn new mathematics/physics. Of course one can't blame mathematicians/physicists for all this given how much of higher education has been neo-liberalised into a market logic of publish or perish ... but I digress).
However I think we can reconstruct it once we know that he was cogniscent of the notion of an atom, after all, he based his theory of the corpuscular nature of light on the same as well as supposing atoms were formed from 'nothingness' by the mere force of Gods will.
He's also known to have written quite heavily over his copy of Lucretious On Nature and obviously he would have known about Aristotles notion of force: that something moves if it is capable of moving and if an external force is applied by contact. This is one half of the first law and one-half of the third law.
Now, Aristotle was merely expounding on a general nature of change whereas Newton specifically wanted to consider motion. Suppose one atom collided with another. They are exactly the same. The force on the second must be due to the impact of the first (see Aristotles law above). Then by symmetry, given both atoms are the same, a force must be applied by the second on the first, and again by symmetry, we see that they are the same!
Then by induction we generalise to large macroscopic bodies made up of atoms. We then test this physical hypothesis by seeing what it says about motion. Because they work, we argue a posteriori that the hypothesis is correct and thus the hypothesis is alchemised into a law.
Notice how he used Aristotles law to come up with his own. This is no different to how today mathematcians/physicists work. When Newton said he could only do what he did because "he stood on the shoulders of giants" - he meant it.
Moreover, when he said that it was "I do not know what I may appear to the world, but to myself I seem to have been only like a boy playing on the seashore, and diverting myself in now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me." He also meant that.