How did Newton come up with mass terms in his gravitation formula and then what led to a confusion between inertial and gravitational mass?

Question

Newton had realised that the acceleration due to gravitational force had to be inversely dependent on the square of the distance between two bodies. But how did he arrive at the conclusion that the force is also directly proportional to the product of the masses of the two bodies? And then how could have the terms "inertial" and "gravitational" mass and the confusion of them being equal started?

Answer 1

The terminology of inertial and gravitational masses does not appear until Einstein, and had to do with his transitioning from classical mechanics to general relativity, see Development of gravitational theory. In classical mechanics it is more natural to think of the same $m$ entering both $F=ma$ and $F=G\frac{Mm}{r^2}$ than about two separate masses which are then made "equivalent". It was not really a "confusion". Here is how Newton phrases it in Definition I of Principia he writes:

"The quantity of matter is the measure of the same, arising from its density and bulk conjunctly... It is this quantity that I mean hereafter everywhere under the name of body or mass. And the same is known by the weight of each body; for it is proportional to the weight, as I have found by experiments on pendulums, very accurately made, which shall be shewn hereafter."

As to the proportionality of the gravitational force to masses, it manifests (along with the "equivalence") in the acceleration of free fall being the same for all bodies of equal weight. To get a universal $a=G\frac{M}{r^2}$ from the second law the gravitational force has to be proportional to $m$. The universality itself was, of course, established already by Galileo, but Newton confirmed it with his own experiments described in Principia, Book III, Proposition VI, Theorem VI:

"It has been, now for a long time, observed by others, that all sorts of heavy bodies (allowance being made for the inequality of retardation which they suffer from a small power of resistance in the air) descend to earth from equal heights in equal times; and that equality of times we may distinguish to a great accuracy, by the help of pendulums. I tried the thing in gold, silver, lead, glass, sand, common salt, wood, water, and wheat... The boxes hanging by equal threads of 11 feet made a couple of pendulums perfectly equal in weight and figure, and equally receiving the resistance of the air. And, placing the one by the other, I observed them to play together forward and backward, for a long time, with equal vibrations."

Answer 2

I'd like to point out that the proportionality to $m_1m_2$ comes from the universality of Newton's proposed gravitational law. For, suppose two identical unit particles exert equal and opposite forces $f=G/r^2$, and let two masses $m_1$, $m_2$ consist of $N_1$, $N_2$ small particles. Then each small particle in $m_1$ will feel a total force of $N_2f$ by universality, and since there are $N_1$ of them, the total force is of $$F=N_1(N_2f)=\frac{GN_1N_2}{r^2}=\frac{Gm_1m_2}{r^2}$$ since $m=N$ units of mass.

Newton never writes $F=ma$ in Principia; instead he uses proportionalities (see https://plato.stanford.edu/entries/newton-principia/#NewLawMot). So the distinction between inertial and gravitational mass did not arise.

Answer 3

As for the mass terms in the gravitational formula, they manifest after the study of bodies in free fall as said by Conifold.

As for the confusion of the inertial and gravitational mass, there was no reason why they should be the same. The two masses appear in phenomena that seemingly had nothing to do with one another. I apply a force to an object, measure the acceleration, and divide. A principle of dynamics. Why should it have any relationship at all to the nature-provided force that holds the Moon to the Earth?