There are several themes in Huygens' unpublished paper De motu corporum ex percussione ("On the motion of bodies out of collisions"), but maybe the most significant is that he frequently investigates a specific or extreme case (where some factor is zero, one, or infinity) first, and then guesses about a general case which might have those at its boundary. Let's call this his "specific-first" mindset and go into his argument.
Prelude: elastic collisions between equal balls
Huygens starts from a basic Galilean perspective where bodies in motion continue in uniform motion in a straight line unless impeded. His problem is, he doesn't necessarily know the best way to phrase what makes a collision elastic. So he starts specific-first: if two balls are identical, then by symmetry, if they both come in towards a center of collision with speed $s$, then in a rigid collision they should both leave with speed $s$.
To generalize this, he adds what we now call today the "freedom of choice of reference frame" or the "principle of relativity," it says that the way physics works on a smooth train/boat ride is the same as it works on the ground, no matter how fast the train/boat is going. He goes specific-first: assume one of the identical balls $Y$ is stationary and the other $X$ comes in from the left at velocity $v_X = u$, then I can use relativity (add $-u/2$ to all velocities) to prove that after the collision $v_X = 0$ and $v_Y=u.$ He then generalizes even more: the proper approach is to consider a boat which sails smoothly from the halfway point between $X$ and $Y$ to the collision point between them: such a boat has speed $v_B = (v_X + v_Y)/2$ and the two velocities are $v_{X,Y} = v_B \pm \delta$, and the collision trades $\pm \leftrightarrow \mp$ so he comes to his conclusion that after any collision of identical balls, they swap speeds (and, of course, the velocities reverse direction).
What about non-identical balls?
So he has handled a very specific case first, but Huygens now wants to generalize his definition of "rigid collision." It's no good to him if that definition only handles identical bodies. But he actually generalizes in a specific-first way, too! He starts with the case of a big unequal ball hitting a small ball at rest, and he says "I won't say exactly what happens, but surely the smaller ball starts moving forward, and the bigger ball moves forward less."
It's important to understand that this is related to what has just been derived. We just saw that if the "bigger" ball were only infinitesimally bigger, it would approximately stop. So by making it larger, it presumably must keep going forward. The limiting case on the other side, that if the smaller ball were infinitesimally small, the bigger ball should continue in uniform motion, also means that the bigger ball never goes faster than it started from this sort of collision.
Huygens uses the principle of relativity to derive the opposite limiting case: If one takes this entire picture $X \gg Y, ~(V_X = u,~V_Y=0)\mapsto (u_-, u_+), u_- < u < u_+$ and shifts it by $-u$ then one finds a picture where $V_X$ starts at $0$ and then becomes some negative number $u_- - u$ while $V_Y$ starts at $-u$ and becomes some positive number $u_+ - u$, ricocheting forwards. Huygens doesn't actually talk about the ricochet all that much, but I think that's because the ricochet was already the best previous prediction, due to Descartes:
Fourth, if the body C were entirely at rest,…and if C were slightly larger than B; the latter could never have the force to move C, no matter how great the speed at which B might approach C. Rather, B would be driven back by C in the opposite direction: because…a body which is at rest puts up more resistance to high speed than to low speed; and this increases in proportion to the differences in the speeds. Consequently, there would always be more force in C to resist than in B to drive, …. (Descartes, as translated in SEP)
Huygens goes on to clearly call out that he is refuting Descartes: if a big object always moves slower when it impacts a small stationary object, these Galileian principles demand that the big object is always moved by any collision from a small impacting object. Huygens is saying "no, the big object emphatically does not stand still!" and that is much more important to him and the physics of his day, than saying that the small object must travel backwards, which was already obvious.
The broader generalization to unequal balls
So Huygens has attacked the problem specific-first and has found these two speeds $u_- < u < u_+$, and he knows at least one limiting case, where the mass ratio is $m_Y/m_X = 1$, where $u_-=0$ while $u_+ = u.$ He presumably already has an idea of the other limiting case, where the "ball" $X$ becomes more of a a "wall" as this mass ratio goes to zero: there are many reasons both experimental and theoretical to imagine that against an immovable wall ($v_X$ goes from $0$ to $0$) a rigid head-on collision should cause a ball to ricochet with the same speed that it impacted ($v_Y$ goes from $-u$ to $u$). And he postulates that the general law is therefore as follows: if you can find a frame of reference (lets call it a "central" frame) where one object enters with velocity $+u$ and then leaves with velocity $-u$, the other object must also see its speed unchanged. This is his original formulation of the conservation of energy, that a central frame for $X$ is, in a rigid collision, also a central frame for $Y$.
Huygens proceeds to come up with an alternative formulation of that condition: the relative velocity of the two bodies are the same before and after the collision. In this central frame this is easy, they go from relative velocity $-|v_X| - |v_Y|$ to relative velocity $|v_X| + |v_Y|$ and those are only different by a sign difference; furthermore any transformation to any other frame preserves relative velocities. He also derives the other limiting case where the mass ratio is zero: if a massive $X$ hits a near-massless $Y$ then to preserve the relative velocity when $v_X$ hardly changes, $v_Y$ must take on the limiting value $2v_X.$
Adding masses into the picture
Up until now, Huygens has been reluctant to place mass directly into his reasoning and that is quite understandable: he wants his arguments to hold independently of Galileo's observations on gravity and how things fall and how scales work. But he insists on connecting his mathematics to Galileo's work eventually and it comes to this argument: that this center-of-mass frame amounts to taking the weight of the objects and multiplying it times their speeds and setting them to be equal. He phrases it as "if two bodies whose speeds are inversely proportional to their magnitudes collide with each other, then each rebounds with the same speed which it had before the collision." So he knows what a "center of gravity" is and he is stating precisely that this central frame for the collision is one where the center of gravity is stationary at the point of collision.
To make the connection, he observes that Galileo showed that falling things fall a distance proportional to time-squared, meaning that their final velocity increases proportional to time. Huygens says that this means that the distance is proportional to velocity squared, and then he says "oh, and I can use this in reverse, too -- because everybody knows that the velocity that something gets as it comes down is exactly what it needs to go back up."
His argument is geometrical so it is a bit clumsy to state in his language, I will rephrase it algebraically. So he says "let's say at time $t=-T$ you release these two masses from rest from different heights, you put them into some sort of curve like Galileo did so that they change directions to horizontal without changing speed. They collide at $t=0$ and bounce back, and they return to different heights. We choose the original velocities according to this balance principle, so let me define the mass ratios $\mu_i = m_i/(m_1 + m_2)$ and we have $v_1 = u/\mu_1$ and $v_2 = -u/\mu_2$ for some number (with dimensions of speed) $u$. Finally, freeze each one when it attains its maximum. Then if they rebound with speeds $s_{1,2}$ they attain heights $s_1^2/(2a)$ and $s_2^2/(2a)$ when we stop them in midair, and their center of gravity thus attains a height proportional to $\mu_1 s_1^2 + \mu_2 s_2^2$ subject to the constraint that $s_1 + s_2 = u\cdot(1/\mu_1 + 1/\mu_2).$
From here what he observes is essentially that one can derive that this maximum center-of-gravity height is in turn minimized by this choice $s_1 = u / \mu_1, s_2 = u/\mu_2:$ any other choice of $s_{1,2}$ leads to a higher center of gravity. Why would this matter? Because in the argument Huygens gives, he states that it is impossible for this collision apparatus to return the center-of-gravity to a greater height than it started. So if we start from the minimum, and cannot return higher, we must return back to the height that we came, and the velocities must be the same out as in. (We would today say that if there are two configurations with no kinetic energy and there is no external energy input, then gravity is a conservative force and thus the height of the center of mass must be strictly lower as the only possibilities are for frictional loss terms, if that helps.)
So the assertion Huygens is making is that, by his day, it is known that when the only external force that is involved is gravity (presumably combined with the forces to fix an object at its maximum height and to change its direction without changing its speed), there is no way for gravity to increase the height of a center of gravity that starts at rest. Since any other allocation of $s_{1,2}$ would increase it, it follows that if we start with the minimum, only the minimizing allocation $s_{1,2} = u/\mu_{1,2}$ works.
Finally deriving conservation of kinetic energy
So Huygens has now identified the "same speeds" frame with the center-of-mass frame, and he gives a rather complicated geometric argument which I think I can replace (like, I'm not even going to try to match his argument) as follows: we first look at a collision in the center-of-mass frame where we have determined that the initial velocities are $v_1 = u/\mu_1, v_2 = -u/\mu_2$, and the final velocities are $V_1 = -u/\mu_1, V_2 = +u/\mu_2.$ We can see lots of things that are invariant in this frame, for example $a_1 v_1^2 + a_2 v_2^2 = a_1 V_1^2 + a_2 V_2^2$ for any $a_{1,2}$ because it's true if $a_1 = 0, a_2\ne 0$ and vice versa. The squaring is important for getting the negative to disappear but otherwise has no deeper purpose here.
But remember that Huygens is interested in the principle of relativity, so he is interested in adding a constant velocity $c$ to all of these numbers. When we do that in the case of these square-sums we'll find $$a_1 (v_1 + c)^2 + a_2 (v_2 + c)^2 = a_1 v_1^2 + a_2 v_2^2 + 2 (a_1 v_1 + a_2 v_2) c + (a_1 + a_2) c^2.$$ Now if we equate this to $a_1 (V_1 + c)^2 + a_2 (V_2 + c)^2$ to find a quantity that's the same before and after, no matter what $c$ is, we find that the leading and trailing terms vanish and the middle term we can divide out by $2c$ leaving just $$a_1 v_1 + a_2 v_2 = a_1 V_1 + a_2 V_2.$$There is a problem here though, the right hand side is the negative of the left hand side. Thus they can only equal if they both equal zero, and $a_1 (u/\mu_1) - a_2 (u/\mu_2) = 0$ leads naturally to choosing $a_{1,2} \propto m_{1,2}.$ So our conclusion is that the sum of $mv^2$ must be the same before and after.
