Newton proved that if the attraction obeys the inverse square law, then the force inside a uniformly charged square is zero. It follows from the description that you give that Cavendish used the converse statement. In fact this converse statement is true though I doubt that Cavendish had a proof of it in full generality. 

It is very common for physicists (and other non-mathematicians) to confuse a direct theorem with the converse one, necessary conditions with sufficient etc. (Even Newton did this). Of course, one can make various a priori assumptions with which this statement becomes easy: for example that the law of attraction is a power. Then the power must be -2. I suppose this is what Cavendish had in mind.

Ref. for a discussion of the converse of Newton's theorem: MR2125274 
S. Stein,
Remarks on the gravity equation. 
Amer. Math. Monthly 112 (2005), no. 4, 322–333. (He discusses another similar theorem of Newton, but this one can be treated in the same way).