Alexandre Eremenko's answer is great, but I figure the page could benefit from an explanation of the method in general.
Cavendish's Experiment
The question Cavendish was facing was this: Given that we know that charged bodies exhibit attractive and repulsive forces, what is the likelihood that, like Newton's law of gravitation, the electric force obeys an inverse square law with distance? (Not probabilistic likelihood, I mean just inquisitively). Newton had shown that if the force law is an inverse square law, then a hollow uniformly massive sphere will not produce a net gravitational force at any point within the sphere.
Cavendish took the converse of this to investigate the behavior of the electric force. That is, he assumed if a hollow uniformly charged sphere does not produce a net electric force at any point within the sphere, then the electric force obeys an inverse square law. Note that, as mentioned by Alexandre, this is logically invalid, because a statement does not imply its converse. Cavendish may not have been aware of this, but nonetheless, his experiment was designed to prove that if the electric force obeys a power law with distance, then the power must be $-2$.
The experiment assumes that (as was known at the time) conductors allow electric "fluid" to move freely. Then a hollow conducting sphere, when charged, will have a uniform charge distribution. If we have a conducting sphere on the inside of this shell, but not in contact, then we expect that if the net electric force at any given point is not zero, then the electric "fluid" (charge) will be distributed about the inner sphere unevenly. If you then connect a small wire from the charged shell to the inner sphere, we would expect a net charge flow because of the net force on points in the sphere.
Net charge flow means that, when we remove the charged shell connection and remove the charged shell, the inner sphere will be charged if there was net force, so if the inner sphere is not charged, then there was no net force. The absence of net force is what Cavendish took as evidence of the inverse square law, so after finding that the inner sphere was not charged, he accepted that the electric force was nearly an inverse square law.
By estimating the sensitivity of his electroscope, Cavendish was able to approximate the uncertainty in his null-charge measurement.
Maxwell's Proof
Maxwell, in his Treatise on Electricity and Magnetism (vol. 1), provides a fairly rigorous (but still flawed) proof of the fact that if the power is an inverse power, then it must be $2$, which he credits partly to Laplace. One problem with the proof is that it depends not on simply testing the null-charge result for one sphere radius, but for every radius, to truly be confirmed. Another issue, is that while the electric force is clearly a monotonically decreasing function, it was not necessarily an inverse power law. This comes back to the issue of "a statement does not imply its converse." However, the evidence for an inverse-square electric force was building, and the assumption seemed increasingly valid.
References: Maxwell's Treatise on Electricity and Magnetism, art. 74, (p. 80).