In the Schiehallion experiment (original paper), Charles Hutton computed the density of the Earth, and from there estimated the densities of the major bodies of the Solar System. His numbers for most bodies were within 30% of currently-accepted values, with the exception of Mercury, which he over-estimated by 70%. Why was he so far off?
I think you may be misinterpreting the WP article. Although I could be wrong, this is too long for a comment, so I'll write it as an answer. It looks to me like Lalande found the relative masses of various bodies in the solar system, but this was before Cavendish, so Lalande had no way to express these masses in terrestrial units such as kilograms. I think Hutton simply found the conversion factor between Lalande's astronomical units and terrestrial units of mass. This was a single number, which he then applied to every body in the solar system. So the poor precision in the case of Mercury is to be blamed on Lalande, not Hutton.
The question is then why Lalande got Mercury's mass wrong by so much. This seems not very surprising. Mercury doesn't have any moons, so its mass can only be estimated by looking for its perturbing effect on the other planets. But because Mercury's mass is small, that perturbation would be quite small and hard to measure.