Much of the early theoretical work in electricity and magnetism appears to have been the result of applying celestial mechanical principles to electrostatics. Examples include Cavendish's inverse-square law experiment (which had an analogous gravitational statement due to Newton), and Poisson's application of potential theory to electrostatics (in fact, generally, Poisson's work in electrostatics involved using analytical tools developed by himself and e.g. Laplace, Legendre, Lagrange, and Biot, mostly in relation to mechanics, to Coulomb's experiments). Not to mention the relationship between magnetism and the Earth, which almost blends the two together. Considering the existence of a large body of work on gravity and celestial mechanics, it seems like the analytical approach to electricity would have borrowed heavily from it. How much does E&M owe to celestial mechanics and the mathematicians who developed it? Were there many essential developments which were due mostly or entirely to celestial mechanics?
General references on the subject are Whittaker's History of the Theories of Aether and Electricity and Timeline Of History Of Electricity.
Newton's persuasive evidence for the inverse square law of gravity was a defining achievent of new science, so of course it invited imitation wherever possible. Newton himself already approaches magnetism with gravity as a template, writing in Principia:
"The power of magnetism... in receding from the magnet, decreases not in the duplicate but almost in the triplicate proportion of the distance, as nearly as I could judge from some rude observations".
In 1750 Michell clarified that this was due to the dipole effect:
"Each pole attracts or repels exactly equally... attraction and repulsion of magnets increases, as the squares of the distances from the respective poles increase".
After Franklin observed that a charge is not attracted to the walls of an electrified can when inside it, Priestley recalled that Newton proved in Principia that a mass inside a spherical shell is not attracted to its walls either. He conjectured in 1766 that the electrostatic force has to fall as the inverse square of the distance for this to happen as well. Cavendish came to the same conclusion in 1771, but "indifferent to fame he neglected to communicate this and other work of importance", as Whittaker put it.
Lagrange's 1773 idea of the source potential came up in the context of gravity, and led Laplace to the now famous equation for it in 1782. Poisson made their potential theory a centerpiece of his electrostatic (Treatise of Mechanics(1811)) and magnetostatic (1824) theories, which indirectly influenced Maxwell, see The Origin of the Displacement Current by Siegel:
[...] "The mathematical paradigm for the treatment of polarized media was Simeon Denis Poisson's treatment of the magnetic case; this analysis had been taken over for the electrical case by Ottaviano Mossotti, whom Maxwell cited explicitly; and standing behind Maxwell's physical interpretation of all of this was Faraday's work on dielectrics, also explicitly cited by Maxwell. The equation that Maxwell wrote for the displacement represented an amalgam of these influences, as well as the requirements of his own specific situation in "Physical lines"."
However, we do not see field treatments of gravity until much later because Laplace demonstrated in Celestial Mechanics (1799) that non-zero latency (forces directed at retarded positions of the gravitating bodies rather than their current ones) would lead to planets flying off of their current orbits almost instantly. Unless, gravity propagation was millions of times faster than light.
If anything, the influence of celestial mechanics on electromagnetism in 19th century was counterproductive. For example, it led Gauss and Weber to constructing alternative electromagnetic theories based on particle interaction laws rather than field concepts, which did not work out, see Hecht's (somewhat cranky) Suppressed Electrodynamics of Ampère-Gauss-Weber. Fluid mechanics and elasticity theory (in the popular after Fresnel elastic theories of ether) supplied more of the field theoretic mathematical machinery.
After Poisson, it is the influence in the other direction, stimulated by Faraday's and Maxwell's field intuitions, that proved to be more fruitful, see What 19th century developments contributed to the General theory of Relativity? For example, Mossotti's 1830 idea that the electric attraction and repulsion do not balance each other exactly, and the difference is gravity, was fateful. In 1900 Lorentz showed that such Maxwellian gravity did not suffer from Laplace's latency problem, because the correction to the force's direction was of the order $v^2/c^2$ rather than of $v/c$, as in Laplace's setting. That was the consequence of the Lorentz invariance compensating for the effects of retardation in the first order, as Poincare pointed out in 1905, and it paved the way to field treatments of gravity, including Einstein's relativistic theory.