Galton's 1883 mention of "muscular memory" quoted by OED is not its earliest occurrence. The loose idea had common currency at the time, mostly based on self-introspection and common observations, combined with mixing older mentalistic concepts with freshly prominent "psychophysics" (a prototype of psychophysiology) championed by Fehner since 1860.
Hillard, in a pop-sci article Reading Aloud, Appleton's Journal, 4 (1878) no.3, p. 258, already uses "muscular memory" in connection with one "Dr. Streeter, of Boston, who spent years in experimenting upon his own vocal organs, and has succeeded in formulating ideas which many teachers have dimly perceived, but have failed to embody in words", whose "book is, unfortunately, written in so arrogant and unpolished a style as to revolt many critical readers to whom its theories would otherwise appeal". The book is The Primary Elements of Music (1873), and Streeter bills himself as a progressive teacher of "voice building". Galton's use is of a similar kind. In 1893 William James mentions "facility of musical execution which, M. Arreat suggests, may be the result of good muscular memory" in a review of a book, which, he says, would have been entitled Anecdotes 30-40 years prior.
On the other hand, Beaunis published, in the spirit of Fehner's more rigorous experimental psychology, Recherches sur la memoire des sensations musculaires (1888) in Revue Scientifique (25:257-263, 361-368, 757-764). According to a 1889 review of memory theories by Burnham:
"Beaunis has reported the provisional results of somewhat similar
experiments upon muscular sensations. He found three phases in
the vanishing of muscular memory when tested by the power to
reproduce the extent and direction of movements: 1st, the phase of
conscious memory; 2d, the phase of unconscious memory, where conscious memory has vanished, but it is still possible to reproduce the
given movements; 3d, the phase of total obliviscence. See Rev.
Philos., XXV, pp. 369-574."
The review is interesting in its own right in describing the debates over the new "physical explanation" vs old "mental explanation" of memory. Smith authored a long article On Muscular Memory published in The American Journal of Psychology, 7 (1896), no. 4, pp. 453-490, which got considerable attention. Bentley in a historical sketch from The Memory Image and Its Qualitative Fidelity, p.4 puts it into a general context that contrasts the new experimental approach to the old associationist school of thought on memory, starting from Aristotle and going through Hume, Herbart and Spencer, among others.
Adams in Historical Review and Appraisal of Research on the Learning, Retention, and Transfer of Human Motor Skills characterizes this "early period" as "struggling for direction and, with the exception of the first topic—the form of learning curves", and names Bryan and Harter (1897,1899) as "fathering research on complex
skills", specifically learning to send and receive Morse code. Thorndike is generally credited with having put research in this field on firmer ground from 1898 on.
In Hillard's retelling of Streeter:
"The proper control of the tongue, which enables us to take the form of $\overline{a}$ and hold it as long as we like, can only be acquired by practice before a mirror until the muscles learn their lesson, and respond to our volitions. "Muscles," says Dr. Streeter, "retain, to an almost indefinite extent, habits acquired," and it is upon this muscular memory that we all depend, to an extent of which we are quite unconscious. That there can be false intonation in speaking as well as in singing, I think we are all aware. The sudden fall or rise of a voice in speaking will strike a sensitive ear at once.
The power of musical intonation, we are told, depends on the power of accurate adaptation of the muscular parts concerned in the production of the voice to a state known to be capable of producing the required sound. This state is at first recognized by the effect on the ear, and afterward directly through the "muscular sensibility" of the muscles concerned ("muscular sensibility" is, in other words, muscular consciousness, which is the first step toward muscular memory). "Thus we have two memories stored up - the memory of a definite sound and the memory of a definite state of muscles. After a time the process becomes automatic." That is, we have acquired that muscular memory of which we have just spoken..."