From what I have read, the atomic theory of matter was cemented by a 1905 paper by Einstein in which he explained the erratic motion of a bit of pollen suspended in water using the assumption that water is made of tiny molecules. The question of whether matter was atomic was subsequently laid to rest with the rise of atomic physics.
My question is: what were the most persuasive non-atomic theories of matter? Did they enjoy any scientific advantages over atomism?
All theories of matter, starting with the ancient Greek philosophers, can be classified as either continuous or discrete (i.e., particulate). This dichotomy is due to Aristotle.
Aristotle held that matter was continuous: infinitely divisible. Aristotle believed that a vacuum was impossible (indeed, he claimed to prove this). Since a particulate theory requires a vacuum between the particles, that ruled out atomism for Aristotle and his followers. The Stanford Encyclopedia of Philosophy is a good starting point for Aristotle's philosophy of matter; here, the sections on Hylomorphism and Substance are most relevant. Aristotle's thought is, as usual, complicated: he did allow that for some sorts of substance (e.g., blood, bone), there was a smallest amount of it that would retain the character of the bulk substance.
Atomism goes back to the ancient Greeks, Leucippus and his student Democritus being given credit. These two currents of thought persist, as you say, right up to the 20th C.
Galileo in a lengthy passage in the Dialog on the Two Chief World Systems discusses material vs. abstract shapes, e.g., a mathematical sphere vs. a sphere made of bronze. Do the theorems of geometry hold for material objects? Galileo (through his mouthpiece Salviati) says yes. It is true that the bronze sphere is probably not perfect, but it does correspond perfectly to some mathematical shape. Moreover, there is no reason in principle why we couldn't have a bronze sphere that did match the mathematical sphere perfectly. Clearly these opinions are incompatible with the atomic view of matter.
Theories of bulk matter, like fluid dynamics or solid mechanics (jointly called continuum mechanics), use continuous models of matter. Nowadays these are regarded as convenient fictions. I am not sure of the views of the pioneers here (Hooke, Euler, Ricatti, Young). Galileo's work on how strength depends on size is well-known, and as noted he was not an atomist.
Skipping to the 19th C.: the chapter "The Reality of Molecules" in Pais's Einstein biography Subtle is the Lord... covers the ground succinctly. On the side of atomism, Dalton's law of multiple proportions and Gay-Lussac's law of combining volumes both seem like powerful arguments for atomism. Prout's hypothesis (that all atomic weights are multiples of the atomic weight of hydrogen) also seems firmly atomistic. But Pais writes:
Yet Prout did not consider his hypothesis as a hint for the reality of atoms. "The light in which I have always been accustomed to consider it [the atomic hypothesis] has been ... as a conventional artifice, exceedingly convenient for many purposes but which does not represent nature."
To quote Pais again: "The principle point of debate among chemists was whether atoms were real objects or only mnemonic devices for coding chemical regularities and laws." In other words, does the atomic hypothesis tell us anything new, beyond what we can already deduce directly from the laws of Dalton and Gay-Lussac?
Among the physicists, the argument centered around the kinetic theory of gases. Mach and Ostwald were the most famous opponents of atomism. At an address Ostwald gave in 1895, he attacked atomism with an argument already made by Loschmidt twenty years earlier: at the microscopic level all known laws of physics are time-reversible, yet at the macroscopic level we have entropy and obvious irreversibility.
You ask about scientific advantages of continuum theories over atomism. We should careful not to import modern ideas of scientific evidence wholesale into early periods (a sin historians call presentism or whiggism). What we dismiss as metaphysical or maybe linguistic arguments held great weight throughout most of the history of science. Parmenides's argument against the vacuum -- to speak of a thing, one has to speak of a thing that exists -- was persuasive to many. While Aristotle rejected Parmenides's argument, he had his own philosophical "proofs" of the impossibility of a void, based on his theories of motion.
A metaphysical argument more palatable to modern ears is Occam's razor: "entities should not be multiplied without necessity". In other words, don't hypothesize the existence of something unless there are testable consequences. This is the argument used in relativity theory against absolute space, and in quantum mechanics against classical trajectories for particles. Mach and many chemists felt that the atomic hypothesis provided nothing beyond the regularities (like the laws of Dalton and Gay-Lussac) used as arguments for it.
Kuhn pointed out that highly articulated, unified theories always have an advantage over less developed theories with multiple variants. In the 19th C. the "atomists" by no means presented a unified front. Were atoms divisible or not? What exactly was the difference between an atom and a molecule? Chemists spoke of the distinction between chemical and physical molecules, with no consensus on what the difference was or if there even was one. (Nowadays we say no difference.) The work of Maxwell and Boltzmann on statistical mechanics seemed merely to reproduce, with great mathematical difficulty, results obtained easily with classical thermodynamics, a continuous theory.
Finally, many physicists (like Planck for many years) regarded the second law of thermodynamics as absolutely and not merely statistically true. I've already noted Loschmidt's devastating argument.
I will finish with a tongue-in-cheek remark. If string theory holds, does that perhaps mean that the continuum crowd was right all along?
