Modern theories reproduce caloric in the effective sense (similar to effective "particles" like phonons) for most phenomena it was used to explain. Mass was not really an issue because the fluid was assumed to be very "subtle" to easily permeate ordinary matter, and therefore to have negligible mass (from modern perspective, heating does add some minuscule mass due to mass-energy equivalence). One weakness emerged in 1820-s with the adoption of wave optics, it became hard to keep interpreting radiant heat as caloric particles streaming through space. But the key weakness was the caloric explanation of heating from friction, namely that friction forces out some fluid from between the atoms. If that were the case the released heat would obey a material conservation law, not grow proportionally to mechanical work as predicted by the mechanical theory.
This motivated Rumford's experiments in 1798. He was relatively successful, but caloric theory had advantages in other areas, and defenders pointed out flaws in Rumford's experiments that he could not compensate for. So in 1820-s Carnot was still inspired by the caloric water wheel analogy in his work on steam engines, and Joules' improved experiments on the mechanical equivalent of heat were still coldly received in 1842. The articulation of the energy conservation law by Mayer in 1841, Thomson paying close attention to Joule's experiments in 1847, and Clausius's mechanical reinterpretation of Carnot's theory in 1850 had to come together to displace the caloric theory in 1850-s. See Fowler's Teaching Heat: the Rise and Fall of the Caloric Theory.
There were two different theories of heat fluid, the earlier one, phlogiston, was introduced by Stahl in 1703, and the later one, caloric, put forth by Lavoisier in 1780-s. The theory was successful at explaining several types of physical and chemical phenomena, namely heat exchange, combustion, latent heat in phase transitions, and decalcination (formation of metals). In decalcination for example, adding phlogiston to "earths" (ores) explained their acquisition of metalic properties. Black's work on latent and specific heats in 1757-1764, and Wilcke's in 1772, was seen as decisive in preferring phlogiston over the mechanical theory. It was hard to explain mechanically where the latent heat went, why capacity for heat was not proportional to density, or how phase transitions took place. Black, on the other hand, naturally explained it all, latent heat was due to phlogiston's ability to combine chemically with matter, and gaseous state was a solution of liquid in phlogiston.
But the addition of mass in combustion (burning) eventually became noticable, and was meticulously measured by Lavoisier. This led to his criticism of the phlogiston theory and its explanation of combustion in 1780s: according to Stahl phlogiston escaped a substance, whereas according to Lavoisier oxygen was added to it. While this part is mentioned in modern expositions the other part is often omitted, Lavoisier's oxygenation theory still required a heat fluid. In the notes on Kirwan's Essay on Phlogiston (1784) he writes:
"...When a metal is heated to a certain temperature, and when its particles are separated from each other to a certain distance by heat, and their attraction to each other is sufficiently diminished, it becomes capable of decomposing vital air, from which it seizes the base, namely oxygen, and sets the other principle, namely the caloric, at liberty... If therefore we attach to the word Inflammability the idea of disengagement of caloric and light, such as takes place in the phenomena of combustion and calcination, we must conclude, that vital air or oxygenous gas is the inflammable body most eminently, since it is principally and almost entirely from this substance that the caloric and light are disengaged".
Moreover, the caloric was needed to retain Black's part of the theory.
Wikipedia's history article is not very good unfortunately, see Chemical Revolution chapter in Friedman's book.