In a way everyone knew that it was heat that is flowing and coldness is absence of heat. But how did they know it?
The answer, quite simply, is that they didn't know it. Coldness was frequently measured in degrees just as heat was, and terms like degrees of frost were in common use even into the early 20th century. Alternative temperature scales like the Delisle scale and the original Celsius scale that increased with degrees of cold were widely recognized historically.
In the book Inventing Temperature, Hasok Chang includes an in-depth discussion of historical theories of cold, which the following summary is derived from. Chang begins (p. 160):
Practical thermometry achieved a good deal of reliability and
precision before people could say with any confidence what it was that
thermometers measured. A curious fact in the history of meteorology
gives us a glimpse into that situation. The common attribution of the
centigrade thermometer to the Swedish astronomer Anders Celsius
(1701-1744) is correct enough, but his scale had the boiling point of
water as 0° and the freezing point as 100°. In fact, Celsius
was not alone in adopting such an "upside-down" thermometric scale.
[...] These "upside-down" scales were in serious scientific use up to
the middle of the eighteenth century.
Chang goes on to give plenty of examples of such scales of "coldness," tracing the history of the concept over time (p. 162).
There have been a number of perfectly capable philosophers and
scientists through the ages who regarded cold as real as
heat--starting with Aristotle, who took cold and hot as opposite
qualities on an equal footing, as two of the four fundamental
qualities in the terrestrial world. [...] Although many of [the
mechanical philosophers of the seventeenth century] subscribed to
theories that understood heat as motion and cold as the lack of it,
the mechanical philosophy did not rule out giving equal ontological
status to heat and cold.
Francis Bacon viewed cold as a type of contractive motion, the opposite of the expansive motion of heat. Robert Boyle tried to rule out the reality of positive cold, but admitted he couldn't do it conclusively. Pierre Gassendi postulated "frigorific atoms" that were the equivalent of the "calorific atoms" that supposedly caused heating.
Thomas Thomson (1773-1852), an early historian of chemistry, claimed that the general opinion of philosophers (i.e., scientists) in the early eighteenth century was that cold was a "positive something, of a peculiar body endowed with specific qualities. [...] According to [these philosophers], cold is a substance of a saline nature, very much resembling nitre, constantly floating in the air, and wafted about by the wind in very minute corpuscles, to which they gave the name of frigorific particles."
Even in the late 1700s, this question had not been settled, as the Encyclopaedia Britannica reported in 1778 that there was no consensus on this question, and came down on the side of the existence of cold as a separate force/entity of some sort. Experiments in the late 1700s and early 1800s seemed to confirm the existence of independent cold of some sort. It would be a bit involved to try to describe these here, but suffice it to say that there was considerable evidence marshaled by some scientists even into the 1800s for the concept of positive cold. One of the last prominent theories of this type was promoted by Count Rumford (1753-1814), who argued for a type of "frigorific radiation" that was a kind of analogue to radiant heat.
In the end, Chang argues that positive cold didn't go away because it was conclusively debunked at the time, but rather because the caloric theory of heat became so systematized and clearly articulated that it made it difficult to retain a place for positive cold within it.
As for the reason why "upside-down" thermometers gradually fell into disuse even before this time, there's no clear rationale given in historical treatises. But one might theorize that the fact that thermometers had scales with a rising expanding liquid might naturally tend toward having a scale marked to indicate the increasing volume by increasing numbers. Again, note that this trend began likely as a matter of computational and theoretical convenience before any consensus had been reached on whether or not cold existed as an independent entity. And, as pointed out in a comment, it is much easier to practically measure the effects of added heat, but much more difficult to create "added cold." Scientists certainly tried to do the latter too, but it's likely the bias of experimentation involving heat sources led to the caloric theory focusing on heat rather than cold, which ultimately led to the demise of the notion of independent coldness.