Fowler's Ratio in Early Greek Mathematics is a standard reference on the subject, see also his book Mathematics Of Plato's Academy (both are freely available).
Book V, Definition 3 of Euclid's Elements reads: "A ratio (logos) is a sort of relation in respect of size between two magnitudes of the same kind". Thus, ratios of magnitudes were indeed non-numbers, but they could be related to measurements. In the simplest form, originally used by Pythagoreans, one could talk about a ratio of magnitudes $a:b$ being equal to a ratio of numbers $m:n$ if there was a common measure $h$, which reproduced $m$ times equals $a$, and reproduced $n$ times equals $b$ (this makes particularly simple sense for line segments). The common measure could be found through the anthyphairesis, the Euclidean algorithm in geometric form, which produces the sequence of partial quotients of the continuous fraction for $m/n$. Of course, Greeks did not talk of fractions or anything like "rational numbers". Here is Fowler:
"Operations corresponding to the multiplication of ratios do occur within the Elements, with treatments that are worthy of note. First observe that although the idea of taking the product of two ratios is never defined — nor could it be since ratio itself is not defined, but only the equality of two ratios — the operation is used in VI, 23... There could not be a general proposition in Book V on the compound of two proportions between magnitudes, since the result might seem to involve the product of the magnitudes, which would not in general be defined. Particular cases do however occur... At no point in the Elements, or in the surviving corpus of classical Greek mathematics does anything remotely related to a formal treatment of addition of ratios of numbers occur."
The situation was different in practical use, and later in Hellenistic astronomy, where fractions and even sexagesimals, were arithmetically manipulated. Fowler believes that experimentation with the anthyphairesis might have led to the discovery that it does not terminate for some pairs of segments (like the sides and diagonals in squares and pentagons), and hence to the discovery that some magnitudes are incommensurable. This led to the first, "constructive", theory of incommensurable ratios developed by Theaetetus based on non-terminating anthyphairesis sequences. It was later replaced by the technically more convenient device of Eudoxus, presented in Book V of Elements, which defines when two potentially incommensurable ratios are equal by cleverly abstracting and modifying the original Pythagorean definition. Some results of Theaetetus's theory, although established by different methods, are presented in Book X of Elements, where Euclid surprisingly does not use the general Eudoxian theory of Book V.
The Eudoxian definition was the basis for the double reductio arguments later termed the method of exhaustion, which go back to Eudoxus himself (volume of the pyramid), and were extensively used by Archimedes. In modern translation, it declares two ratios $a:b$ and $c:d$ to be equal if for any pair of numbers $m,n$ (meaning positive integers, of course) the equality/inequality between $na$ and $mb$ always goes in the same direction as the one between $nc$ and $md$. Archimedes's results also provide nice examples of how the theory of ratios was used to talk about sizes. Instead of expressing areas and volumes as numbers, the way we do today, he states in On Sphere and Cylinder that the volume of a sphere is to the volume of the circumscribed cylinder as $2:3$, see details in Who calculated for the first time the volume (and surface area) of the sphere exactly?
How much the attitudes have changed by the end of 17th century is attested to by Newton's Arithmetica Universalis (1707), where he writes:
"We understand a number not as a set of units, but as the abstract ratio of
one magnitude to another magnitude of the same kind taken for that unit."
