A useful source on this subject is Olaf Pedersen's Early Physics and Astronomy: A Historical Introduction, (first published 1974, CUP reprint 1993).
Much is shrouded in the mists of pre-history, but the question appears to suppose that any model would take it that planets are actual bodies at different distances. The sources that do exist show that this supposition corresponds to what would have been already a somewhat advanced state of knowledge. Consider, for example, Pedersen's mention (p.40) of
"the theory of Empedocles of Agrigentum (c.493-433 BC),
for whom the Sun did not exist at all as a material entity.
He explained day and night by the assumption that a bright
and a dark hemisphere revolve around the Earth, the light
from the bright hemisphere being reflected from the Earth
back onto the heavens as a strongly illuminated spot which
we call the Sun."
"In the thinking of Anaximander, for example, a fairly
complete theory of the solar system had already emerged.
His conception of the planets as wheels filled with fire
rotating about the Earth is perhaps the first Greek attempt
to explain astronomical phenomena by means of a mechanical
Pedersen went on to mention (p.60) what may perhaps have been the first Greek model to incorporate the more modern suppositions made in the question:
"Aristotle (who never mentions Pythagoras by name and
possibly did not believe in his existence) says that the
so-called Pythagoreans assumed the existence of a central
fire in the middle of the spherical universe. This 'fire
of Hestia' ... the Pythagoreans believed, is surrounded by
ten concentric spherical shells or spheres. The inmost but
one of these spheres takes the Earth round the central
fire in the course of one day. The inmost sphere moves
a globe invisible to us, the 'anti-Earth', which also
performs a revolution about the centre in the course of
one day, always opposite to the Earth. Outside the Earth
are the planets in the following order:
Moon, Sun, Venus, Mercury, Mars, Jupiter, Saturn.
Each of these spheres turns from west to east, completing
a revolution in a period characteristic of the planet.
Furthest out is the tenth sphere containing the fixed
stars, which also moves, but so slowly that it is
imperceptible to the eye. In this form the theory is
usually connected with the rather obscure Pythagorean
Philolaos, who may have lived at the end of the fifth
century b.c. "
Nothing is said about how or why Philolaos and the Pythagoreans arrived at this view of the correct order of the bodies.
What seems possible (and I haven't seen any citations of evidence either for or against it) is that (as already suggested elsewhere) the ancients may have made inferences about relative distances based on visible phenomena such as the apparent speed of motion of the visible celestial objects relative to the fixed pattern of the starry background. In any case, it is clear that these motions of the planets relative to the fixed pattern of the other stars were actually noticed -- because they underlie the very name and idea of 'planet' i.e. 'wanderer'. As others have mentioned, perhaps the inference may have been, that the slower the planet moved, the farther away it was.
Mars, Jupiter and Saturn share a visible characteristic, that at some time in each year or a little longer, each is visible in the south at midnight, moving slowly west (what we now call retrograde) relative to the fixed stars. The motion of these three relative to the stars is always slower than that of the Sun, Saturn being the slowest, then Jupiter and Mars. This order of appearances might plausibly be at the origin of the idea that Saturn is farthest away, &c.
On the other hand Mercury and Venus are never seen as far from the Sun as the others, they are never visible at midnight, and at their greatest apparent speed, unlike Mars, Jupiter and Saturn, they move faster than the Sun (Mercury faster than Venus).
Many ancient theories accordingly differentiated between Mars, Jupiter and Saturn on the one hand, and Mercury and Venus on the other, but they showed a diversity of view about which is closer, Mercury or Venus.
Ptolemy's 'Almagest' is of little help in answering the present question. He treats the answer as either a settled consensus (without explaining how it came to be) or else (in respect of Mercury and Venus), a matter of long uncertainty or controversy. Thus, from Toomer's translation (1984) of Book IX, section 1 "on the order of the spheres of sun, moon and the 5 planets", Ptolemy writes:
"... we see that almost all the foremost astronomers agree that all
the spheres are closer to the earth than that of the fixed stars, and
farther from the earth than that of the moon, and that those of the
three [outer planets] are farther from the earth than those of the
other [two] and the sun, Saturn’s being greatest, Jupiter's the next
in order towards the earth, and Mars' below that. But concerning the
spheres of Venus and Mercury, we see that they are placed below the
sun’s by the more ancient astronomers, but by some of their successors
these too are placed above [the sun’s], for the reason that the sun
has never been obscured by them [Venus and Mercury] either. To us,
however, such a criterion seems to have an element of uncertainty ... ."