It seems to me that ancients (greeks at least) knew that in their geocentric model Venus and Mercury were closer to them than the sun, and correctly differenciated inner (Mercury and Venus) from outer planets (Mars, Jupiter...).
I'm trying to understand how they arrived at that conclusion, but I'm unable to.
A useful source on this subject is Olaf Pedersen's Early Physics and Astronomy: A Historical Introduction, (first published 1974, CUP reprint 1993).
Much is shrouded in the mists of pre-history, but the question appears to suppose that any model would take it that planets are actual bodies at different distances. The sources that do exist show that this supposition corresponds to what would have been already a somewhat advanced state of knowledge. Consider, for example, Pedersen's mention (p.40) of
"the theory of Empedocles of Agrigentum (c.493-433 BC), for whom the Sun did not exist at all as a material entity. He explained day and night by the assumption that a bright and a dark hemisphere revolve around the Earth, the light from the bright hemisphere being reflected from the Earth back onto the heavens as a strongly illuminated spot which we call the Sun."
And then,
"In the thinking of Anaximander, for example, a fairly complete theory of the solar system had already emerged. His conception of the planets as wheels filled with fire rotating about the Earth is perhaps the first Greek attempt to explain astronomical phenomena by means of a mechanical model."
Pedersen went on to mention (p.60) what may perhaps have been the first Greek model to incorporate the more modern suppositions made in the question:
"Aristotle (who never mentions Pythagoras by name and possibly did not believe in his existence) says that the so-called Pythagoreans assumed the existence of a central fire in the middle of the spherical universe. This 'fire of Hestia' ... the Pythagoreans believed, is surrounded by ten concentric spherical shells or spheres. The inmost but one of these spheres takes the Earth round the central fire in the course of one day. The inmost sphere moves a globe invisible to us, the 'anti-Earth', which also performs a revolution about the centre in the course of one day, always opposite to the Earth. Outside the Earth are the planets in the following order:
Moon, Sun, Venus, Mercury, Mars, Jupiter, Saturn.
Each of these spheres turns from west to east, completing a revolution in a period characteristic of the planet. Furthest out is the tenth sphere containing the fixed stars, which also moves, but so slowly that it is imperceptible to the eye. In this form the theory is usually connected with the rather obscure Pythagorean Philolaos, who may have lived at the end of the fifth century b.c. "
Nothing is said about how or why Philolaos and the Pythagoreans arrived at this view of the correct order of the bodies.
What seems possible (and I haven't seen any citations of evidence either for or against it) is that (as already suggested elsewhere) the ancients may have made inferences about relative distances based on visible phenomena such as the apparent speed of motion of the visible celestial objects relative to the fixed pattern of the starry background. In any case, it is clear that these motions of the planets relative to the fixed pattern of the other stars were actually noticed -- because they underlie the very name and idea of 'planet' i.e. 'wanderer'. As others have mentioned, perhaps the inference may have been, that the slower the planet moved, the farther away it was.
Mars, Jupiter and Saturn share a visible characteristic, that at some time in each year or a little longer, each is visible in the south at midnight, moving slowly west (what we now call retrograde) relative to the fixed stars. The motion of these three relative to the stars is always slower than that of the Sun, Saturn being the slowest, then Jupiter and Mars. This order of appearances might plausibly be at the origin of the idea that Saturn is farthest away, &c.
On the other hand Mercury and Venus are never seen as far from the Sun as the others, they are never visible at midnight, and at their greatest apparent speed, unlike Mars, Jupiter and Saturn, they move faster than the Sun (Mercury faster than Venus).
Many ancient theories accordingly differentiated between Mars, Jupiter and Saturn on the one hand, and Mercury and Venus on the other, but they showed a diversity of view about which is closer, Mercury or Venus.
Ptolemy's 'Almagest' is of little help in answering the present question. He treats the answer as either a settled consensus (without explaining how it came to be) or else (in respect of Mercury and Venus), a matter of long uncertainty or controversy. Thus, from Toomer's translation (1984) of Book IX, section 1 "on the order of the spheres of sun, moon and the 5 planets", Ptolemy writes:
"... we see that almost all the foremost astronomers agree that all the spheres are closer to the earth than that of the fixed stars, and farther from the earth than that of the moon, and that those of the three [outer planets] are farther from the earth than those of the other [two] and the sun, Saturn’s being greatest, Jupiter's the next in order towards the earth, and Mars' below that. But concerning the spheres of Venus and Mercury, we see that they are placed below the sun’s by the more ancient astronomers, but by some of their successors these too are placed above [the sun’s], for the reason that the sun has never been obscured by them [Venus and Mercury] either. To us, however, such a criterion seems to have an element of uncertainty ... ."
Of course everything orbits around the earth.
Lumenlearning
Some planets (the inner planets) sometimes pass between the sun and the earth, meaning their orbints are inside the orbit of the sun. Other planets (the outer planets) never pass between the sun and the earth.
This is an immediate conclusion from direct observation of the sky. Mercury and Venus accompany the Sun: are never far away from the Sun. Outer planets behave very differently: they can be at any angular distance from the Sun.
By the way, the choice between geocentric and heliocentric system has nothing to do with the phenomenon: what we see in the sky can be described within any coordinate system with any precision.
In terms of Ptolemy model, the distinction means that the radius of epicycle is smaller than the radius of the deferent for Mercury and Venus, while for the outer planets it is larger. The order of the planets corresponds to their periods of revolutions on their epicycles.
I think the question is historically vague so it could be answered by extending the time frame. The conviction that the Moon is closer than the Sun could be taken for granted, so what remained to be settled is to order the 5 wandering stars. The most simple hypothesis is that planets move at the same speed, so the longer orbit needs a longer period. Identifying Vesper as Phosphor and establishing that its return to the same place takes longer than a year is sophisticated achievement. What was generally known is that it either precedes or follows the sun, so the opinions where to situate it, above or below, seem to have been divided and the same goes for Mercury. The difference with the 3 outer planets that move much slower is emphasized by the fact that they can be observed at any distance from the Sun. (Without looking for a reference I have a dim memory that the Maya listed the planets just as the Greeks)
Inner planets in their wanderings cover only a small region of the Zodiac (or Kalachakra) but the outer planets move all around the Zodiac. They can't be observed all the time but only before Sunrise or after sunset. They move with the Sun plus or minus an angle smallest for Mercury and larger for Venus (called evening star for obvious reason). The Sun itself travels a full orbit per year along the Zodiac.
