No they did not know this.
The motion of the Sun, Moon and planets (as seen from the Earth) was known, in the sense that it could be predicted with reasonable accuracy.
To do this, they used an ingenious model, approximating the periodic motions by combinations of circular motions, the same principle that is used nowadays for predictions of celestial motions.
The sizes of the planets and distances to them were not known, except for the Moon but the order (Mercury, Venus, Mars, Jupiter, Saturn was known, or better to say "correctly guessed"). About the Moon, they new approximate distance, so they could compute the size. They had no means to estimate the distance to the Sun
and planets, thus no means to estimate their sizes. There were attempts to do this but they came with wrong answers by the orders of magnitude. But Ptolemy understood that "The Earth is like a point in comparison to
distances to planets and stars".
They understood that stars and planets must be at an enormous distance since they experience no visible parallax. But they could not imagine how really enormous these distances are. (The absence of parallax was the strongest argument against
the heliocentric system which they proposed and then rejected.)
All this reflect the knowledge at the time of Ptolemy, 2-nd century AD, Egypt, Roman empire. He is considered a Greek because he wrote in Greek, as all other astronomers in the Roman empire. Most of the earlier work is lost, and we know about it only from mentioning in the secondary sources
and Ptolemy himself.
What math did they use? Arithmetic (they did complicated calculations in sexagesimal system), Geometry and Trigonometry.
Trigonometry was actually invented for this very purpose, and the earliest surviving source on trigonometry is also Ptolemy. (Ptolemy mentions Hipparchus, but the work of Hipparchus did not survive. There is also strong indirect evidence that the tables of sines existed before Ptolemy.)
Important remarks. You are asking about "ancient Greeks". I interpret this in a broad sense. Very little survives of Greek astronomy before Ptolemy.
Ptolemy was a "Greek" only in the sense that he wrote in Greek (as all other astronomers of the Roman empire). He was born at the time of Trajan and died at the time of Marcus, and probably was a Roman citizen. He probably spent most of his active life in Alexandria, in the Roman Egypt. Almost all that we know about Greek astronomy comes from his work.
