Who first understood that celestial bodies (stars, planets, and moons) are moving in a 3D space and not points of light moving on Earth's dome? Also, when and how did they realize that celestial bodies have actual masses just like the Earth?
As it almost always happens it is impossible to tell exactly "who was the first". But Hellenistic scientists certainly knew this. The moon has visible parallax, so that they could prove that Moon is much closer than the Sun and planets. According to Ptolemy, Hipparchus knew about Moon's parallax.
They supposed (correctly but without proof) that stars are much more remote than Sun and planets. The standard model in the medieval times was Moon, then inner planets, then Sun, then outer planets and fixed stars. This order is essentially correct, and it is present in the works of Ptolemy (our principal source for Greek astronomy).
But how it evolved in the ancient Greece and Hellenistic states, we do not know because few other astronomical texts survive. Neither the books of astronomers themselves nor the Hellenistic books on history of astronomy.
We know that from the very beginning of the Greek astronomy, they tried to build space models for planet motions (Eudoxus). Nothing like this was present in the earlier, Babylonian astronomy.
Of course, they did not know the exact ratios of distances, and they knew nothing about fixed stars, except their positions on the celestial sphere. So they attached the fixed stars to one large sphere, but they had no way to check any conjectures about this.
The speculations that stars are actually like the Sun, and that they are on different distances started in 17th century but no proof could be given until 1838 when the stellar parallax was successfully measured by Bessel.
As others have said, it's likely we'll never know the absolute first because this was before written history was common enough to take note of every discovery. But the first I know of is the pre-socratic philosopher Anaximander (c. 600 BC). He thought the sun and moon were rings of fire that were enclosed by solid wheels with holes in them. You can see a video model of his cosmos here.
Granted, that's somewhat similar to points of light on the earth's dome. The next candidate would be Anaxagoras (c. 500 BC). He imagined that the sun and stars are fiery stones and that the moon was just an earthy rock that reflected the sun's light. Incidentally, he was tried for impiety for these unorthodox ideas.
If you want something a little more concrete than philosophical debate, Aristarchus of Samos (c. 300 BC) was the first to measure the sizes and distances of the sun and moon by using a little geometry and a poor man's early trigonometry.