At the beginning,how and using which principles scientists determined the mass and distance from sun of various planets?
For the distances, between planets, see my answer to the next question, Historical knowledge of Distance of Earth from Sun Once you know the distance and the apparent (angular) size you get the radius. Apparent size was known only for the Sun and Moon before telescopes came in use. But with a telescope, you see a planet as a disk, and can measure the angular size of this disk.
Mass is trickier. The size of a planet can be measured with a telescope and knowing the distance. Then you make some guess about the density. This gives some estimate. It is as good as your guess of the density.
For planets having satellites the situation is better. You know the orbit of a satellite, and can compute the force of gravitational attraction. Using Newton's law of attraction, you find the mass. This is about all you could do in the era before the space craft. See
A more subtle method involves the perturbation by a planet of the orbits of other planets. For example when Neptune's existence was calculated (before it was discovered), this calculation came with a rough estimate of its mass. This was long before its satellites were discovered.
(EDIT: See the very interesting reference in the comment of Franz Lemmermeyer, where the results obtained with this method are described.)
The difficult case is Mercury and Venus. The mass of Venus was reliably measured only after a space craft was sent to it. Observing the motion of the spacecraft near the Venus, you can compute the mass using the same Newton's Law.
A remarkable story is related with Sirius. It has a satellite (also a star, but invisible: too small). This star orbits Sirius and perturbs Sirius's orbit. From this perturbation, the mass of the satellite could be found. So in the beginning of 20th century we had a paradoxial situation: we knew the mass of the invisible satellite of Sirius (9 light years away!) but did not know the mass of our closest neighbor (Venus).
In case of the Earth's mass, it was measured by William Cavendish through his experiment with a spring pendulum (like Coulomb did for the determination of the Coulomb's Constant).