The movement of planets is not "chaotic". Otherwise prediction would be impossible. The planets move according to Kepler Laws, with very good accuracy.
Strictly speaking the description of the motion as regular or chaotic depends on
the time scale. On the scale of few thousand years, it is very regular,
obeying the Kepler laws. Deviations from this regular motion are small.
At the time of Kepler they were not observed.
In 18-19 centuries these deviations were observed, explained on the basis of Newton's laws, and permitted the discovery of new planets.
But they may accumulate with time. There was a long discussion whether they actually accumulate or not (that is whether the mathematical model of the Solar system based on the Newton laws is stable or not. This discussion is not finished).
If we look at the scale of billions years instead,
it may look much more complicated. But whether it is really chaotic or not
(on the scale of billions years) we do not really know, and the question has little practical meaning, because our solar system lifespan is only few billion years.
The question how Kepler derived his laws based on Brahe's observations was
discussed on this site:
What data did Kepler work out his laws from?
The best source for this is Kepler's Astronomia Nova itself. There is an English translation. In it Kepler described in the great detail, how after VERY many trials he figured the correct law. Unlike ost other authors, he describes the whole process, including all his mistakes, not only the final result.
EDIT. Explaining what and how Kepler did, even very roughly, needs some background in history of astronomy, and a bit more space than is allowed here. So I link a file with a non-technical explanation with essential background:
http://www.math.purdue.edu/~eremenko/dvi/kepler.pdf