Basically, Euclidean geometry was considered the purest math since it carried fewer flaws than the proposed number theory at the time since numbers used "natural numbers" (i.e., integers), which produced glaring flaws. There was no such thing as decimal numbers, but to produce a decimal number, it would just be recognized that quantities could be described as an integer number of parts of another. For example, a "third" of an apple is one-part-of-three of a whole apple, and by multiplying this quantity by three, returns the whole apple.
However, to the horror of early mathematicians, integer numbers as a basis for math produced contradictions. The story of Hippasus comes to mind in which he proved that there are some numbers that can't be expressed as a ratio of two integers, hence the number was "irrational. Legend has it that the Pythagoreans drowned him for exposing this flaw.
https://en.wikipedia.org/wiki/Hippasus
So how do we fix this? We revert to something more fundamental than numbers: magnitudes (e.g., lines, areas, angles, apples, oranges, etc.), whose size is regardless of quantity. What I mean by this is that a magnitude can exist alone and still have size; and when alone, it's absurd to assign a value to it other than that the magnitude exists, and likewise, it's absurd to assign a quantity of it. In contrast, if we were to claim that the magnitude has size "five," then we're admitting that there is five of some other magnitude that defines the first magnitude. Furthermore, for a standalone magnitude, we can't say there are "one" quantity of magnitudes, but rather we only claim that it exists. For this reason, Greek math is said to start counting starting with "two" instead of "one," which is strange to us modern counters, but after all, what's the point of counting if there's only a single magnitude? In other words, counting and numbers only make sense once there's a plurality of magnitudes.
Once we add a second magnitude to the mix, then we can compare (i.e., =, <, >) them to each other, which then provides a structure that allows addition, subtraction, multiplication, and division. Euclid's Elements Book 1 is dedicated to establishing this foundation.
Furthermore, ratios of these magnitudes may be taken, which are valid regardless of number, but a ratio is only understood once we introduce a third magnitude. For example, the ratio A:B can exist, but it's useless by itself; however, introduce the ratio A:C and let A:B = A:C, then we can infer that B = C. Again, all this is done without number! The theory of ratio and proportion is all laid out in Elements Book 5.
It's not until Elements Book 7 that number theory is developed and is not investigated with ratios until Book 10. In the latter, concepts of "commensurable" and "incommensurable" are introduced, which are loosely interpreted as rational and irrational quantities. The idea is that, if we take two arbitrary magnitudes, the question is whether we can multiply each of them respectively in a way that they can create two new equal in size magnitudes. If the equality can be made with integer numbers of each, then the magnitudes are commensurable. If not, then they are incommensurable. For example, for magnitudes A and B, they are "commensurable" if 3A = 5B, which we can see that a ratio of two numbers can be produced: A:B = 5/3 (hence 5/3 is a rational number). However, for magnitudes B and C, it's possible that any number multiplication of each will fail to produce equality. In other words, no value for x or y integers satisfies the equality xB = yC, thus B and C are incommensurable and cannot be arranged to produce a ratio of numbers (i.e., the ratio of A and B is irrational). However, the ratio of B:C still exists and remains useful so long as we avoid equating them through any integer numbers. Therefore, it is much more logically sound to demonstrate relationships through geometry where irrational numbers can be avoided and ratios are preserved.
For suggested reading, I recommend portions of Euclid, particularly books 1, 5, and 6. I also found this article by math historian Grattan-Guinness to be helpful in providing historical context, where he discusses how our modern understanding of geometry was reframed through "algebraic-geometry" which resulted in subtle losses of information. For example, algebraic-geometry will say area is length times width, but euclidean geometry never defines area. Instead, the ratio of the areas of two rectangles are the same as the complex ratio of their lengths and widths, which takes on a broader meaning without the use of numbers.
https://www.tau.ac.il/~corry/teaching/toldot/download/IGG.pdf
For these reasons above, Euclidean geometry was heavily relied upon up until algebra and the continuous number line was polished and widely accepted (it became somewhat solidified with Descartes). Since the magnitudes and their ratios were so powerful over numbers, it comes as no surprise that squaring the circle was a problem attempted to be solved by many; however, it wasn't until 1882 that Ferdinand von Lindemann proved that squaring the circle is impossible with Euclidean methods, primarily because, not only is pi irrational, it's also a transcendental number!
