The answer is yes. There was a split. First of all, for the Greek mathematics (and very long after them)
"numbers" were integers. "Rational numbers" were called fractions, and no concept of
real number existed. Therefore, mathematics was essentially split into two independent areas: arithmetic and geometry. Diophantus wrote on arithmetic, ...
If you consider Diophantus "ancient" then the answer is "no". In his "Arithmetic" numbers are not necessarily related to geometry or physics. For Pythagoras, indeed, numbers did not exist without geometry or physics interpretation.