The source of species (in mathematics) seems to be François Viète.
See Logistice speciosa (algebra) in contrast to Logistice numerosa (arithmetic), into his In artem analyticem isagoge (1591), page 19 :
Logistice numerosa est quae per numeros, Speciosa quae per species seu formas exhibitur [Numerical logistic is (a logistic) that employs numbers, symbolic logistic one that employs symbols or signs for things].
See Introduction (page 13) for comments :
Samuel Jeake in 1696, has it that this "name ... with the Latins serveth for the Figure, Form or shape of any thing" and that, accordingly, "Species are Quantities or Magnitudes, denoted by Letters, signifying Numbers, Lines, Lineats, Figures Geometrical, &c." Alexandre Saverein's Dictionnaire Universel de Mathematique et de Physique (Paris, 1753), vol. I, p. 17, says that the expression "algebre specieuse" derives from that fact that quantities are represented by letters which designate "leur forme et leur espece," adding "d'ou vient le mot spécieuse."
Smith thinks Diophantus "the most likely source for Vieta's use of the word 'species' " and that it is, in effect, his substitute for Diophantus' εἶδος : form, image, shape.