It is not quite accurate to say that ancient astronomy made use of epicycloids. Epicycles, yes, but curiously enough, hypocycloids, epicycloids and cycloids were not studied as geometric curves (as far as we know), despite the use of epicycles in astronomy. It would have been possible to draw tangents to them by the same technique that Archimedes used for his spiral, for example, but it was not done until modern times.
Epicycles were viewed as merely computational devices, and their geometry was of little relevance even in geometric constructions used by Ptolemy and his Islamic successors, see e.g. Episodes From the Early History of Astronomy by Aaboe. These curves visualize motion of the planets, but for determining measurable parameters of orbits line and circle constructions were enough. Perhaps, geometers wanted to keep their external applications apart, or perhaps they did not prove as useful internally, but their geometry was not explicitly studied until 16th century.
This said, Greeks were not restricted to straight lines and conics for their curves. Acerbi gives an overview of curve constructions available to them in Homeomeric Lines in Greek Mathematics by Acerbi:
"1) Generative constructions. These normally amount to cutting a surface with a plane.
In this way the conic (Con. I.11–3) and toric sections (iE, 112.4–8) were defined.
2) Pointwise geometrical constructions. A procedure is given to find isolated points
that lie on the curve; the latter is then approximated by joining such points by line
segments or arcs of other known lines...
3) Intersections of surfaces, in their turn obtained by rotation of plane figures (conic,
cylindrical, toric surfaces). This is the case of the curves implicit in Archytas’
method for solving the problem of duplication of the cube (Eutocius, on the
authority of Eudemus, in AOO III:84.12–88.2).
4) “Mechanical” constructions, in which some geometrical objects, for instance
straight lines, are allowed to move, the curve being generated by the motion
of some suitable point on them...
5) Setting forward a property that univocally identifies the curve..."
"Mechanical curves", studied as such, included quadratrix, Archimedean spiral, helix, conchoid and cissoid. Conchoid and cissoid are algebraic curves, the rest are transcendental. Astroid, the evolute of ellipse, and those of parabola and hyperbola, appear implicitly in Apollonius's Conica, in the solution to the problem of the number of normals that can be drawn to a conic from a point, see Apollonius' ellipse and evolute revisited by Hartmann-Jantzen. But he does not study them as curves.
As for acceptance, there was a stigma attached to mechanical (and even intersectional) curves, because they were "impure" under Plato's strictures, and "corrupted the good of geometry", as he put it, see When were the concepts of pure and applied Mathematics introduced? Nonetheless, they were acceptable in applied contexts. Plato himself called on mathematicians to "save the phenomena" in astronomy by resorting to "impure" methods, and Eudoxus, his close associate, cleverly used one of the spiric sections, hippopede, in his models of the retrograde motion of the planets, see Yavetz's A New Role for the Hippopede of Eudoxus and Riddel's Eudoxan Mathematics and the Eudoxan Spheres. The original construction of the hippopede was also mechanical, by composing motions of spheres revolving at angles to each other, and, according to some reconstructions, it was the simplification of replacing spheres with flat circles that led Apollonius to introducing epicycles.
However, when it came to solving mathematical problems proper, mechanical curves were frowned upon. Pappus in his Collection classifies problems into plane (solvable by straightedge and compass), solid (solvable by conics) and the rest, strangely, to us, dubbed "linear" (to Greeks "line" meant what we call "curve") Pappus makes it clear that each type of problem ought to be solved by "appropriate" methods, with mechanical curves being the last resort, when all else fails. As Acerbi writes:
"It was obvious to ancient geometers that any
plane problem could also be solved by solid methods, etc., and quite clear to them that
any single problem falls by its essence into only one of these categories if minimality
of mathematical tools employed is required – even if no proof is offered of this
(cf. Pappus, Coll. III.21 and IV.59; the normative character of Pappus’ prescription
is likely to originate with Apollonius, as we shall see in the companion paper)."
The quadratrix, spiral and helix were used just so, by Dinostratus, Archimedes and Apollonius, respectively, for one such problem, squaring of the circle. Eudoxus, and his teacher Archytas, used the hippopede for duplicating the cube, as Diocles used the cissoid, quadratrix and conchoid were used for the angle trisection. But Menaechmus's solution by intersecting conics was preferred for both.
