You are on to something here. Section X of Book 1 of Principia discusses the motion of particles constrained to curves or surfaces under the action of central forces (that is forces directed towards a fixed point, the center, with the magnitude depending only on the distance to it). This is an example of constrained dynamics, now associated with "D'Alembert's" principle of virtual work (formulated by John Bernoulli, see Maugin's Glimpse at the Eighteenth Century), and if we take the special case of zero central force, the trajectories will indeed be the geodesics on the surface. Taking the center to infinity while increasing the intensity can produce the constant force field in the limit, Newton considers this simpler case for curves but not surfaces.
Let us translate Proposition 56 into modern terms. "Centripetal force toward a given center" is a central force, "curved surface whose axis passes through that center" is a surface of revolution whose axis passes through the center of force, and "granting the quadrature of curvilinear figures" means that Newton is content to reduce problems to finding areas of curved regions, to quadratures. Thus, the proposition says that the initial value problem for motion on a surface of revolution under the action of a central force reduces to quadrature. The proof relies on the preceding Proposition 55, where Newton proves that if the particles trajectory on the surface is projected on a plane perpendicular to the axis its radius-vector will sweep equal areas in equal times. This is a generalization of the Kepler's second law to arbitrary central forces and motions constrained to surfaces of revolution rather than just planes. In calculus terms, this gives a first integral of motion, which explains the reduction to quadrature.
The geodesic problem is a particular case of Newton's. This is not exactly "solving" it (as was finding ellipses, parabolas and hyperbolas for the inverse square law), but still. Struik gives some early history of geodesics in his
Lectures on Classical Differential Geometry (4-2) without mentioning Newton and Principia, even as he states that the geodesic problem on a surface of revolution reduces to quadrature:
"The history of geodesic lines begins with John Bernoulli's solution of the
problem of the shortest distance between two points on a convex surface (1697-
1698). His answer was that the osculating plane ("the plane passing through
three points `quolibet proxima "') must always be perpendicular to the tangent
plane. For the further history see P. Stackel, Bemerkungen zur Geschichte der
geodatischen Linien, Berichte sachs. Akad. Wiss., Leipzig, 45, 1893, pp. 444-467. The name "geodesic line" in its present meaning is, according to Stackel, due to J. Liouville, Journal de mat hem. 9, 1844, p. 401; the equation of the geodesics was first obtained by Euler in his article De linea brevissima in superficie quacumque duo quaelibet puncta jungente, Comment. Acad. Petropol. 3 (ad annum 1728), 1732; Euler's equation refers to a surface given by F(x, y, z) = 0."
The earlier propositions of Section X concerning motion constrained to curves are also of interest. Propositions 50-53 deal with the motion constrained to a cycloid and cast a new light on its famous relation to pendulum clocks discovered by Huygens . This is discussed in detail in Nauenberg's Huygens and Newton on Curvature and its applications to Dynamics:
"The question concerns the underlying reason for the isochronous property of the oscillations of a body moving on a cycloidal path under the action of the constant force of gravity. They found that the component of the force of gravity tangent to a cycloidal path is proportional to the distance along this path. This result implies that other physical systems where the force is linear with the distance leads to isochronous or harmonic oscillations. The best known example is the restoring force of a spring."
You may find Chandrasekhar's Newton's Principia for the Common Reader of interest. He systematically translates Newton's Euclidese into modern calculus, section by section, and comments on them, but he conspicuously skips Section X. Bressoud's Second Year Calculus: From Celestial Mechanics to Special Relativity deals with the Kepler's second law for gravity both in Newton's and in the calculus notation.
