Euler was most certainly not motivated by matrices and eigenvalues, the chain of causation goes the other way. Sylvester only introduced the term "matrix" to denote an array of numbers in 1850, and did not do much with them. Only Cayley discovered in 1857 that each matrix satisfies equation of its own order, which led to "characteristic values". However, already in 1829 Cauchy "proved that symmetric matrices have real eigenvalues". How did he accomplish such a feat? What he actually proved was that any quadratic form can be reduced to a weighted sum of squares by a linear change of variables. Cauchy even used Lagrange multipliers to find extremal values of quadratic forms subject to normalization constraint, and obtained equation for the "characteristic values" identical to Cayley's for the corresponding matrix. More on history of matrices and eigenvalues in Victor Katz's article in Learn from the Masters.
Now back to Euler. Cauchy considered quadratic forms in any number of variables, but the case of two and three was known long before that because they correspond to conics and quadrics. Euler's Curvature of Surfaces paper appeared in 1760, but 12 years earlier in an appendix to the second volume of the famous Introductio in Analysin Infinitorum Euler showed how to reduce equations of quadrics to the simplest form, and gave explicit but cumbersome formulas for the transformations that accomplish that.
So the quadratic expression in sines and cosines in the denominator of the "osculatory radius" (i.e. the curvature) would have been familiar to him from even simpler case of conics in polar coordinates. He knew that ellipses and hyperbolas have perpendicular principal axes, that their equations look simplest with those as coordinate axes, and how to change the coordinates. This also explains the word "principal", now attached to directions and curvatures. In the paper Euler finds the extrema the modern way, by taking the derivative and setting it equal to zero, and then transforms coordinates by two reflections, rather than a single rotation used today.
We should also add that curvature of curves was first introduced in 1659 by Huygens via osculating circles to construct evolutes for his ideal pendulum, and was published in his famous Horologium Oscillatorum of 1673 that Euler was certainly familiar with. Jerry Lodder's Curvature in the Calculus Curriculum gives a step by step breakdown of both Huygens's and Euler's curvature computations along with insightful commentary.
What Euler suspected before starting the computation calls for speculation of course, so I will entertain some. If we take a generic point and zoom in on its vicinity it will approximately look like a paraboloid, elliptic or hyperbolic, standing on its vertex. In modern notation we get its equation by keeping terms up to order two in the Taylor expansion of $z=f(x,y)$. Cross-sections parallel to the surface will then be approximately ellipses or hyperbolas, and mentally fitting osculating circles perpendicular to the surface at the vertex suggests, with elliptic one at least, that they have the largest and the smallest radii along their principal axes. That Euler had this in mind is supported by the fact that after considering curves obtained by intersecting a surface with an arbitrary plane in general coordinates, he switches to normal planes and chooses $x,y$ in the tangent plane to the surface. Of course, this intuition was made fully explicit only much later in Dupin indicatrix, but considering Euler's facility with infinitesimals and quadrics it is not too far fetched that he had some idea of what to expect from geometry.