History does not often develop in the order of textbook expositions. Today the exponential map is introduced early in both Riemannian geometry and Lie group theory, but many results it is used to derive were originally derived without it. There is no "exponential map" in Gauss's General Investigations of Curved Surfaces (1825,27) or Riemann's On the Hypotheses which lie at the Bases of Geometry (1854), even though they introduce special coordinates (Gauss for surfaces, Riemann generally) which can be interpreted as combining the inverse of the exponential map with a choice of orthonormal frame in the tangent space. Those are now called geodesic polar coordinates, although Bonnet only used "geodesic curvature" in 1848, and "geodesic" as a curve does not appear until Stäckel in 1893, see Struik's Lectures on Classical Differential Geometry.
Nor does one find "exponential map" or "matrix exponential" (or matrix groups for that matter) in Lie's various papers and monographs of 1880-90s on continuous groups. He was talking about groups of transformations acting on domains of
$\mathbb{R}^n$, and his "infinitesimal transformations" generating them were vector fields. He does write expressions like $f+X(f)+\frac1{2!}X^2(f)+\cdots$, but without symbology or much fuss about convergence. Schur added the last bit around 1890, when he gave a more rigorous proof of Lie's theorem on reconstructing Lie group from its Lie algebra, along with implicit version of the Campbell-Hausdorff formula for $Z$ in $e^Xe^Y=e^Z$. But only Campbell in two short notes of 1897 on that formula invokes exponentials directly:"If x and y are operators which obey the ordinary laws of algebra, we know that
$e^ye^x = e^{y+x}$. I propose to investigate the corresponding theorem when the operators obey the distributive and associative laws, but not the commutative". Poincare arrived at the same ideas independently, and developed the exponential formalism more fully in On Continuous Groups (1899), where he also introduced what is now called "universal enveloping algebra" (rigorous proof of the formula by modern standards was only given by Hausdorff in 1906). See Schmid's lecture Poincaré and Lie groups, which cites the original sources.
As for the Riemannian exponential map it took considerably longer to appear, and it was motivated by the Lie group examples rather than by some formal property (there isn't any straightforward one). General Riemannian geometry remained an obscure subject in 19th century with only a small group of Italian geometers like Bianchi, Ricci and Levi-Civita working on absolute (now tensor) calculus. The fad of the day were Kleinian geometries (quotients of projective groups), and especially space-forms, the ones of constant curvature. Killing realized that to classify the latter one needs to classify algebras of Lie groups first, and in a remarkable tour de force of 1885-1890 he managed to do just that, albeit with wobbly rigor. Cartan's grand arrival on the scene was his thesis of 1894, where he reworked Killing's classification rigorously. This meant that Cartan was well aware of the Killing form, the bi-invariant metric on a Lie group. See Rowe's review of Hawkins's Emergence of the Theory of Lie Groups and Petersen's Aspects of Global Riemannian Geometry for further historical details.
Mathematicians rushed into Riemannian geometry only after Einstein introduced general relativity in 1915. He was lucky that his friend Grossman was even aware of absolute calculus, and even luckier that Levi-Civita took it upon himself to tutor him on it via correspondence later. Cartan was one of the first to get involved too, in 1922 he proposed a non-zero torsion version of general relativity, which Einstein himself later pursued, see What have we learned from Einstein's unsuccessful dream? And in 1925 he published the first major work on global Riemannian geometry, Geometry of Riemannian Spaces. By then the notion of parallel transport was already worked out by Levi-Civita and others, and in 1923 Cartan himself introduced affine connections, generalizing Weyl's notion (see The origin of the name "connection" in differential geometry). So it would have been obvious (to Cartan anyway) that a vector spread around by group action along a geodesic would be parallel transported. The coincidence of the two exponential maps then follows directly from the definitions. So we should see both exponential maps so named in Cartan's monograph, right?
But we do not. The Riemannian exponential map along with much other modern lingo first appears in the work of his student, Ehresmann, in 1935-1960. Hermann in his notes to the English translation of the second edition of Cartan's book makes interesting remarks on the origin of Ehresmann's innovations:"Motivated partly by Cartan, partly by Lie and Vessiot, and in part by his own mathematical intuition (which was more algebraic and topological than Cartan's), he introduced many geometric superstructures built into this basic "calculus on manifolds"... in his classic work [Les connexions infinitésimales dans un espace fibré différentiable], formalized this Cartan's connection material in a certain form... However, there is a certain difference in methodology between Cartan and these later workers. They have tried to be determinedly index-free..."
