I'm pretty sure they weren't using tensors in the modern sense at that point, but to what extent did Riemann lay out the structure or significance underlying his eponymous tensor? In his habilitation lecture of 1854, I don't see anything about the notion of parallel transport around a loop. He does talk about the sectional curvature as obtained by extending all geodesics from a given point in a particular "surface-direction" (now thought of as a tangent plane I guess), and as I understand it the Riemann tensor can be completely determined by the sectional curvatures. But then who introduced the modern conception of the Riemann tensor? Christoffel? Ricci? Or was it a gradual process? And is the English translation of the relevant source available?
A short answer: yes, he did.
Riemann's habilitation lecture was aimed at a broad non-mathematical audience, so he did not use formulas in it, trying to explain everything in words. The curvature tensor is explained in Chapter II, section 2. Riemann does not talk here about parallel displacement, he is only interested in the condition that the space is flat (and this condition is vanishing of the curvature).
The mathematical contents of this section is explained (with formulas) in the commentary of H. Weyl included in Riemann's collected works. He credits Christoffel and Lipschitz for the first published account of the analytic apparatus translating Riemann's words into formulas.
Riemann himself did this only partially in his another work, which is known as the "Paris Memoir". The complete title is "Mathematical composition containing an attempt to give an answer proposed by the famous Paris Academy" (Werke, XXII). This work is unfinished, since Riemann was missing the deadline for the competition. He sent his unfinished manuscript, and it was rejected.
This work was neither completed nor published by Riemann, but it is included in his Collected works, with a commentary by H. Weber. It contains the explicit formula for the curvature (in dimension 3), and an indication of a proof that this is a tensor. Parallel displacement was later introduced by Levi-Civita.
Edit. The Paris memoir is about some technical problem of heat conduction, which is reduced to the question, when a quadratic form with non-constant coefficients can be transformed to one with constant coefficients, by a change of coordinates. The condition is that the metric determined by this quadratic form has zero curvature, and this is what Riemann wrote. He did not relate this to geometry. For a detailed analysis of what Riemann probably knew and a plausible recovery of his arguments see
Oliver Darrigol, The mystery of Riemann curvature, Historia Mathematica 42 (2015) 47-83.