Riemann discussed a "unified field theory", including light, electromagnetism and gravity, in Neue Mathematische Principien der Naturphilosophie (New Mathematical Principles of Natural Philosophy, 1853, the title obviously alludes to Newton's), and in Gravitation und Licht (Gravity and Light), the last section of his Fragmente on Naturphilosophie (published posthumously in 1876), see e.g. Collected Papers, translated by Baker, Christenson, and Orde, 2004. Useful summaries are given by Bottazzini-Tazzioli in Naturphilosophie and its role in Riemann's mathematics and Gray in Riemann on Geometry, Physics, and Philosophy.
Most of Riemann's natural philosophy was motivated by early Herbart's interpretation of Kant (1802), including the idea that the worldspace (Weltraum) is filled with ether which flows through the atoms and then dematerializes (Herbart is also the only person besides Gauss credited by name in Riemann's famous geometry lecture). Some quotes:
"My main work concerns a new conception of the known laws of nature — an expression of them by means of other fundamental concepts — whereby the use of experimental data concerning the interaction between heat, light, magnetism, and electricity would make possible an investigation of their interrelationship. I was led to this primarily through the study of the works of Newton, Euler and, on the other side, Herbart.
[...] *Both classes of phenomena may be explained, if we suppose that the whole of infinite space is filled with a uniform substance, and each particle of substance acts only on its immediate neighbourhood. The mathematical law according to which this occurs can be considered as divided into
- the resistance of a particle of substance to alteration in volume;
- the resistance of a physical line element to alteration in length.
Gravitation and electrostatic attraction and repulsion are founded on the first part; propagation of light and heat, and electrodynamic or magnetic attraction and repulsion on the second.*"
Mathematically, according to Bottazzini-Tazzioli, Riemann’s model amounts to supposing that ether is an elastic, homogeneous, isotropic medium, a popular theory at the time, to which Cauchy, Lame and others contributed earlier. It was vaguely discussed already by Newton. Euler expressed even closer ideas in his Letters to German Princess (1760s), but they were largely forgotten. In 1858 Riemann wrote a paper deriving equations for a theory of electrodynamics, but withdrew it, probably due to a mistake in exchanging the order of integration. In his 1861 lectures he tried to relate it to the propagation of light between neighboring particles of ether.
In Gravity and Light Riemann even writes "this substance can therefore be conceived as a physical space whose points move in geometrical space", but he never related gravity to curvature or curved space, and even wrote that "the basis for the metric relations must be sought outside it, in binding forces acting upon it". The idea of gravity as related to the curvature of space itslef was first suggested by Clifford in his 1870 lecture to the Cambridge Philosophical Society On the Space-Theory of Matter (published in 1876), in a very telegraphic style, and in his book The Common Sense of the Exact Sciences (1885). See useful commentary by Galindo and Cervantes-Cota in Clifford's Attempt to Test his Gravitation Hypothesis. Clifford did see himself as following in Riemann's footsteps, but, as Gray writes, there is no evidence that Riemann took that step.