If I understand correctly, the concept of a Minkowski space/metric was already known before Einstein's paper on special relativity. Was there any physical motivation for studying this type of metric spaces and how did Einstein become aware of these studies?
Not quite. Minkowski had the idea of representing special ralativity as geometry in 1907 under the direct influence of Einstein's 1905 paper, and he developed it in Raum und Zeit (1907) and Zwei Abhand lungen über die Grundgleichungen der Elektrodynamik (1909). See Minkowski on MacTutor. Before that only classical "spacetime" appeared, and only superficially. D'Alembert wrote in the Dimension entry of the Encyclopedia (1756):"One could consider time as a fourth dimension, so that the product of time by volume would, in a certain sense, be the product of four dimensions; this idea is perhaps debatable, but I feel that it has certain merits..." At the end of 19th century the idea became quite fashionable in the popular culture. In 1885 Nature published an unsigned letter Four-Dimensional Space, which read:
"Since this fourth dimension cannot be introduced into space, as commonly understood, we require a new kind of space for its existence, which we may call time-space... We must, therefore, conceive that there is a new three-dimensional space for each successive instant of time; and, by picturing to ourselves the aggregate formed by the successive positions in time-space of a given solid during a given time, we shall get the idea of a four-dimensional solid, which may be called a sur-solid... Let any man picture to himself the aggregate of his own bodily forms from birth to the present time..."
And in 1895 Wells, inspired by a fellow student at the Imperial college, immortalized it in his Time Machine, see When Einstein met H.G. Wells by Halpern.
Before Minkowski, what we call Minkowski geometry in two and three dimensions appeared on a long list of geometries in Klein's classification under his Erlangen program (1872). But it only came up as a collection of objects invariant under all transformations that preserve a quadratic form with indefinite signature. Erlangen Program in general was about interpreting geometries as invariants of various transformation groups acting on projective space, it greatly expanded the number of known homogeneous geometries from the originally discovered hyperbolic and elliptic ones. Unlike them however, it had no spatial interpretation. Before 1910 in his popularizations of the Erlangen program Klein only treated spatial geometries in any detail. Earlier he did not even count the total number of geometries his construction produced, and mentioned higher dimensional generalizations required for spacetime only in single sentences.
There is a connection to split complex numbers introduced by Cockle (1848) and studied by Clifford in 1873 under the name of double numbers. But that was mostly algebraic playing around, and the connection to Minkowski geometry was pointed out only after Minkowski. This is despite the fact that Clifford was deeply involved with non-Euclidean geometries, and his short note On the Space-Theory of Matter (1876) anticipated Einstein's idea of reducing matter to curvature (but of space, not spacetime). Even Poincare, who in 1905 expressed Lorentz's theory of ether (mathematically equivalent to special relativity) in terms of what we call Poincare group of transformations, and was well familiar with the Erlangen program, did not make the geometric connection either. In Science and Hypothesis (1902) he even mentioned "forth geometry" of space, in addition to elliptic, hyperbolic and Euclidean ones, and pointed out that in it lines can be perpendicular to themselves. But according to Weinstein's Poincaré's Dynamics of the Electron, in his famous 1904-06 papers "Poincaré did not associate this quadratic form with propagation of light in order to define a null interval like Einstein or a metric like Minkowski". It took Minkowski to take that extra step of treating time as space.
Yaglom's Felix Klein and Sophus Lie gives many historical details on late 19th century geometry and its relation to relativity with references to the original sources.
Minkowski space time was considered by mathematicians before Einstein and before Minkowski. Of course the name "space-time" was not used. The reasons were purely mathematical, not physical. The most important application was the Klein model of the hyperbolic geometry (non-Eucludean geometry of Bolyai and Lobachevski). More presicely, Klein considered the metric $$ds^2=-dx_0^2+dx_1^2+\ldots+dx_n^2$$ in the $n+1$ space. This metric, restricted to the upper portion of the hyperboloid $$-x_0^2+x_1^2+\ldots+x_n^2=-1$$ is the Klein model of the hyperbolic geometry. In physics when $n=3$ the space with metric $-ds^2$ is the Minkowski space-time. So Klein's model of a hyperblic space is a hypersurface in the Minkowski space time, and this is how it was introduced.
When I wrote that the reasons were pure mathematical, I did not take into account that Lobachevski and Gauss admitted the possibility that hyperbolic geometry holds in the real physical space. Both of them realized that no experiment really tells us that the geometry of our physical space is Euclidean. An experiment can tell us the opposite: you can never be sure that the sum of the angles of a triange exactly equals 180 degrees but an experiment can show that it is less, if it is less indeed). There are indication that Gauss really tried this sort of experiment (measuring angles of a really big triangle between mountain tops). But apparently connections with physics were not among Klein's motivations.
Complementing Conifold's answer, it might perhaps be instructive to quote from Minkowski's presentation Raum und Zeit at the 80th Congress of German Natural Scientists in Cologne on the 21st of September 1908, a short few months before his own death. In this speech he makes it clear the physical origin of his eponymous spacetime:
Mein Herren! Die Anschauungen über Raum und Zeit, die ich Ihnen entwickeln möchte, sind auf experimentell-physikalischem Boden erwachsen. Darin liegt ihre Stärke. Ihre Tendenz ist eine radikale. Von Stund′ an sollen Raum für sich und Zeit für sich völlig zu Schatten herabsinken und nur noch eine Art Union der beiden soll Selbständigkeit bewahren.
or in translation
Gentlemen! The views of space and time that I wish to lay before you have sprung from the soil of experimental physics and therein lies their strength. They are radical. Henceforth, space by itself, and time by itself, are doomed to fade away into mere shadows, and only a kind of union of both will retain an independent reality.
Though the idea of combining space and time dimensions was already tacit in Galileo's Relativity Principle, which expands the set of classical symmetries (translations and rotations for space, translations for time) to include a symmetry transform (the "Galilean" boost) that involves both space and time dimensions, it was not explicitly recognized as such before the 20th century. So the marriage of the two was an elopement that was not consummated until a few hundred years later.
The idea that the two should be combined with the specific metric that Minkowski advocated was certainly not present before Einstein. Though Lorentz introduced a set of symmetries to try and recover the "stationary" form of Maxwell's isotropic constitutive laws, the equations he actually published were not relativistic (contrary to a claim made by an earlier respondent), but actually covariant with respect to the Galilean transform.
The idea that Lorentz's theory was relativistic is a widespread folklore myth present, even in the Physics community. But, in fact, the non-relativistic nature of Lorentz's theory was a point that Einstein, himself, made shortly after the publication by Minkowski and later by Einstein and Laub of a relativistic theory for moving media in 1908.
Minkowski's paper was, in fact, where he also introduced his geometry. The note about Lorentz's theory was "Comment on the paper of D. Mirimanoff 'On the Fundamental Equations...'" Annalen der Physik 28 (1909): 885-888, contained in the compilation below. And, I'll say a few words on it here, below, as well.
The Einstein-Laub papers are discussed in further detail here https://einsteinpapers.press.princeton.edu/vol2-doc/539 along with some notes on their relation to Minkowski's "moving media" paper ("Die Grundgleichungen für die elektromagnetischen Vorgänge in bewegten Körpern", Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Mathematisch-Physikalische Klasse, pp. 53–111). As you can see in the discussion in the attached link: initially Einstein and Laub were not keen on the idea of fitting Relativity into a Minkowski geometry.
Minkowski's paper actually went a bit further, by the way, than just laying out a set of equations for moving media: bringing up notions of transformation groups, for instance. I also notice that he appears to have used some kind of 5th coordinate (which he denoted by ϑ). I haven't read the paper in enough detail, yet, to see why he did that, but it's something that's of interest to me, for reasons that will shortly become clear.
The Maxwell equations for moving media with what's now known as the Maxwell-Minkowski constitutive laws are
∇·𝐁 = 0, ∇×𝐄 + ∂𝐁/∂t = 𝟎
for electric force 𝐄 and magnetic induction 𝐁,
∇·𝐃 = ρ, ∇×𝐇 - ∂𝐃/∂t = 𝐉
for electric induction 𝐃 and magnetic force 𝐇, with charge density ρ and current density 𝐉, and
𝐃 + α 𝐆×𝐇 = ε(𝐄 + 𝐆×𝐁), 𝐁 - α 𝐆×𝐄 = μ(𝐇 - 𝐆×𝐃)
for the Maxwell-Minkowski constitutive law, for moving media with a reference velocity 𝐆.
What Minkowski and Einstein-Laub published was the version of the equations equivalent to the above where α > 0, with light speed identified as c = 1/√α. For the vacuum, εμ = α, and provided that |𝐆| < c, the constitutive equations are equivalent to the "stationary" isotropic form for 𝐆 = 𝟎.
In contrast, what Lorentz published (as Einstein also pointed out in 1909) is equivalent to the non-relativistic version of the constitutive law, which is the case where α = 0. The resulting equation set, in turn, is consistent with those used by Heaviside and Hertz. All of them are non-relativistic.
What Lorentz did in his papers was perform a Galilean transform on the equations to get the non-relativistic (and incorrect) constitutive law with α = 0; and then to handle the case of the vacuum, posed a second ad hoc transform to recover the "stationary" form of the law. That second transform is equivalent to composing a reverse Galilean transform and forward Lorentz transform. So, in this way, he stumbled onto what we now call the Lorentz transforms.
But instead of characterizing it as a fundamental property of space-time and then most importantly making the relativistic corrections to the constitutive law, he kept the constitutive law intact in non-relativistic form and pushed the inverse-Galilean + forward-Lorentz fix as something that probably arose from the mechanical properties of the electromagnetic field, itself. His equations are non-relativistic and Galilean-covariant; and, because of that, even with his extra fix, his constitutive equations for the moving media were simply wrong.
The equations, when α > 0, are associated naturally with a geometry which has - as its invariants -
dt² - α (dx² + dy² + dz²), ((∂/∂x)² + (∂/∂y)² + (∂/∂z)²) - α (∂/∂t)², dt (∂/∂t) + dx (∂/∂x) + dy (∂/∂y) + dz (∂/∂z),
which is the same as what characterizes Minkowski geometry.
In contrast, the equations published by Lorentz -- and others pre-dating Einstein -- (as well as the equations describing the fundamentals of Newtonian physics) accord with the geometry that has these invariants
dt², (∂/∂x)² + (∂/∂y)² + (∂/∂z)², dt (∂/∂t) + dx (∂/∂x) + dy (∂/∂y) + dz (∂/∂z).
And this is what people pre-dating the 20th century would have been faced with, if they wanted to devise a unified chrono-geometry for space-time ... rather than Minkowski geometry. So, for that reason it never really got anywhere; and also for that reason, no notion of Minkowski geometry would have arisen before the 20th century in connection with electromagnetic theory or any part of Newtonian Physics.
In fact, as you're about to see, there would have been a strong disincentive against putting forth a four dimensional geometry for one very simple reason - a reason that wasn't given a proper account until well into the 20th century. And it may also be what lay behind Einstein and Laub's initial resistance to Minkowski's 4D geometry idea even if they themselves didn't realize it at the time.
The reason is this: mass, kinetic energy and momentum transform together as a 5 vector, not a 4 vector! Before Relativity it would have been natural to consider the mass-momentum-energy 5-vector (m, 𝐩, H). Einstein, in his earlier papers treated H and m separately and didn't write anything like E = Mc², but rather M = m + H/c² for "moving mass" and kinetic energy. Under a Galilean boost transform, with boost velocity 𝐰, the 5 vector transforms as (m, 𝐩, H) → (m, 𝐩 ‒ 𝐰m, H ‒ 𝐰·𝐩 + ½ w² m). Under an infinitesimal boost 𝞄, it transforms as Δ(m, 𝐩, H) = (0, ‒𝞄m, ‒𝞄·𝐩).
For relativistic theory, what Einstein showed is that the infinitesimal transform has to be modified to Δ(m, 𝐩, H) = (0, ‒𝞄(m + αH), ‒𝞄·𝐩), taking α = 1/c² > 0, in place of α = 0. Hence, the earliest papers asserted only that "energy has inertia, too" and identified the "moving mass" of a body as M = m + αH.
After Minkowski's paper, the consensus fell on strengthening the statement "energy has inertia, too" to "mass is energy" and combining the kinetic energy and mass into the total energy E = mc² + H; shifting the focus by writing E = Mc², instead of M = E/c²; and then considering only the 4-vector (𝐩, E) instead of the 5-vector, since it is closed under transforms, with the infinitesimal boost given by Δ(𝐩, E) = (‒α𝞄E, ‒𝞄·𝐩). That's the energy-momentum 4-vector.
Probably the most important feature of this 4-vector is that it forms an invariant with the coordinate differentials (dx, dy, dz, dt) = (d𝐫, dt), since their boost transform is given infinitesimally by Δ(d𝐫, dt) = (‒𝞄 dt, ‒α𝞄·d𝐫). Consequently, together they yield the invariant
𝐩·d𝐫 ‒ E dt.
No such invariant can be formed - either relativistically or non-relativistically - with the momentum and kinetic energy. In fact, what you find is that
Δ(𝐩·d𝐫 ‒ H dt) = (‒𝞄(m + αH))·d𝐫 ‒ (‒𝞄·𝐩) dt + 𝐩·(‒𝞄 dt) ‒ H(‒α𝞄·d𝐫) = ‒m(𝞄·d𝐫)
so you lose one of the main advantages of the 4D geometric approach used in Relativity. It wasn't until well into the 20th century - the 1950's in fact - that this issue was resolved - to associate another coordinate (u) with the mass m, possessing the infinitesimal transform Δ(du) = 𝞄·d𝐫, and then one recovers an invariant for the 5-vector:
Δ(𝐩·d𝐫 ‒ H dt + m du) = 0.
This applies to both the non-relativistic case α = 0 and the relativistic case α > 0! In the former case, the geometry is known as a Bargmann geometry and the symmetry group that goes with it is called the Bargmann group. It is equivalently described as the central extension of the Galilei group.
This suffices, in fact, to give you the basis for the Newtonian analogue to General Relativity, as discovered much more recently in the 1980's(!) "Bargmann Structures and Newton-Cartan Theory" https://inspirehep.net/literature/202550. The non-relativistic version of the Schwarzschild solution is given by
dx² + dy² + dz² + 2 dt du ‒ 2U dt² = 0
where U = -GM/r is the gravitational potential per unit mass. The geodesics are the orbits described by Newton's Law of gravity.
In the relativistic case, the group and geometry have no name that I am aware of; but it is closely linked to a variety of disparate issues that have never really fit well Relativity under the framework provided by Minkowski geometry (e.g. complexification of the Dirac algebra, the "Stuckelberg trick", as well as a host of other issues). It also arises in 5D cosmology, and the extra coordinate can be re-interpreted as "historical time" s = t + αu, since the coordinate (s) transforms as an invariant. Ironically, that both recovers and gives geometric realization to Lorentz's "absolute time" versus "local time". So, you might say, on account of this, that Lorentz was already heading in this direction.
The symmetry group for the geometry is just the 10 generator inhomogeneous Lorentz group (= Poincaré group), itself, trivially extended with a translation symmetry for the u coordinate as an 11th generator.
The Schwarzschild solution, itself, can be written equivalently in this geometry as
dx² + dy² + dz² + 2 dt du + α du² ‒ 2U dt² ‒ 2αU dr²/(1 + 2αU) = 0
where r = √(x² + y² + z²) and dr = (x dx + y dy + z dz)/r; with the s coordinate giving you the proper time for the Schwarzschild metric.
This is what people before the 20th century would have been faced with, if they tried to unify space and time into a single chrono-geometry. With α > 0, using a 5D geometry is optional, since you can get away with 4D. But with α = 0, it virtually mandatory to have the extra (u) coordinate in place, too, to get things to work consistently. Like Michael Fox would have said, after traveling back in time to steam-punk days, I don't think they would have been ready for that.
Einstein was a mathematical physicist rather than a pure mathematician, that is one interested in mathematics for its own sake. It was likely that he was aware of non-Euclidean geometry given its prominence in mathematical circles, but given that it hadn't been shown to be useful in physics, he knew nothing about it until Minkowski's reformulation brought it to his attention. Mathematicians were already interested in this space as it was an instance of hyperbolic space as opposed to our usual Euclidean space; the other main example of non-Euclidean geometry being spherical or elliptical space.
This was of some consequence, as it was in this language that he formulated General Relativity - however, he always insisted on the primacy of the physical; for example, when a colleague said that motion in GR should be understood geometrically via the geodesic equation, he insisted otherwuse; to him, that equation was to be understood as a unification of inertia and gravitation. In this sense, it harks back to Hertz who was looking at a way of unifying physics by expressing everything through constraining forces.