When and how was the geometric understanding of gauge theories developed?

Question

In theoretical physics, the modern perspective on gauge theory is that it is most elegantly described in the 'language' of differential geometry. I am interested in the history behind these ideas.

Firstly, it appears (e.g. this anecdote by C.N. Yang) that the 'fathers' of (physical) gauge theory were unaware of the deep connection to geometry. In fact, I am not sure the rather advanced mathematical framework that one needs to understand gauge theories was even in place (i.e. understood as a fully abstract theory) at all at the time. I also think, but do not know for sure, that the mathematical theory of principal bundles and other related mathematical objects and/or structures was first developed fully abstractly, before it was realized how useful it was in describing physics.

I guess what I'm asking for is a (cursory) account of the following:

1) What were the key insights that allowed physicists and mathematicians to understand gauge theories in this light?

2) When (and by whom?) were these essential steps first made?

---

For those interested, this is a (tangentially) related question of mine, from physics.SE

Answer 1

I'll focus on the geometry of Yang-Mills theories specifically, but as Conifold's answer points out, gauge theories were studied geometrically long before the work of Yang and Mills.

The foreward to volume 5 of Atiyah's collected works (on gauge theories) contains some historical comments on this from the mathematics side. You can read it here. This is probably somewhat later than what you're looking for, since it has more to do with the later study of the deep geometric properties of Yang-Mills theory than the (comparatively) simple interpretation of Yang-Mills as an action on the space of connections.

Atiyah says his own interests in the subject began in 1977, when he says interest in gauge theories among mathematicians was just beginning to start (citing Yang's influence in mathematical circles). That matches pretty well his writings. The first writing included in his collection is [1]. This paper he cites with bringing many of the ideas in gauge theory to the mathematics community. In it, he shows that the multi-instanton problem can be reduced to constructing suitable vector bundles on 3-dimensional projective space. The construction was completed in [2]. He wrote several more papers that year and in the following years on the geometry and topology of Yang-Mills fields. His (and other) papers in the late 70s are the earliest that I know of by mathematicians on the geometry of Yang-Mills theory.

By the early 80s, several other people began publishing on this topic. Some of the big names are Donaldson, Hitchin, and Witten. In particular, Donaldson's study of 4-manifolds via instantons in [3] proved to be of great interest. By that point, it had become clear that the Yang-Mills equations could be used to great effect for more than just physics. It's fair to say that the interest in them continued well through the 80s and in some cases to the present day.

The earlier developments before this were almost completely taken on by physicists. I know less of the story here because physicists seem less inclined to write detailed accounts of the chronology of events. It's clear that by 1977, it was already known by physicists that Yang-Mills could be viewed in terms of an action functional on the space of connections, though the geometric consequences had not been explored. (Of course, physicists had bigger problems to deal with before that, like understanding how to give gauge bosons mass and proving the renormalizability of quantum Yang-Mills.) The earliest source I know for this is by Popov in [4] in 1975. In this, he shows that the now-standard geometrical interpretation of Yang-Mills via principal bundles and connections yields the Yang-Mills equations. However, it's quite possible that some of the ideas there originated earlier, though I can't see anything in the citations to indicate such.

References:

[1] M. F. Atiyah and R. S. Ward: “In­stan­tons and al­geb­ra­ic geo­metry,” Comm. Math. Phys. 55 : 2 (1977), pp. 117–​124.

[2] M. F. Atiyah, N. J. Hitchin, V. G. Drin­fel’d, and Yu. I. Man­in: “Con­struc­tion of in­stan­tons,” Phys. Lett. A 65 : 3 (1978), pp. 185–​187.

[3] S. K. Donaldson, "An application of gauge theory to four-dimensional topology", Jour. Differential Geometry 18 (1983), 279-315.

[4] Popov, D. A., "Theory of Yang-Mills Fields", 1975, Teor. Mat. Fiz. 24, 347.

Answer 2

I will focus on the history before the Yang-Mills paper. The first harbinger was the introduction of the scalar potential for the gravitational field by Lagrange in 1773. In 1864 Maxwell introduced vector potential for the magnetic field, which can be interpreted as a connection form, making magnetic theory the first gauge theory in hindsight. In the famous 1905 paper Einstein joined the scalar and the vector potentials into a 4-potential, which is a connection form on the four dimensional background. At the same time developments in differential geometry led to another type of connection, the Riemannian one, in the form of covariant derivative. Christoffel symbols were the first appearence of it, and Ricci and Levi-Civita developed the theory based on "absolute differentiation" (covariant derivatives) systematized in their 1900 book.

But this qualifies as more of a pre-history. The real history of gauge theory begins in Hermann Weyl's papers of 1918-1920, where he pulled together different strands that developed up to that point. Ironically, Weyl was also motivated by the theory of gravity, the general relativity. First, he noticed that one does not need a Riemannian metric to define parallel transport, what is now called affine connection suffices. He then realized that Riemannian geometry is not entirely local, lengths of vectors at different points, being numbers, can be compared in the absolute sense. To localize it entirely he switched to conformal metrics accompanied by a field of scales, in modern terminology sections of a principal bundle with the fiber the multiplicative group of positive real numbers $R^+$. Transporting scales (gauges) requires specifying a 1-form, a principal connection form, and changing them induces a transformation of this form, a gauge transformation. Weyl then formulated for the first time the principle of gauge invariance, the form of natural laws must be invariant under local changes of gauge.

Since the formulas he obtained were identical to those of electromagnetism Weyl conjectured that the curvature of his scale connection was exactly the electromagnetic field. He even wrote coupled equations for electromagnetic and gravitational fields producing the first unified field theory (Kaluza proposed his five dimensional theory around the same time, he published in 1921). Unfortunately, it was unphysical, as Einstein pointed out shortly.

Weyl came back to the gauge idea in 1929, from the quantum mechanical perspective. This time instead of the field of scales he used the field of phases, that is replaced $R^+$ with U(1) as the fiber. The formulas are almost the same, except for the presence of i, he never considered non-Abelian principal bundles. In 1930 Dirac used non-trivial U(1) bundles to describe magnetic monopoles. These developments are described in detail by Varadarajan in his survey paper.

After Weyl the mathematical and the physical paths diverge once again. Elie Cartan used specialized connections to study Pfaffian systems in 1926, they lived on fiber bundles with fibers being homogeneous spaces (Kleinian geometries), his work solidified the view of connections as matrix-valued 1-forms. Two big developments came in 1950, Koszul gave a general algebraic description of connections on vector bundles as covariant derivatives, and eliminated the need for non-tensorial objects like Christoffel symbols. Ehresmann, Cartan's student, finally gave a general definition of connection on a principal bundle, Abelian or not, and clarified the general relationship between connections on principal and associated bundles. His notion of connection was very abstract though, a horizontal distribution in the tangent bundle to the total space, which can be used to define the parallel transport directly.

When Yang and Mills introduced in their paper the first non-Abelian gauge theory (with SU(2) as a fiber) in 1954 they weren't aware of these mathematical developments, the relationship between the principal connections and gauge fields was clarified in the years following their publication.