Have orthogonal complex matrices appeared in the literature?

Question

According to https://en.wikipedia.org/wiki/Orthogonal_matrix,
https://en.wikipedia.org/wiki/Unitary_matrix, and Friedberg et al.'s Linear Algebra (4th edition), a matrix $A\in F^{n\times n}$

• is unitary if $A^*=A^{-1}$ and
• is orthogonal if $A^T=A^{-1}$

no matter whether $F$ is $\mathbb{C}$ or $\mathbb{R}$. Therefore, if $F=\mathbb{R}$, then $A$ is unitary if and only if $A$ is orthogonal (since $A^* = A^t$). However, if $F=\mathbb{C}$, then a unitary matrix (e.g., the matrix consisting of a single entry $i$) is not necessarily orthogonal. I have seen a lot of unitary (i.e., orthogonal) real matrices and unitary complex matrices. But, I cannot find anything related to orthogonal complex matrices. Have orthogonal complex matrices appeared or been studied in the literature? If they have, can someone please provide a reference?

Also, if almost all orthogonal matrices in the literature are unitary, then are there any reasons why the definition of orthogonal matrix stays the current way? Wouldn't things be easier to define orthgonal matrix as unitary real matrix, just like orthogonal linear operator is defined as unitary linear operator on $\mathbb{R}$ (see https://en.wikipedia.org/wiki/Orthogonal_transformation, https://en.wikipedia.org/wiki/Unitary_transformation, and Friedberg et al.'s book)?

Answer 1

They did. A natural way to treat such matrices is to introduce an indefinite inner product on $\mathbb{C}^n$, a non-degenerate bilinear form $(z,w):=z_1w_1+z_2w_2+\dots z_nw_n$, instead of the usual sesquilinear one. Then $A^T=A^{-1}$ is equivalent to $(Az,Aw)=(z,w)$, i.e. complex orthogonal matrices are isometries of this space. The "orthogonal"/"unitary" split in terminology is useful in signaling whether real or complex spaces are involved, exactly because complex orthogonal matrices are rarely considered.

General bilinear forms are not as technically handy as sign definite sesquilinear forms, so they are less studied. On complex orthogonal matrices specifically the volume of publications is fairly small. In 1982 Choudhury defended a thesis, The Role of Complex Orthogonal Matrices in Matrix Factorization. Some of the results are published in a paper with Horn, An analog of the singular value decomposition for complex orthogonal equivalence (1987), see also Horn-Merino, The Jordan Canonical Forms of complex orthogonal and skew-symmetric matrices (1999). The most recent entry in MathSciNet is Celakoska-Hadzieva, On complex homogeneous space of vectors with constraints (2017). Occasionally they are asked about on Math SE, e.g. Is there an orthogonal matrix that is not unitary? and On eigenvalues of complex-orthogonal matrices.

However, some linear spaces with indefinite inner products (not of the sort introduced at the top), including infinite-dimensional ones, have been studied in functional analysis since 1940-s. Pontryagin's paper, Hermitian operators in spaces with indefinite metric, is from 1944. After Krein's work in 1950-s a subclass of them (including those considered by Pontryagin) got the name Krein spaces, those admit direct decomposition into two subspaces on which the form is sign definite. Early work of Phillips, Krein and Naimark on operators in such spaces is discussed e.g. in Helton, Unitary Operators on a Space with an Indefinite Inner Product, a book treatment is Azizov-Iokhvidov, Linear operators in spaces with an indefinite metric. From 1973 Mathematics Subject Classification includes 47B50: Operators on spaces with an indefinite metric.

Answer 2

Yes, orthogonal matrices with complex entries appeared at least as early as 1900, in E. Cartan's classification of simple Lie algebras (and Lie groups). In many ways, the complex numbers could be replaced by any algebraically closed field of characteristic $0$.

Thinking of complex orthogonal groups as real Lie groups ("forgetting" the complex structure) does produce a family of semi-simple real Lie groups, about which we have many structure theorems (Iwasawa decomposition, Cartan decomposition, Bruhat decomposition). The representation theory of such groups, both finite-dimensional and even more so infinite-dimensional, is of interest both in its own right, and in applications to automorphic forms and number theory (e.g., Langlands programme and such).