I recently read the following quote (unfortunately, I copied it down without attribution):
You may be surprised to know that Fourier analysis played a role in the early development of set theory. In fact, it was a Fourier-analytic question that led to Cantor's introduction of ordinal numbers.
How was Fourier analysis important to the development of set theory? This does indeed seem rather surprising to me.
It was Fourier series rather than Fourier transform. Considering that the sets where Fourier series converge can be very intricate it is not that surprising that they led Cantor to develop set theory for subsets of real numbers. But at some point he took a turn into the abstract (for which he is best known today) that was not really motivated by the initial problem, but matched his more metaphysical interests, see Ternullo, Gödel’s Cantorianism.
The specific problem posed to Cantor by Heine was the following, see Srivastava, How did Cantor Discover Set Theory and Topology? Suppose a trigonometric series converges to $0$ pointwise, must all of its coefficients be $0$ as well? Essentially, it is the uniqueness problem for the Foureier series. Dirichlet, Heine, Lipschitz and Riemann tried their hand at it, but were only able to prove the result under heavy restrictions (e.g. Heine assumed uniform convergence).
Cantor not only was able to prove it in full generality (1870), but noticed that the assumption of pointwise convergence everywhere could be relaxed. He introduced a "set of uniqueness" as such a set that the uniqueness was assured by pointwise convergence outside of it. Utilizing Heine's notion of ‘condensation point’ (now limit or accumulation point) Cantor defined the derived set $P'$ of a set $P$ as the set of its condensation points. He then proved first (1871) that $P'=\emptyset$ was sufficient for $P$ to be a set of uniqueness, and later that even $P^{(n)}=\emptyset$ for some finite $n$ sufficed. It is the derived set construction that served as a springboard for Cantor's set theoretic and point set topological breakthroughs.
The proof required, in particular, making the notion of real numbers more precise, and led to Cantor's construction of them in terms of Cauchy sequences of rationals. After that Cantor's interests shifted from the trigonometric series to the more abstract properties of point sets of reals and then to abstract sets in general. He introduced denumerable (now countable) sets, identified rationals and algebraic numbers as denumerable, and then proved non-denumerability of open real intervals. This led to the idea of comparing "sizes" of infinite sets, their cardinalities, and eventually to the diagonal argument and the continuum hypothesis.
In another line of inquiry, after finding examples of sets such that $P^{(n)}\neq\emptyset$ for any finite $n$, Cantor extended the recursion into the transfinite (his set of uniqueness result continues to hold when $n$ is replaced by any countable ordinal). Transfinite ordinals were a new notion that he had to introduce, develop, and defend against the Aristotelian preconceptions about actual infinity, see Why did Cantor (and others) use c for the continuum? The transfinite induction, as well as the notions of dense and perfect sets came out of this circle of ideas. But the ultimate solution to the uniqueness problem for the Fourier series was no longer in Cantor's view, and had to wait until the introduction of Lebesgue measure theory. The sets of uniqueness turned out to be the sets of Lebesgue measure zero.
Unfortunately, I do not have a concrete reference... but I seem to recall that Cantor's earliest work was about "sets of uniqueness" for Fourier series (I think not Fourier transforms, but I could easily be mistaken).
This would be similar to other late 19th century "constructive" analysis projects, where limits of limits of ... continuous... functions were taken. Even into the early 20th century, there were attempts to "classify" subsets of the real line, but/and, to my understanding, it turned out that there are tooooo many, and toooo complicated, such sets.
Part of that classification attempt involved transfinite limits... though not using that word. Long ago, I had some Dover reprints of some archaic "real analysis" texts that took that approach.
paul garrett has the idea. $E \subseteq \mathbb R$ is a set of uniqueness if: given a trigonometric series $\sum_{n=-\infty}^\infty c_n e^{int}$, if it converges to $0$ except possibly on $E$, then $c_n = 0$ for all $n$.
Here is a description.
The empty set is a set of uniqueness. This is just a fancy way to say that if a trigonometric series converges to zero everywhere then it is trivial. This was proved by Riemann, using a delicate technique of double formal integration; and showing that the resulting sum has some generalized kind of second derivative using Toeplitz operators. Later on, Cantor generalized Riemann's techniques to show that any countable, closed set is a set of uniqueness, a discovery which led him to the development of set theory.
Cantor's research (I think) went like this. A finite set $E$ is a set of uniqueness. A set $E$ with finitely many limit points is a set of uniquness. A set whose limit points have finitely many limit points is a set of uniqueness. Continuing with this, Cantor was led to transfinite ordinals... Of course, when he was doing this, "countable" and "closed" were not standard notions.
Here are the references to the original papers of Cantor:
On a theorem concerning trigonometric series. (Ueber einen die trigonometrischen Reihen betreffenden Lehrsatz.) Borchardt J. LXXII, 130-138 (1870).
Proof that a function given for every real value of by a trigonometric series has only one representation in this form. (Beweis, dass eine für jeden reellen Werth von durch eine trigonometrische Reihe gegebene Funktion sich nur auf eine einzige Weise in dieser Form darstellen lässt.) Borchardt J. LXXII, 139-142 (1870).
On trigonometric series. (Ueber trigonometrische Reihen.) Clebsch Ann. IV, 139-143 (1871).
On the extension of a theorem from the theory of trigonometric series. (Ueber die Ausdehnung eines Satzs aus der Theorie der trigonometrischen Reihen.) Clebsch Ann. V, 123-133 (1872).
Borchardt J. = Journal fur die reine und angewandte Mathematik,
Clebsch Ann. = Mathematische Annalen.
Both journals are available on Internet.
