I can't imagine mathematics without sets, but the question "what was mathematics like before there were sets" is not answerable, I think. Instead, a good answer to the title question should cover a certain aspect of the more general question.
An immediate motivation of Cantor to work on what became set theory was his earlier work on trigonometric series. To solve a problem in that domain he considered the set (a closed set) of zeros of such a function, then the derived set of this set, the derived set of this set and so on. This is all still classical, but then had to go a step beyond that to consider first the intersection of all these sets, and then the derived set of that set and so on.
So he came to consider transfinite ordinals.
This is discussed in various places, including "Set Theory and Uniqueness of Trogonometric Series" by Kechris or "Uniqueness of trigonometric series and descriptive set theory, 1870–1985" by Roger Cooke (Archive for History of Exact Sciences, 1993)
The original paper is (I think) "Ueber die Ausdehnung eines Satzes ais der Theorie der trigonometrischen Reihen (Math. Annalen, 1872)"
Another motivation was his earlier work on number theory. Using what is now called a diagonalisation argument he was able to prove results on the existence of transcendental numbers. This is in his 1874 paper "Über eine Eigenschaft des Inbegriffes aller reellen algebraischen Zahlen" ("On a Property of the Collection of All Real Algebraic Numbers")
In brief, the original motivation was to have better tools for making progress on existing problems.
Actually Cantor was working on a specific problem from the theory of trigonometric series, the so-called uniqueness problem (I cannot be more specific until MathJax is introduced to this site). This problem led him to consideration of arbitrary sets on the real line. I mean more complicated sets than finite sets or finite union of intervals. At that time there was no tools and no terminology to study arbitrary sets, so all this had to be created.
In the process of this study he created not only the set theory but also what is called now General topology. (In is interesting to notice that the original problem about trigonometric series has no complete solution to this day:-)
The original method of proof, the so-called "diagonal procedure" goes back to Cantor's predecessor, Paul du Bois Reymond, who was also studying trigonometric series.
According to Cantor himself it was his desire to replace the mechanical explanation of nature by a more complete theory. See several aspects in What of Cantor's claims has become true??