The following post is long, but I decided to write more rather than less in case it's helpful. I tried to make it clear, quick, and easy to skip to the short version of my question, so the reader can decide whether they're interested in answering this.
For context and contrast
In most mathematical subjects, I can give a story about why it is what it is. For instance, in algebra, if I were to explain what this subject is about, I would tell a story about some quantity that is unknown but what is known is its relationship to other quantities. One large initial concern of algebra is to infer the unknown quantity. From there, the focus can shift to understanding the abstract algebraic properties of the operations themselves, how one quantity varies as a function of another, and so on.
In calculus I can give a story about modeling physics. Suppose you are a medieval scientist who rejects Aristotle's physics, and you conjecture that gravity gives objects a constant acceleration. But to test the theory out you need to determine from that hypothesis a measurable, and therefore you need to infer the position function. From there you can get into all the other topics. It's an over-simplification of the actual history, but I find it good enough to help me understand the subject, and it helps me explain the condensed essence of the subject to a new student.
And I could go on, with complex analysis, real analysis, abstract algebra, linear algebra, mathematical logic, set theory. For each one of these I can give a story that at least I find satisfying, and hopefully so would a student new to the subject.
For topology I don't know how to do that. I can tell half intelligible stories. Like we wanted to understand continuity for many-variable complex functions, and so we became interested in the idea of an open ball in the codomain, and corresponding open ball in the domain. And with this you can motivate the idea of metric spaces more generally. And you can then point out that some spaces don't have a natural metric, yet you still want to define continuity, and to do that you want to be able to say what the open sets are without appealing to any metric.
Ok, all of that makes sense, but then there's the leap from there to the modern definition of a topology. That leap is too large. It doesn't tell you why "a subset of the powerset containing $\emptyset$ and the universe, closed under union and finite intersection" is what we want from open sets, or why it delivers a satisfying version of continuous functions.
For instance, it doesn't tell me why we shouldn't include in the definition extra axioms like the separation axioms--that too seems like a generalization of open sets, so why not pick that? Or we could go in the other direction and remove the requirement that open sets be closed under finite unions. That too gives you objects which are "like" open sets. The open sets of real numbers, for instance, would still have this property. So maybe that's the generalization we want. But we drew the line here, at the modern definition of a topology, and as a mathematical community, said "anything more general, we are not willing to call an open set, and anything less general we call a special case of a more general notion of an open set". Why draw the line here?
I'm sure there's an explanation, I just don't know what it is. I've read about six chapters of Munkres so I know maybe the first thing about point-set topology. I've solved a couple problems having to do with the Zariski topology, and I've heard about (but really know nothing about) $C^*$-algebras. I know essentially nothing about algebraic and analytic topology. So that's about where my current level of understanding is. But as far as I can tell, the Zariski and Sierpinski topologies, and $C^*$-algebras, were all developments long after the definition of a topology was decided. So they seem a little out of bounds to explain why mathematicians did and should have defined topology the way they did.
My actual question
What caused mathematicians to pick these axioms and not some more or less restrictive axioms, to define a topology?
My current best answer to the question
The answer I give below is almost certainly wrong in several places, so I welcome corrections. I'm including it here to possibly give some sense of what is missing and possibly wrong about my current understanding.
So Newton invents the calculus. Riemann much later contributes to a more rigorous foundation, and also advocates for a more abstract understanding of mathematics generally. He worked particularly on the idea of a manifold, and it was in this setting that he repeated throughout his career that we should develop a mathematics of geometry without concern for a metric. The foundational crisis occurs in set theory and logic. We start to see how we can make set theory well-founded, and people start developing several mathematical disciplines on set theory.
Now here I really get my timeline fuzzy, but I know that at about this time Klein is working on the uniformization theorem (I don't know what that is, but it seems to be an initial motivation for thinking about "covering", which I assume is the same as "an open cover" and the idea of a compact set having a finite subcover.). I guess somehow the modern idea of a topology "fit well" with Klein's work? Like, I guess lots of various proposed axiom sets existed, which delivered variously special and general ideas of open sets. Some of them were general enough to apply to this problem, and others weren't?
I think, earlier than this, Hilbert had listed several geometric axioms. Later, Riesz proposed one set of axioms, which were equivalent to the modern definition of a topology, but with the extra requirement of the Hausdorff axiom. After that, Hausdorff took Hilbert's list of geometric axioms and crossed off all but four of them, and this collection of axioms was equivalent to the modern definition of a topology, which ended up being equivalent to what Riesz had done. But later Kuratowski did the same as Hausdorff but dropped the Hausdorff axiom, which might mark the start of the modern definition of a topology.
Because Kuratowski's system applied more generally than Riesz's/Hausdorff's, but was still sufficiently powerful to prove the uniformization theorem, ... maybe that's why mathematicians all agreed that Kuratowski's system was the ideal place to draw the line? Is that at least a plausible oversimplification of the story?
I know that Poincare was important, both before and after this. But as far as I can tell, his importance was in developing algebraic topology, so I'm not sure that it's necessary to bring him into the history of the founding of point-set topology.
I've also seen some people's names thrown around as contributing to the founding of modern point-set topology, but I haven't yet been able to really appreciate the nature of their contributions. Betti, Brouwer, Cantor, Borel, and more, I'm not sure exactly how they fit into this story, but I know that a more complete history includes them.
Resources I've consulted so far
For posterity I'll collect a bunch of resources I've consulted, to one degree or another, for my current understanding. Because I'm not a mathematician, I am not able to understand everything that I've read in these resources and that may explain any mistakes I've made above.
Dieudonne, A History of Algebraic and Differential Topology, 1900-1960
Fierrieros, Labyrinth of Thought
James, History of Topology
Aull and Lowen, A Handbook of the History of General Topology
I did not consult the following text because it's just too expensive, but I have heard people say it's very good:
Manheim, The Genesis of Point-set Topology