For first use of mathematical terms, see http://jeff560.tripod.com/mathword.html
CLOSED SET. Georg Cantor (1845-1918) in "De la puissance des ensembles parfaits de points," Acta Mathematica IV, March 4, 1884, introduced (in French) the concept and the term "ensemble fermé" [Udai Venedem].
.
OPEN SET. René Baire, “Sur les fonctions de variables réelles,” Annali di Matematica Pura ed Applicata (3) 3 (1899), 1-123 has:
Je dirai que $S$ est une sphère fermée, $S'$ une sphère ouverte. La différence essentielle entre ces deux ensembles réside dans la fait suivant:
Etant donné un point quelconque de $S'$, il existe une sphère ayant ce point pour centre et de rayon positif, dont tous les points font partie de $S'$.
J'appelle d'une manière générale domaine ouvert à n dimensions tout ensemble de points possédant cette propriété.
A translation is:
I will say that $S$ is a closed sphere and $S'$ is an open sphere. The essential difference between these two sets resides in the following fact:
Given any point in $S'$, there exists a sphere having this point for its center, and of positive radius, of which all its points are a part of $S'$.
More generally, I will call an open domain in n dimensions any set of points that possesses this property.
This citation and translation were provided by Dave L. Renfro, who writes, “I believe that, in addition to being the first appearance of the term ‘open set,’ this may also be the first appearance of the concept itself. Prior to this, I think everyone just talked about the interior of a set of points, or about the ‘sum’ (i.e. union) of the interiors of sets from some specified collection of sets (often intervals).”