Georg Cantor was the first; see at least:
By an "aggregate" (Menge) we are to understand any collection into a whole $M$ of definite and separate objects $m$ of our intuition or our thought. These objects are called the "elements " of $M$.
In signs we express this thus :
$$M = \{ m \}.$$
The formal set builder notation:
$\{ x \mid \varphi(x) \}$
is an evolution of W&R's Principia Mathematica notation for "abstraction": $x \in \hat z(\psi z) \equiv \psi x$.
An early example is in:
The class operator $\{ x \mid \mathfrak A(x) \}$ "the class of the $x$ such that $\mathfrak A(x)$" [footnote : This symbol (used in some newer papers) is here taken, for the sake of easier print, instead of Russell's class symbol $\hat x \mathfrak A(x)$, whose adoption was first intended.]
The 2nd French edition of:
On introduira [...] un symbole fonctionnel; dans ce qui suit, nous utiliserons le symbole $\{ x \mid R \}$; le terme correspondant ne contient pas $x$. C'est de ce terme qu'il s'agira quand on parlera de «l'ensemble des $x$ tels que $R$». [...] par suite la relation $R$ est équivalente à $x \in \{ x \mid R \}$.
The notation was not present in the English translation of 1968.