According to Klein, the first mathematician who considered space-time as a 4-dimensional manifold was Lagrange, but these ideas were not immediately developed by others. Then he mentions Cauchy and Cayley (Cayley published in 1844 "Chapters on the analytic geometry in $n$ dimensions"), and credits Grassmann (1844) with the first systematic exposition.
While geometers discussed $4$ dimensions, people doing statistical
mechanics (Maxwell, Bolzmann and others) were inevitably lead to consideration
of $N$-dimensional manifolds where $N$ is of the order of Avogadro number.
Ref. F. Klein, Vorlesungen uber die Entwicklung der Mathematik im 19 Jahrhundert
(there is an English transl.), Ch. IV, 3-d part: "Spaces of n dimensions".