The man is Paul Erdős. As pointed out by the OP in the comments, his "nomadic" lifestyle is briefly mentioned in the Brown Numbers - Numberphile video, 3.05-3.30. It is also concisely described in Currey's book Daily Rituals read in this YouTube video that refers to Hoffman's popular biography The Man Who Loved Only Numbers, which is featured in Hoffman's own video.
Despite Hoffman's title, Erdős's interests were not confined to number theory but covered also combinatorics and graph theory (where he pioneered probabilistic methods), and analysis (approximation of functions), and he made sporadic contributions to other areas, such as general topology (an example of a totally disconnected space that is not zero-dimensional). AMS Notices surveyed Erdős's mathematics in some detail in 1998, 2 years after his death.
Here is Currey's excerpt:
"He was also, as Paul Hoffman documents in
his book The Man Who Loved Only Numbers, a true eccentric — a
“mathematical monk” who lived out of a pair of suitcases, dressed in
tattered suits, and gave away almost all the money he earned,
keeping just enough to sustain his meager lifestyle... Erdős liked to work in short, intense collaborations with other
mathematicians, and he crisscrossed the globe seeking fresh talent,
often camping out in colleagues’ homes while they worked on a
problem together. One such colleague remembered an Erdős visit
from the 1970s:
"…he only needed three hours of sleep. He’d get up early
and write letters, mathematical letters. He’d sleep downstairs.
The first time he stayed, the clock was set wrong. It said 7:00,
but it was really 4:30 A.M. He thought we should be up
working, so he turned on the TV full blast. Later, when he
knew me better, he’d come up at some early hour and tap on
the bedroom door. “Ralph, do you exist?” The pace was
grueling. He’d want to work from 8:00 A.M. until 1:30 A.M.
Sure we’d break for short meals but we’d write on napkins
and talk math the whole time. He’d stay a week or two and
you’d collapse at the end."
The colleague is Ralph Faudree, and the quote is taken from Hoffman's book, p.256.