We often forget that even the minute is not an SI unit, only the second and its decimal multiples and fractions are. It is a leftover of the sexagesimal system (base 60), whose use predates the decimals by many centuries, and goes back to the ancient Babylon. So are the angular degrees.
Heinrich Hertz was born in 1857, and the International Electrotechnical ...
I looked in the Oxford English Dictionary (subscription required) for English usages.
physics used in the medical sense, I do not report on.
physics used for "natural science" in general: The oldest instance they have is from 1487.
J. Skelton tr. Diodorus Siculus Bibliotheca Historica iii. 174 Among the Grekes, as it is many tymes seen, it is ...
Some references :
consider that Lord Kelvin published is well-known textbook in 1867 with the title Treatise on Natural Philosophy.
J.C.Maxwell begins his Matter and Motion (1876) with "Physical Science…".
In K.Pearson's The Grammar of Science (1892), the terms : physics and science/scientist, are definitely there.