It is natural for physicists to consider the group $SO(3)$. Presumably, $SU(2)$ came into physics because of quantum mechanics. How did people realize that when studying rotation of a physical system, sometimes it is the group $SU(2)$ instead of the group $SO(3)$ that really matters?
It came to physics a bit earlier than quantum mechanics. The homomorphism $SU(2)\to SO(3)$ was discovered by Cayley (1843), Hamilton (1847), and Klein (1875) in their pure mathematical studies, and came to the attention of physicists through the theory of rigid body rotation (classical mechanics). It was Klein who brought it to the attention of physicists.
In old books on mechanics this is called the "Cayley-Klein's parameters". Klein explained this correspondence to physicists in his 1895/6 lectures which were later published as a book under the title "Theory of the top" (joint with his student A. Sommerfeld). (A concise exposition in English is in his 1897 Princeton lectures on the same subject. But the 4 volumes of Klein and Sommerfeld has been also translated into English).
In his Princeton lectures Klein says:
Instead of either of these commonly used systems of parameters, I propose to introduce another, which so far as I know has not yet been employed in dynamics.
Later he writes in the same lectures:
There is nothing essentially new in these considerations. I have merely attempted to throw a method already well known into the most convenient form for applications in mechanics.
Among the predecessors of Cayley, Hamilton, and Klein, Euler has to be mentioned: he discovered a rational parametrization of the rotation group which is equivalent to quaternion parametrization, in his work on arithmetic. It was apparently not noticed by physicists, but Klein credits the "quaternion parametrization" to Euler.