Explicit applications of group representation to physics start with
E. Noether, Invariante Variationsprobleme, Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen. Mathematisch-Physikalische Klasse. 1918: 235–257.
where the celebrated Noether's Theorem was proved.
Applications to quantum mechanics were systematically developed by H. Weyl and E. Wigner:
The theory of groups in quantum mechanics, first edition 1928
Group theory and its application to the quantum mechanics of atomic spectra,
first edition 1931.
There are many good modern books with "group theory" and "physics" or "quantum mechanics" in the title,
my favorite is S. Sternberg, Group theory and physics, Cambridge, 1994.
Remark. The word "explicit" in the beginning of my answer is important.
Considerations based on symmetry are as old as physics itself: they were used already by Aristotle. A major 19th century application was crystallography.