In theory, relativistic mass was preceded by the "electromagnetic mass" introduced by J.J. Thomson in 1881, and further developed by Heaviside (1888), Searle (1897), Poincaré (1900), Abraham (1902), Lorentz (1904).
https://en.wikipedia.org/wiki/Electromagnetic_mass
In practice, the first experiments precise enough to measure such effects were conducted by Kaufmann in 1901-1905, followed by experiments by Bucherer in 1908 and others, even though it took some time until their meaning was clarified.
https://en.wikipedia.org/wiki/Kaufmann%E2%80%93Bucherer%E2%80%93Neumann_experiments
After the advent of special relativity by Einstein in 1905, "electromagnetic mass" was superseded by the concept of "relativistic mass" first formulated by Lewis and Tolman in 1909. However, modern authors try to avoid the concept of relativistic mass, instead they use the concepts of "relativistic momentum" and "relativistic energy" in order to describe the experiments in particle accelerators, while "mass" is now simply denoted as "rest mass" or "invariant mass".
https://en.wikipedia.org/wiki/Mass_in_special_relativity#The_relativistic_mass_concept
Regarding the evolution of the mass energy equivalence: In terms of electromagnetism some elements of that concept were known to Poincaré (1900) and Hasenöhrl (1904), while the complete equivalence in terms of all forms of energy was first pointed out by Einstein in 1905.
https://en.wikipedia.org/wiki/Electromagnetic_mass#Inertia_of_energy_and_radiation_paradoxes
https://en.wikipedia.org/wiki/Electromagnetic_mass#Modern_view