In theory, relativistic mass was preceded by the "electromagnetic mass" introduced by J.J. Thomson in 1881, and further developed by Heaviside (1888), Searle (1897), Poincaré (1900), Abraham (1902), Lorentz (1904).
In practice, the first experiments precise enough to measure such effects were conducted by Kaufmann in 1901-1905, followed by experiments by Bucherer in 1908 and others, even though it took some time until their meaning was clarified.
After the advent of special relativity by Einstein in 1905, "electromagnetic mass" was superseded by the concept of "relativistic mass" first formulated by Lewis and Tolman in 1909. However, modern authors try to avoid the concept of relativistic mass, instead they use the concepts of "relativistic momentum" and "relativistic energy" in order to describe the experiments in particle accelerators, while "mass" is now simply denoted as "rest mass" or "invariant mass".
Regarding the evolution of the mass energy equivalence: In terms of electromagnetism some elements of that concept were known to Poincaré (1900) and Hasenöhrl (1904), while the complete equivalence in terms of all forms of energy was first pointed out by Einstein in 1905.