What motivated physicists to abandon the idea of anti-particles having a negative mass?
Nothing. They never took the idea seriously. Don't forget that in his 1905 E=mc² paper Einstein said “the mass of a body is a measure of its energy-content”. You can remove energy from a thing just as you can shorten a pencil. However when you remove all the energy from a thing it just isn’t there any more, just as a pencil shortened to 0cm isn’t there any more. You cannot then remove more energy to create a negative-energy particle, just as you cannot shorten a pencil to less than 0cm long.
Anyway, what we're really talking about here is Dirac’s 1930 negative-energy “hole theory” paper a theory of electrons and protons. Oppenheimer shot it down straight away in his 1930 paper on the theory of electrons and protons. He said Dirac’s derivation of the Thomson formula was invalid, and there are several grave difficulties which arise when “one tries to maintain the suggestion that protons are the gaps of negative energy, and that there are no distinctive particles of positive charge”.
Dirac then wrote quantised singularities in the electromagnetic field. This is the paper where he’s said to have predicted the positron. But what he was really doing was moving the goalposts is response to Oppenheimer’s paper. Graham Farmelo talks about it in his 2010 article did Dirac predict the positron? He says Dirac’s “close friend Patrick Blackett, one of the leading players in the story’s denouement, denied it”. And that “very few physicists took Dirac’s hole theory seriously”. He also says “Victor Weisskopf later recalled the idea ‘seemed incredible and unnatural to everybody’”. I would say it's been glamourised since then, without enough attention to the historical facts.