I would like to know some references (if any) for the claim that Grothendieck didn't like the idea of elementary topos.
The main evidence of this seems to be that whenever Grothendieck says topos, he means Grothendieck topos (unless someone can find a contradicting reference). However, it doesn't seem to be the case that he didn't like elementary toposes, so much as he didn't think he needed them for what he was doing.
According to page 6 of this interview with Lawvere http://www.mat.uc.pt/~picado/lawvere/interview.pdf, Grothendieck did call Lawvere the "main contradictor", but this was with reference to politics rather than mathematics. Grothendieck does use the subobject classifier in a topos in Pursuing Stacks, and even calls it the "Lawvere element" of a topos.