I know this might be a silly question, but I think it's worth asking. For his undergraduate degree at MIT, did he do a double major with math, or did he really just get that good at mathematics by himself?
Read the Feynman interview at aip.org/history/ohilist/5020_2.html. Pretty early in the interview he says that he started as a math major, but it was getting abstract so he asked one of the professors in the MIT math department what abstract math was good for besides teaching it. He did not get a satisfactory answer, so he switched to electrical engineering and later moved into physics. He only got a physics degree in college.
To save you the bother, here's the relevant excerpt from the link posted in the other answer:
Weiner: Were you majoring in mathematics?
Feynman: Oh, yes. That’s interesting. At first I was in the mathematics course. It doesn’t make any difference what course you are in, really, very much. For the first year or so you take more or less the same thing. You take physics and chemistry and electrical engineering, mathematics, and so on. English. Somewhere around the first year I began to get upset. This wasn’t right. The mathematics, I looked at it, the mathematical things, were too abstract. They weren’t connected to anything, mathematics. And I went to the head of the mathematics department. This was in 1936, now, so you know it’s still in the Depression. I said to him, “Sir, what is the use of higher mathematics besides teaching more higher mathematics.” So he said, “Well, you could become an actuary,” calculating the insurance rates for an insurance company. This didn’t sit well with me, see. He also said that a man who asked that kind of a question is perhaps not right for mathematics. And I thought the thing I ought to do — I mean, I liked to get my hands dirty. I’d had a laboratory. The physical world was real, and the mathematics, I had become enthralled with, but not for itself, really — you know what I mean? It was fascinating, but my real heart was somewhere else. So I decided, I have to get my hands dirty, I can’t stand these abstract things. So I changed to electrical engineering, because there was something that was real. But then some few months later, I realized I’d gone too far, and that somewhere in between — that physics was the right place. So I moved around a little bit at the beginning, and ended up with the physics course.