I seem to remember that as late as in the XIX century there was a prominent mathematician who denied that the theory of probability was part of mathematics, since it does not deal with certainty. Do you know who was that mathematician?
It was Hardy and Littlewood. They said in a footnote to an article in the early 1920s (this was notably before Kolmogorov's measure-theoretic foundations were developed) that "Probability is not a notion of pure mathematics, but of philosophy or physics." See p. 37 of their paper Some Problems of Partitio Numerorum III.