We have price on the vertical axis because that's how Alfred Marshall (1890) drew his graphs in Principles of Economics. For better or worse, Principles was hugely influential. And so the present-day convention is Marshall's convention. As Humphrey (1992) writes:
The Marshallian cross diagram bears Marshall's name because he gave it
its most complete, systematic, and persuasive statement, not because
he was the first to invent it. His account was definitive, not
pathbreaking. For this he received — and deserved — credit.
$$$$
The above answer is not completely satisfactory. It merely pushes the question back one level: Why did Marshall put price on the vertical axis?
Short answer: Marshall first introduced a demand curve in 1879. There, he thought of quantity as the independent variable, with prices adjusting to clear the market. It was thus perfectly logical for him to have quantity on the horizontal axis.
Long answer
Marshall (1879) was not the first to draw demand or supply curves. According to Humphrey, we have:
- Augustin Cournot (1838). Price on horizontal axis.
- Karl Rau (1841). Price on vertical axis.
- Jules Dupuit (1844). Price on horizontal axis.
- Hans von Mangoldt (1863). Price on vertical axis.
- Fleeming Jenkin (1870). Price on horizontal axis.
I'd also add
- William Stanley Jevons (1871, figure). Price on vertical axis.
Did any of the above influence Marshall?
Prior to his 1879 publication, Marshall may not have been aware of Rau or Mangoldt's work, who were less well-known. But Marshall was aware of Cournot, Dupuit, Jenkin, and Jevons's work (see e.g. Whitaker, 1975). But of these four, only Jevons had price on the vertical axis. Cournot, Dupuit, and Jenkin all had price on the horizontal axis.
Moreover, according to the man himself, Marshall's main influence was Cournot:
following the lead of Cournot I had anticipated all the central points of Jevons book and had in many respects gone beyond him,
my obligations are solely to Cournot; not to Fleeming Jenkin or Dupuit. (Quoted in Whitaker.)
But Cournot had price on the horizontal axis! I can find in Marshall's writings no explicit explanation for why he chose to deviate from Cournot, who was his greatest influence (at least in this matter).
However, when introducing for his very first demand curve (1879), he does write the following:
we may draw what may be called "the Demand curve," thus: Let $M$ be
any point on $Ox$ (fig. 20), and let the price at which it is possible to
dispose of $OM_1$ coals annually be estimated and found to be equal to $ON_1$. 
That is, Marshall first takes the independent variable to be the quantity demanded (the $OM_1$ coals to be disposed of). Marshall's demand curve then tells us about the dependent variable, i.e. the price at which such coal can be disposed.
I thus conclude that Marshall thought of quantity as the independent variable, with prices adjusting to clear the market. It was thus perfectly sensible for him to put quantity on the horizontal axis and price on the vertical.
This way of thinking seems slightly unnatural to me (and many others). But perhaps this is only because we've been trained, since our first economics classes, to think of price as the independent variable.
Moreover, as some here have pointed out, it's not really all that absurd to think of quantity as the independent variable, as Marshall did.
$$$$
P.S. There's also another cynical explanation I once came across: Marshall simply wanted to differentiate himself from Cournot et al. So he deliberately switched the axes. In my opinion this can plausibly serve as part of the explanation. After all, Marshall felt that he had done a lot of his own original work and was somewhat peeved that others, such as Jenkin, had anticipated his work (see e.g. Marshall's reaction to Jenkin's 1870 article, quoted in Whitaker, 1975).
$$$$
Supply and Demand Diagrams before Marshall (1879)
