Surely, what I refer below it is not one of the earliest prizes, but it is important in the history of mathematics.$^1$
There is a famous story about heat diffusion, at the beginning of 19th century, that involved J. B. Fourier and his formulation of trigonometric series, and a prize which Fourier won, but with reserve and critics. The rather troubled story of this prize is intertwined with the history of the development of Fourier’s series and heat theory.
There was, at the beginning of the 19th century, in France, a sound interest, theorical and practical, about the subject of heat nature and diffusion, on which had worked mathematicians and physicists as Lagrange and Lavoisier. During these years, Fourier began to deal with the subject in an original manner.
His most important book, La Théorie Analytique de la Chaleur, published in 1822, in which Fourier presented his theory, was written in several stages.
It was preceded by a memoir that Fourier submitted to the Paris Academy of Sciences in 1807, but its publication was rejected because of the opposition of Lagrange, who considered not rigorous the Fourier's treatment of trigonometric series.
Fourier answered to the objections of Lagrange in an article, sent to Lagrange, and in another article sent to the Academia, in 1808. But his 1807 memoir remained unpublished, and has been published only recently.$^2$
But the question of heat diffusion remained open and, in 1810, the Academy of Sciences announced a prize competition on heat diffusion. Fourier submitted a revised version of his memoir, and won the competition, but was again criticized for lack of rigour and generality, as can be read in the report of the judges of the competition:
This work contains the true differential equations of heat diffusion
[...] the novelty of the subject, together with its importance,
convinced the Class of Mathematical, Physical and Natural Sciences of
the Academy to award the prize to this work,[..] but observing that it
isn’t without difficulties and that its analysis, to integrate them,
is still unsatisfactory, with regard to generality and also with regard to
rigour.$^3$
For this reason, probably, this memoir was not published until 1824, when Fourier had become secretary of the Academy of Sciences.
The Théorie Analytique de la Chaleur had been published in 1822, after Lagrange’s death and when Fourier’s reputation among the Academy of Sciences was rising.
$^1$ I don’t need to tell that Fourier’s work about heat diffusion and trigonometric series are one of the most important achievements in the first half of 19th analysis, and beyond: think that also Cantor’s set theory came from Cantor’s studies about Fourier’s series.
$^2$ In Grattan Guinnnes I. and Ravetz J.R., Joseph Fourier, 1768-1830, Cambridge, Mass., 1972.
$^3$ Quoted in U. Bottazzini, Il calcolo sublime, Boringhieri, 1981. p. 60 (en. transl. Bottazzini, U, The Higher Calculus: A History of Real and Complex Analysis from Euler to Weierstrass, Springer/Verlag, 2012.