Kelvin did think that "uniform density with a fudge factor" was good enough. The more detailed considerations of the density profile were made by T.J.J. See in 1899, where he used the "Lane's law of temperature" to study central condensation without uniformity assumptions, see his informal self-summary in The Atlantic. But this only increased the estimate to 32,000,000 years, in agreement with Kelvin's earlier ballparking.
Even before that, Kelvin was reassured, to put it mildly, when his and others' estimates of the Earth's age, based on different considerations, produced a comparable figure of 24,000,000. As far as Kelvin was concerned, geologists had to yield, see Cajori, The Age of the Sun and the Earth for a good historical account:
"Lord Kelvin's address of 1897 is permeated, as Prof. Chamberlin puts it, “with an air of retrospective triumph and a tone of prophetic assurance.” “It is only by sheer force of reason,” says Kelvin, “that geologists have been compelled to think otherwise, and to see that there was a definite beginning and to look forward to a definite end of this world as an abode fitted for life.” Nor was this feeling of retrospective triumph confined to Lord Kelvin or to the students of the problem of the age of the sun and earth. At the close of the century physicists and chemists gloried in the triumphs of their predecessors..."
In hindsight, it was "retrospective triumph" of the same nature as Kelvin's two clouds or Michelson's ode to the aether a bit later. 'Amazingly', geologists were not impressed, as Geikie told the British Association back in 1892. Now that was prophetic:
"Lord Kelvin is willing, I believe, to grant us some twenty millions of years, but Prof. Tait would have us content with less than ten millions... That there must be some flaw in the physical argument I can, for mypart, hardly doubt, though I do not pretend to be able to say where it is to be found. Some assumption, it seems to me, has been made, or some consideration has been left out of sight, which will eventually be seen to vitiate the conclusions, and which when duly taken into account will allow time enough for any reasonable interpretation of the geological record."
By the way, Kelvin's, or rather Helmholtz's, theory was not exactly ordinary gravitational collapse (as it would require unobserved changes in the Sun's diameter), but rather what he called "meteoric theory": "coalition of smaller bodies, falling together by mutual gravitation, and generating... an exact equivalent of heat for the motion lost in collision."
Kelvin was well aware of the role of the density profile, but was content with supposing that Helmholtz's 1854 estimate of 20,000,000 based on the ultimate uniform density is roughly in the ballpark by the order of magnitude. He first ballparked possible deviation in On the Age of the Sun’s Heat (1862), and then came back to it in The Sun's Heat (1887) lecture to the Royal Institution of Great Britain. But his view of the density profile's minor influence has not changed:
"In all our calculations hitherto we have for simplicity taken the density as uniform throughout, and equal to the true mean density of the sun, being about 1.4 times the density of water, or about a quarter of the earth's mean density. In reality the density in the upper parts of the sun's mass must be something less than this, and something considerably more than this in the central parts, because of the pressure in the interior increasing to something enormously great at the centre.
If we knew the distribution of interior density we could easily modify our calculations accordingly; but it does not seem probable that the correction could, with any probable assumption as to the greatness of the density throughout a considerable proportion of the sun's interior, add more than a few million years to the past of solar heat, and what could be added to the past must be taken from the future."
