It is hard to say why history happened one way rather than another. But we can point at some differences between ancient Greece, Hellenistic Mediterranean or Renaissance Europe and ancient and medieval China that arguably contributed not only to the persistence of flat Earth, but to the cultural slow motion generally, see Is the Scientific Method uniquely Western? for a broader theme. The former were a motley of city-states or countries with high levels of diversity, mobility and cultural exchange, while China was a much more culturally uniform traditional society that favored isolationism. Traditional societies effect change differently than dynamic ones, their tools and procedures improve through accumulation of incremental glacial changes, generational trial and error, rather than through intentional innovation guided by scientific inquiry. Here is Stein's characterization of Babylonian mathematics, that applies as well to Chinese astronomy:
"...for the Babylonians, mathematics was not an enterprise, but a lore and a skill possessed by the priestly or scribal class, for use in essentially administrative functions, and passed on from generation to generation much as were the techniques of the craft guilds in the middle ages. Of course, the techniques had to have been discovered; and innovations might occur from time to time; but invention or discovery was no more the business of those trained in the lore, than it was the business of the medieval master of a craft."
In recent decades there has been some progress in evolutionary modeling of cultural development, see. e.g. Is human cultural evolution Darwinian? by Mesoudi et al. (2004), although the approach is still controversial. From this perspective traditional societies suppress random drift, and do not provide enough opportunities for cultural "mutations" (so-called "memes") to spread and gain a foothold even when they do arise. This is how endemic populations are maintained in stable isolated environments.
Flat square Earth surrounded by round heaven was the traditional belief integrated with other mythical beliefs, the language used to think about the heavens was mythical and poetic, not geometric or precise. Zhang Heng (78-139 AD) writes:
"The heavens are like a hen's egg and as round as a crossbow bullet; the earth is like the yolk of the egg, and lies in the centre... Heaven takes its body from the Yang, so it is round and in motion. Earth takes its body from the Yin, so it is flat and quiescent". [in case you thought that the "yoke" of Earth is spherical]
Was there a compelling reason to change this belief? It depends on one's view of "compelling". And we often forget how the same observations can be interpreted in many different ways, and reconciled with just about anything. Essentially the same measurements that Eratosthenes used to estimate the Earth's circumference Chinese astronomers who wrote Huainanzi in the 2nd century BC interpreted to calculate the height of the Sun above the earth. This is just geometric inversion. What about the shadow of the Earth on the Moon during the eclipses, which is one of Aristotle's main arguments for spherical Earth? Eclipses are rare events of little practical significance, and in a traditional society would not be valued as highly as in the cosmopolitan Hellenistic world. But for what it is worth some Chinese astronomers, like Li Ye in 13th century, Jamal-ad-Din's contemporary in Kublai Khan's time, did challenge the idea that the Earth was square, he thought it was still flat but round like the sky.
There was one earlier exception that confirms the rule. China underwent a remarkable period of cultural innovation during the Song dynasty (960–1279), sometimes compared to European Renaissance. The invention of gunpowder and widespread use of printing press and paper money in China date to this period. Shen Kuo (1031–1095), a polymath often compared to Leonardo, among other things invented nautical compass, measured the distance between the North star and the true north, improved designs of the armillary sphere, and, most importantly for us, wrote on solar and lunar eclipses explaining how phases indicated that the Sun and the Moon were spherical rather than flat like fans. The Sun and the Moon, yes, but not the Earth, see Did the ancient Chinese know the earth is a sphere? Close, but Song China did not develop systematic theoretical and empirical science just as Shen Kuo stopped short of spherical Earth.
As to why the Jesuits had more of an effect than Jamal ad-Din that is relatively easy to explain. Kublai Khan let him set up an Islamic Astronomical Bureau in Beijing, rival to the traditional Chinese bureau, and had his subordinates check calculations of the Chinese. This maintained the traditional bureaucracy but hardly inspired the spirit of scientific cooperation. And the surrounding traditional community would not have been affected much even if the Chinese bureau did change its mind. Jesuits, on the other hand, were preaching directly to the people.
A good source on historical interaction of China with Western science is Elman's On Their Own Terms: Science in China, 1550-1900.
