"I've browsed many math history books" -- I'm not sure what this means. And "I've never read too deep into any single one" is not any more encouraging. A book on the history of mathematics may have a focus: What is the history of the current mathematics -- that is, what led up to it? Maybe it will stop some decades or even a century before "now," whatever time "now" was when the book was written. Or maybe a book will be about the mathematical activity in human cultures throughout time and the world. A book may focus on a topic or certain topics; "calculus" is likely to be a somewhat popular choice in the current state of affairs. I have not seen one focused on an imperialistic point of view, that is, how empires, and lets broaden it to include trade, promulgated learning throughout an empire and its trading partners, and whether this was bad or good (no doubt a question with nuanced answers).
Of the first kind, which deals with the history of the mathematics we study in universities around the world, the books I've read have been classics: Morris Kline, Dirk Struik, Carl Boyer. In one way they're a bit dissatisfying: they more or less tell the same story. It would be worse, however, if they told vastly different stories. The main thread of these books traces the development of mathematics from some starting points to Europe from the Renaissance through the 19th century, at which point it begins to take root almost everywhere. The starting points are not in Europe, but in Egypt, Babylon, Persia, India, China, and Japan. At some point mathematics starts to take hold in Greece. From there, the next flourishing is in Alexandria and then to the Arabs. (Rome, while having a highly educated class, did not contribute much to mathematical research, it seemed.) From the Arabs, mathematics takes root again in Europe. China and Japan were somewhat isolated from Europe, and it seems the discoveries of each did not influence each other much. Or such is the broad arc of the story. If you're interested in how we arrived at the mathematics that people study through the first or second year an undergraduate program, this is it. The European flourishing brought modern versions of algebra, analysis, and statistics, which contain the principal subjects studied. Naturally, the people involved were European. I'd assume these Europeans were white, if ever I thought about it, but the races of the mathematicians were not mentioned (or I don't remember it). Most of them were male, which reflects inequities in the society at the time, not bias in the historian; women were mentioned by name, nonetheless, for the mere offense of having done some significant mathematics (a worthy reason, by the way, which the sarcasm is meant to highlight), but only very few were mentioned. I'm told that even "Men of Mathematics" features a female mathematician (one classic I have not read). I'd assume (again) that very few women had the opportunity to do significant mathematics, although there is some evidence that it could be done privately and not be published (be "silenced" if you prefer). It is important to recognize why only few women are mentioned -- and omitting to do so may reflect bias, or at least a blind spot, in the historian -- because historically, and even currently, the lesson drawn has been either that women are incapable or that they should not be allowed to do it, which is wrong. So wrong that it needs to be pointed out.
That "math is primarily the result of individual genius" is an impression one sometimes gets. Kline's book often shows how an idea evolves due to the work of several people, but I've usually preferred his book for its focus on ideas rather than on people, although it does contain biographical tidbits that the OP complains about. I think an in-depth history is a complicated business, and books with a broad scope like the ones I'm discussing do not take the space to go into the details. For a in-depth look at, say, Cauchy's contribution to single-variable calculus, see Judith Grabiner, The Origins of Cauchy's Rigorous Calculus. At some length, she details what Cauchy had to work with, how many of the ideas he put together had been thought either clearly or obscurely by others, and why what he did was significant and in what sense original. She also argues her thesis when it diverges from other historians', and an interested reader could follow up her references and try to sort it out. Alas, I don't believe any of the historical personages are female (perhaps I don't know everyone's gender) or people of color. I assume that is historically accurate and that she has not erased them from history. She does mention Algarotti's Newtonianism for the Ladies, so there is at least a nod to half the world's population. I think it shows how difficult it is to write about the history of an idea and satisfy a reader who wants it to be a history of disfranchised segments of the population.
If you want a glimpse at how mathematical ideas are built, some original books or papers explain what they are building on. For instance, Lagrange's Fonctions Analytiques or Riemann's Ueber die Darstellbarkeit.... The first few pages of each show what the author says he is responding to and building on.
If you want a different sort of focus, perhaps you would like Marcia Ascher's Ethnomathematics
or other books. Whether they are historical or anthropological, or what the distinction is, is open to debate (one I'm unqualified to engage in, btw), but they have a non-western focus.
