(1) Weapon-grade uranium is hard to make - it took almost a year to separate 64 kg of weapon-grade uranium for the bomb in the Y-12 Oak Ridge facility, so "Little boy" bomb was not even tested (they had no spare). And because of the "Little boy" bomb design, most of that expensive uranium was NOT involved in the nuclear explosion (bomb exploded into sub-critical mass before most of the uranium was involved in the chain reaction).
How hard is to make the weapon-grade uranium: in Y-12 complex in Oak Ridge electromagnetic isotope separation, they used 12 300 tonnes (1 tonne = 1000 kg = 2204 lb, $300 million worth) of silver (copper as strategic material was not available in necessary quantities) converted to wires for the coils. After manufacturing the wires, facility was burned down to extract spilled silver from the ashes. Less than 0.036% was lost. As you can see, Y-12 is extremely complex process, resulting in building 1 bomb per year.
(2) But to have more feasible nuclear bomb ("Fat Man"), you need plutonium, which has to be first isolated (in 1940) and researched, then, industrial amounts of sufficiently pure specific isotope of plutonium (Pu-239, because not all isotopes are equally good for chain reaction - one of the problems they need to figure out and to overcome) has to be manufactured in a breeder nuclear reactor, which is also an easy to bomb target (and extremely expensive to build).
(3) Shaped explosion: Scientist were well aware of the inefficiency of the "Little boy" design, therefore they used shaped explosion to create superctitical mass of 11 kg of Plutonium - which has to be created from industry-grade (not weapon-grade) uranium in a breeder reactors, so it is much cheaper and faster process (it took about 1 month to breed enough Plutonium for 1 bomb). But you need to build also breeder reactors to manufacture the plutonium.
Because scientists were not sure if plutonium bomb would even work (decision could not be made based on the minute amount of the plutonium isotopes available), Manhattan Projects decided to invest in both approaches to save time and halve the risk.
(4) Calculating explosion-shaped charge used in Fat Man design is substantially more complex design (as compared to just two pieces of metal hitting each other as in Little boy): requires precise synchronization of multiple explosions (to shape explosion inward: hundred nanoseconds difference can make it or break it), so mathematical models of such explosions had to be developed, and calculated (by hand), therefore it was tested - Trinity - and they could not use computer simulation for explosion because computers were not invented yet, all calculation were made by hand. And wiring to deliver signals to start shaped explosions needed to be invented and tested.
(5) Bomb delivery is another big problem: Developing B-29 "strategic" bomber was even more expensive than the Manhattan project. It was about twice heavier than "standard" heavy bomber, B-17 and so substantially more complex. Getting enough power from just 4 engines, and fully pressurized cabin (to be able of long flights in high altitude, where enemy fighters cannot reach it), was especially hard.
Unexpected part that it at the end, nuclear bombing not only saved lives (it prevented expected 500K US casualties and tens of millions expected Japanese casualties during the planned invasion Operation Downfall - bigger than D-Day), but also saved money: because it shortened war by several months, war cost were about 1 Billion per week, and cost of B-29 and Manhattan Project were in total only about $4B.