Why deep learning infrastructure may be the next trillion-dollar bottleneck, and who's racing to fix it

For the past two years, the public conversation around artificial intelligence has fixated on a single metric: parameter count. Bigger models, better benchmarks, another leap in reasoning. But behind the launch announcements, a different arms race is unfolding, one fought not in code but in concrete, copper, and cooling towers.

The problem is straightforward. The compute required to train frontier models has been doubling roughly every six months since 2018, while the efficiency gains in transistor density and chip performance have slowed to a crawl. Moore's Law, the observation that the number of transistors on a chip doubles about every two years, has effectively decelerated. The result is an unprecedented hunger for power, space, and thermal management that the data-center industry was never designed to satisfy.

A perfect storm of demand and physics

Training a single large language model like GPT-4 or Gemini Ultra can consume tens of megawatts for weeks on end, a power draw equivalent to a small town. Each query served by a deployed model adds a non-trivial energy cost that, multiplied by billions of daily requests, turns inference into a significant line item on the balance sheets of any company operating at scale.

The bottleneck is not limited to power generation alone. Data centers were traditionally built to run workloads that could be scheduled, paused, and queued, not the unrelenting, high-density thermal loads that modern AI accelerators produce. An Nvidia H100 GPU, for example, can draw up to 700 watts under load, and racks packed with dozens of these chips generate heat densities that overwhelm conventional air-cooling systems.

Liquid cooling, once a niche solution for supercomputing labs, has moved from exotic to essential. Several major colocation providers, including Equinix and Digital Realty, have begun retrofitting facilities with direct-to-chip and immersion cooling technologies. The shift is expensive and slow, but it is already reshaping where and how the cloud's largest customers choose to build.

The hyperscaler response, and the startup opportunity

The hyperscalers, Amazon Web Services, Microsoft Azure, and Google Cloud, each operate their own playbook, but the direction is the same: build bigger, build differently, and build closer to available power. Amazon has announced plans to spend over $150 billion on data-center infrastructure over the next decade. Microsoft has committed to doubling its global data-center capacity roughly every two years, with a particular emphasis on locations in Ireland, Sweden, and other regions where renewable energy is abundant.

Yet the hyperscalers alone cannot solve the infrastructure gap at the speed the market demands. A new cohort of startups is betting that the next trillion-dollar market lies not in building bigger models, but in reimagining the physical substrate on which AI runs. Companies like Crusoe Energy, which deploys modular data centers at stranded gas wells to capture otherwise-flared natural gas as a power source, are pioneering "behind-the-meter" computing. Others like CoreWeave began as a cryptocurrency mining operation and pivoted to become one of the largest providers of GPU-accelerated cloud compute, leasing Nvidia hardware at a scale that rivals the hyperscalers.

The cooling question

Cooling accounts for roughly 30 to 40 percent of a data center's operating budget today. As thermal loads rise, that figure is expected to climb, unless new approaches change the equation. Immersion cooling, submerging servers in non-conductive dielectric fluid, can eliminate fans and drastically reduce energy spent on heat rejection. Several providers, including Submer and LiquidStack, now sell pre-built immersion systems that reduce cooling power consumption by up to 90 percent compared to traditional air cooling.

But the transition carries a capital cost. A facility retrofitted for immersion cooling requires new floor layout, fluid handling systems, and maintenance procedures that diverge from decades of operational practice. Operators face a choice: absorb the upfront expense and prepare for the future, or risk being stranded with assets that cannot support the next generation of accelerators.

Geopolitical dimensions

The infrastructure race is inseparable from geopolitics. Export controls on advanced semiconductors, particularly Nvidia's H100 and forthcoming B100 chips, have forced some countries and regions to accelerate their own domestic chip and data-center initiatives. The European Union, through its Chips Act and the EuroHPC Joint Undertaking, is subsidizing the construction of AI-optimized supercomputers that can serve member states without relying on US supply chains. China, meanwhile, continues to invest heavily in domestic chip design and data-center buildout despite the export restrictions.

The result is a fragmented global infrastructure map where power and physics, not just market demand, are deciding where the computing capacity of the next decade will reside.

The bottom line

The AI industry's most celebrated advances, from GPT-4 to Claude 3 to Gemini, rest on a foundation of physical plant that is being stretched to its limits. The companies that solve the infrastructure puzzle, whether through new cooling technologies, alternative energy sourcing, or more efficient chip designs, will hold an advantage that goes beyond any single model release. The race for the next frontier of AI is being built not just in the lab, but in the data center, one watt at a time.