When it comes to powering AI infrastructure, NVIDIA is the undisputed leader right now. While the market expected AMD and others to challenge NVIDIA’s dominance, Google and AWS surprised many by consistently rolling out their own AI chips. These chips are now powering not only their own products and services but also third-party customers, including Anthropic, one of the fastest-growing AI startups in the world.
At Google Cloud Next today, Google announced two new eighth-generation TPUs: TPU 8t and TPU 8i. Both of these TPUs are purpose-built architectures for training and inference, respectively. With these new chips, Google is positioning its latest AI infrastructure as a serious alternative to NVIDIA for both frontier model development and production AI services.
Google claims that the new TPU 8t is designed for training frontier models faster by cutting development cycles from months to weeks, offering nearly 3x more compute performance per pod than the previous generation. The company claims that a single TPU 8t superpod can now scale to 9,600 chips and 2 petabytes of shared high-bandwidth memory to offer 121 exaflops of compute, doubling the interchip bandwidth compared to the previous generation.
Google also highlighted that TPU 8t comes with 10x faster storage access and support for TPUDirect, which allows data to be pulled directly into the TPU. The company claims that its new Virgo Network, along with JAX and Pathways, enables near-linear scaling for up to one million chips in a single logical cluster.
The new TPU 8i is built for low-latency, reasoning-heavy inference workloads and can deliver 80% better performance-per-dollar than the previous generation. The TPU 8i chip combines 288GB of high-bandwidth memory with 384MB of on-chip SRAM to keep a model’s active working set closer to the processor.
Google is also ditching x86 CPUs for its own custom Axion Arm-based CPUs for better overall efficiency. For modern Mixture of Experts (MoE) models, Google has doubled interconnect bandwidth to 19.2Tb/s, and the new Boardfly architecture reduces maximum network diameter by more than 50% to reduce system latency. The new on-chip Collectives Acceleration Engine (CAE) offloads global operations and reduces on-chip latency by up to 5x, according to Google.
Both TPU 8t and TPU 8i chips will be generally available later this year.
