For years, one fleet of accelerators has handled both halves of an AI model’s life. It trains the model for weeks, then answers user requests once the model ships. Google now argues those two jobs need different silicon, and its google tpu 8i chip is built for the second one.
Disclosure: Cloudfresh, a Google Cloud partner, offers Google Cloud Consulting Services and ran the customer project described in this article. Its consultant is quoted below.
At Google Cloud Next in April 2026, Google announced the eighth generation of its Tensor Processing Units, or TPUs, the custom chips it designs for machine learning. For the first time, that generation includes two chips: TPU 8t for training and TPU 8i for inference, the stage where a trained model responds to requests. Google says it saw inference demand rising several years ago, as frontier models moved into production, and concluded that agents would benefit from chips built separately for each job.

Why agents strain hardware built for training
Training rewards raw throughput. The faster a chip pushes data through a model, the sooner the model is ready. Agents put a different kind of load on AI accelerator chips, and Google describes them as systems that reason through problems, run multistep workflows, and learn from their own actions in loops.
Each step waits on the one before it. That makes memory access and latency nearly as important as compute. The google tpu 8i chip pairs 288 GB of high-bandwidth memory with 384 MB of on-chip SRAM, a fast memory built into the chip itself. Google says that SRAM is three times the amount in the previous generation, and the goal is to keep an agent’s active working set on the chip.

Google argues that interactions between agents at scale amplify even small inefficiencies. Both chips can run either workload, the company says, but specialization is where it expects the efficiency gains to come from.
If agents are moving toward dedicated hardware, the economics get more personal too. $499 Linux device built for AI agents shows what agent-focused hardware looks like on a desk.

What the google tpu 8i chip does to cost per unit of work
Google attaches numbers to the gain. It says the google tpu 8i chip delivers 80 percent better performance per dollar than the previous generation, enough to serve nearly twice the customer volume at the same cost. Both new chips reach up to twice the performance per watt of Ironwood.
Power matters here for a practical reason. Google says power has become a binding constraint in data centers, alongside chip supply.

For FinOps teams, the practice of managing cloud spending across finance and engineering, the change lies in what gets measured. A single blended line for accelerator hours hides two different cost curves. Training spend behaves like a project with a start and an end. Serving spend rises with every request an agent handles. Splitting the budget into two lines, and tracking serving cost per completed task, shows which agents justify their compute.
The software layer and model choice
An agent is only as efficient as the software that drives it, and Google has reworked that layer too. Vertex AI is now the Gemini Enterprise Agent Platform. Its Agent Runtime supports long-running agents that hold state for days.

Agent Evaluation and Agent Observability trace an agent’s reasoning step by step in production. Those traces reveal where an agent loops, retries, or calls a larger model than the task requires. That is where serving costs quietly build up.
Model selection is the second lever. Model Garden offers more than 200 models, including Gemini 3.1 Pro, the open Gemma 4, and Anthropic’s Claude models. Each step in a pipeline can run on a model sized for that step.

Google’s model push also raises provenance questions, and its detection tools are widening. Google’s SynthID Detector Access Expands Worldwide explains who can now check whether media was made by AI.
Oleh Maksymovych, co-founder and general manager at Cloudfresh, puts the sequence plainly: “Once an agent is in production, the question shifts from how to train a model to what each answer costs. That’s why we start with the workload: which tasks need a large model, which run well on a smaller one, and how often each is called. The hardware decision comes after that.”

Cloudfresh’s client project shows what that sequence can deliver. A product company with more than 80 million users across dating and women’s mental health apps first built AI features on open Gemma models running on rented GPUs. It later added external AI providers. As use cases multiplied, the team needed more stability, scalability, and control.
After the move to Gemini, the team assigned specific models to specific tasks and tuned request costs to production load. Profile review coverage rose from 17 percent to 100 percent. According to the product company, costs and resource demands did not rise. About 80 percent of profiles now get approved automatically in under five seconds.

Custom silicon raises lock-in concerns. Google lists native support for JAX, MaxText, PyTorch, SGLang, and vLLM on both TPU 8 platforms. Teams that already serve models with those frameworks can test the google tpu 8i chip using the tools they have.
Building a cost map before buying hardware
Start with an inventory rather than a purchase order. Split accelerator hours into two buckets: hours spent building models and hours spent answering live traffic. The split may be rough at first, but it gives the team a baseline.

Next, tie each dollar of serving spend to the agent and task that generated it. A cost-per-task figure shows which workflows earn their compute and which drain it without delivering much value.
Then review the model each task calls. A step that runs a large model where a smaller one would produce the same output is the first place to look. Once that map exists, deciding whether an agent should move to the google tpu 8i chip becomes a calculation based on the team’s own usage numbers.

The TPU split is a hardware story, but the bill is really a story about answers. Teams that can name what each answer costs will make the chip decision on their own evidence, not on a vendor’s benchmark.

H/T designboom.com
Frequently asked questions
What is the difference between Google’s TPU 8t and TPU 8i?
TPU 8t is built for training AI models, while TPU 8i is built for inference, the stage where a trained model responds to requests. Google announced both as part of its eighth generation of Tensor Processing Units at Google Cloud Next in April 2026.
Why does Google say agents need different chip hardware?
Google describes agents as systems that reason through multistep workflows, where each step waits on the one before it, so memory access and latency matter nearly as much as compute. The TPU 8i pairs 288 GB of high-bandwidth memory with 384 MB of on-chip SRAM, three times the previous generation’s amount, to keep an agent’s active working set on the chip.
How much better is the TPU 8i’s performance per dollar?
Google says the TPU 8i delivers 80 percent better performance per dollar than the previous generation. According to Google, that is enough to serve nearly twice the customer volume at the same cost.
How should FinOps teams track spending on AI serving costs?
The article suggests splitting the accelerator budget into two lines, since training behaves like a project with a start and an end while serving spend rises with every request. It also recommends tracking serving cost per completed task to show which agents justify their compute.
More in
Technology
All TechnologyMore like this

Playfool's The Server Turns Data Centre Vacuum into Lesson on Hidden Labour

SlashVibe Lets You Speak to Agents From a Detachable Wireless Controller

Stephen Voss Turns Blank Data Center Walls into Screens for Science Fiction Passages

Voters Curb Data Centres as Nvidia's Huang Quits, Leaving AI on Back Foot

Meta's Muse AI Agent Challenges Apple's Ecosystem: Bank of America Highlights Risks




