AI Agents Are Pushing CPUs Back Into the Data Center Spotlight: A MyTechDailyNews Explainer
Ask most people which chip powers artificial intelligence and they will answer GPUs. According to Mark Lohmeyer, vice president and general manager for AI and computing infrastructure at Google Cloud, that answer is already out of date. Agentic systems — AI that carries out multi-step tasks instead of only replying to prompts — are shifting a meaningful share of work back onto ordinary CPUs and, in doing so, reshaping how data centers are built. He laid out the case in an interview with EE Times, and this MyTechDailyNews explainer turns it into something everyday readers, product teams and market watchers can actually use.
Lohmeyer’s starting point is the difference between talking to a model and delegating to one. “Chat is relatively straightforward—there’s a question and an answer, and it relies heavily on inference across GPUs or TPUs,” he said. “With agents, the nature of those interactions is changing quite a bit.”
Chatbot vs. Agent: Same Request, Very Different Machinery
Under the hood, an agent deployment begins with a person’s high-level intent. That intent is divided into subtasks and handed to a group of agents that may call other APIs, visit websites, and write and run code — work that falls to CPUs. Sessions also stretch out: they can last much longer, involve multiple human users, and must keep context and state alive across days or weeks.
| What you are looking at | Conversational AI | Agentic AI |
|---|---|---|
| Unit of work | A question and an answer | A goal split into subtasks |
| Who acts | The model replies | Multiple agents call APIs, browse sites, run code |
| Silicon carrying the load | Largely GPUs and TPUs | More CPU work alongside accelerators |
| Session length | Short exchanges | Long sessions holding state for days or weeks |
| Activity rhythm | Prompt in, answer out | Bursty and often idle |
The Idle-Then-Frantic Rhythm That Breaks Old Assumptions
“Agents are often bursty, and often idle,” Lohmeyer said. “They don’t do anything for [say] a day or two until you come back to them, or maybe until some API responds… It’s unique from older CPU workloads, in that it needs to be stateful over a long period of time, but is often idle during that period of time, and then when it’s not idle, it’s highly bursty.”
Managing that stop-start pattern with both performance and security intact is, in his telling, one of the industry’s central challenges right now. “Demand is already very high and is growing rapidly,” he said. “So, how do we provide the infrastructure that can support that growth and do it cost-effectively?”
Google’s Arm-based Axion CPU
Two Halves of One Workload
Lohmeyer separates the compute into two pieces. The first is running the agent itself, a new workload that requires new infrastructure. The second is the familiar plumbing — website and API access — which already existed but previously interfaced with humans. “Agentic workloads will have a meaningful impact on both of those elements,” he said.
The familiar half expands fastest, because agents are far more demanding than humans and can be brought online in hundreds or thousands quickly, while human user counts cannot rise anywhere near that fast.
Distance Becomes a Design Decision
“Ideally, you’d like to have those agents and their context, the applications and data, relatively close to each other from a latency perspective, for performance reasons,” Lohmeyer said.
Locality matters, yet he expects enterprises to keep operating in a distributed world, with most still using multiple clouds and private clouds for now. That keeps high-performance networking important inside data centers and between them.
“Dynamic, stateful agents will need to have a new type of computing infrastructure that supports them,” he said. “They’re accessing all the existing apps and data, so the demand there is growing, and all this needs to be connected together through the network. Google is investing in all of these elements.”
How Much CPU, Exactly?
There is no settled recipe yet. The optimal CPU-to-GPU compute ratio for agentic AI keeps evolving with the workload, Lohmeyer said, but the share of CPU compute required is definitely increasing as the shift moves from training foundation models to agentic inference.
“We expect both [CPU and TPU/GPU] workloads to continue to grow rapidly, but the proportion of CPUs is going from a relatively small portion to a meaningful portion, and we think that will continue to evolve as agents expand into new use cases and new environments,” he said.
Co-Design From Silicon Up to Software
The answer to agentic inference, he argues, will keep depending on co-design across the stack — CPUs, storage, memory and networking tackled together rather than one part optimized in isolation.
“Google Cloud has always had a philosophy of workload-optimized infrastructure,” he said. “This is more important than ever in the agentic era.”
Because these workloads are bursty and often leave the CPU idle, agents can effectively be “hibernated”: their state is moved to storage so the CPU can serve other agents, then reloaded quickly when it is needed again.
“This requires a new type of orchestration in software, and in this environment, having really high-performance storage that’s attached to that server, so I can restart it quickly, is important,” he said.
Thousand Tiny Boxes on One Server
Google Cloud uses microVMs, described as a “slimmed down” version of a virtual machine, to create a secure boundary between agents packed onto the same hardware. Rather than 100 traditional VMs, the same machine might run 1,000 microVMs.
“Security and governance for agents is absolutely critical, and Google is investing a lot in every layer of the stack,” he said.
| Isolation choice | Typical count on the same hardware | Trade-off described |
|---|---|---|
| Traditional virtual machines | 100 | Full feature set, larger attack surface |
| MicroVMs | 1,000 | Streamlined feature set, better performance, smaller attack surface |
At the lowest layer, virtualization is key. Google plans to support multiple ways to sandbox agents, with the microVM particularly important. The company is also investing in agent gateways that can provide egress controls at the network level, along with other higher-level controls.
The Payoff Everyone Is Betting On
Looking ahead, running thousands of microVMs for agents will affect core density, storage, how storage is attached, and the orchestration software used to manage it all. Higher hardware utilization is the expected benefit, and that should lower the cost per agent.
“This will allow our customers to scale their business, but do so with a good business model, good profitability, but without having to sacrifice performance as they are scaling,” he said. “These capabilities will be very important going forward.”
Four Misconceptions Worth Correcting
- Agents replace the software you already run. They do not. They access existing apps and data, which is exactly why demand for those systems grows.
- AI growth only means buying more accelerators. As the industry moves from training foundation models to agentic inference, the CPU portion is climbing from relatively small to meaningful.
- An idle agent is wasted money. Hibernation shifts its state to storage and frees the CPU for other agents until it wakes.
- Ordinary virtual machines are isolation enough. MicroVMs trade features for better performance and a smaller attack surface.
Action Checklist: What to Watch Next
- Re-check the CPU side of your AI stack, not just the accelerator side; the balance between them is moving.
- Map how far agents sit from their context, applications and data — latency gaps show up as sluggish agent tasks.
- Plan for long-lived sessions that hold state across days or weeks, not single request-and-reply exchanges.
- Treat security and governance for agents as a first-order requirement: sandboxing, egress controls and higher-level policy.
- Follow utilization economics closely; better packing and faster restarts are the levers that decide cost per agent.
FAQ
Why would AI make CPUs matter more again?
Because much of what agents actually do — splitting intent into subtasks, calling APIs, visiting websites, writing and running code — is CPU work rather than accelerator work.
Are GPUs and TPUs becoming irrelevant?
No. Both CPU and TPU/GPU workloads are expected to keep growing rapidly; what changes is the proportion CPUs represent of the total.
What does “stateful” mean for a normal user?
It means an agent keeps its context alive even while nothing is happening, so it can pick up where it left off when you return or when an API finally responds.
What is a microVM in plain terms?
A slimmed down virtual machine used as a secure boundary between agents sharing a server — the kind of setup where 100 traditional VMs might become 1,000 microVMs.
Who is describing all of this?
Mark Lohmeyer, vice president and general manager for AI and computing infrastructure at Google Cloud, speaking with EE Times.
Bottom Line
For the MyTechDailyNews readership, the story is less about any single chip and more about a change of rhythm: AI is drifting from tidy question-and-answer bursts toward long-lived digital coworkers that nap for days and then sprint. That pattern pulls storage, networking, virtualization and the humble CPU into what looks like the next defining infrastructure race — and it is worth watching even if you never set foot in a data center.