Nvidia Shifts Focus to AI Inference as $1 Trillion Market Emerges

The center of gravity in artificial intelligence is shifting from training massive models to inference – the continuous, rapid, and profitable application of those models for users and businesses. This transition, signaled for months, is now driving significant changes in the market and among key players, fostering a growing supply chain around this new focus.

The ‘Inference Inflection’ at GTC 2026

A pivotal moment occurred at Nvidia’s GTC 2026 conference (March 16-19, 2026) in San Jose, where CEO Jensen Huang linked this phase change to the rise of software agents, which demand more constant and distributed computing power than traditional large training cycles. Nvidia now estimates the revenue opportunity for its Blackwell and Rubin chips to reach at least $1 trillion by 2027 .

Gartner predicts that infrastructure spending on inference will surpass that of training in 2026 and will double it by 2029 . This signifies a maturation of the AI ecosystem, moving beyond a simple race to train the most powerful models.

A Four-Layered Ecosystem

GTC 2026 highlighted a new hierarchy encompassing at least four interconnected layers: chips, memory, and networking; software for orchestrating inference; cloud and data centers; and applications that generate revenue from tokens. Nvidia responded by presenting Dynamo 1.0, an open-source software platform for large-scale generative and agentic inference, described as the distributed operating system for AI factories. Combined with TensorRT-LLM optimizations, Dynamo can reportedly increase inference performance on Blackwell by up to 7x, reducing the cost per token.

The ability to transform computation into a continuous industrial service – with low latency, predictable costs, efficient resource management, and coordinating software – is now paramount. Nvidia is positioning itself not just as a semiconductor manufacturer, but as the unifying layer for token production.

Beyond GPUs: Vera Rubin and Specialized CPUs

Nvidia is expanding beyond traditional GPUs with the Vera Rubin platform. Vera CPU is presented as a processor specifically designed for agentic and reinforcement learning workloads, and paired with Groq’s 3 LPU integration, delivers 35x inference throughput gains . This architecture, with Rubin handling prefill and Groq handling decode, represents Nvidia’s new inference offensive.

The $1 trillion estimate by 2027 covers Blackwell and Rubin, but excludes CPU, networking, Groq, Rubin Ultra, and H200 sales to China, demonstrating the breadth of Nvidia’s core business and its efforts to expand beyond a single, symbolic number.

The Challenges to Nvidia’s Dominance

As the market increasingly focuses on the unit cost of inference, Nvidia’s historically high margins may be harder to maintain. While Nvidia is unlikely to lose its leadership position, the competitive landscape is becoming less favorable for a single dominant supplier. Pressure is mounting from established CPU manufacturers like AMD and Intel, as well as new players like Cerebras and hyperscalers such as Amazon and Google, who are developing custom chips.

Software as a Key Differentiator

Inference is not simply a “lighter” version of training; it’s a new form of complexity requiring real-time orchestration of memory, cache, network, models, tools, and agents. This is why Nvidia emphasizes software like Dynamo, runtimes like OpenShell, and toolkits for building secure and private agents. The goal is to create an “AI industrial operating system” that efficiently manages computational resources.

The Rise of Specialized Components and Hyperscalers

The shift towards cost-effective inference is benefiting specialized actors. The Nvidia-Groq architecture and the AWS-Cerebras partnership (combining Trainium3 and Wafer-scale Brains chips) exemplify this trend. Memory, storage, and interconnect manufacturers are also gaining prominence, with Micron shipping HBM4 36GB 12H designed for Vera Rubin, offering increased bandwidth and power efficiency.

Hyperscalers like Alphabet, Amazon, Meta, and Microsoft are investing heavily in AI infrastructure (approximately $650 billion in 2026, up from $410 billion in 2025) and are becoming market architects, combining proprietary and third-party components to optimize cost and performance.

From Chatbots to Agents and Physical AI

The focus is shifting from chatbots to agents capable of performing tasks autonomously. Nvidia presented NemoClaw for the OpenClaw community and an Agent Toolkit to add runtime, guardrails, and security infrastructure for autonomous agents. The company is also expanding into physical AI applications, including robotics, computer vision, autonomous vehicles, and digital twins, with a Physical AI Data Factory Blueprint for generating and augmenting training data.

Geopolitical Considerations

Nvidia has received approval from Beijing to sell H200 chips in China, and is preparing compatible Groq chips for that market. This highlights the importance of serving markets with varying geopolitical and regulatory constraints.

The Future of AI: An Industrial Supply Chain

The future of AI will resemble an industrial supply chain focused on agents, continuous inference, hybrid infrastructures, energy efficiency, and cost optimization. Nvidia remains a key leader, but must defend its position in a market that rewards efficiency and integration. The new winners will be those who control the cost, latency, distribution, energy consumption, security, and ability to put AI to operate effectively.