Version 2

In mid-May 2026, AMD, Google TPU, and NVIDIA are competing simultaneously on hardware cost, software infrastructure, and open-source benchmarking standards. AMD's MI355 GPU delivers 40% lower inference cost than NVIDIA's B200 on GLM5 single-node FP8 workloads [1], and AMD's MI355X has been confirmed to cost more to manufacture than its NVIDIA rival yet sells at a meaningfully lower price [2] — a deliberate margin sacrifice. Google added nightly CI for the llm-d Kubernetes inference framework on TPU hardware [3] and donated llm-d and TPU drivers to the CNCF [4], deepening its open-source infrastructure commitment. AMD made a historically novel upstream contribution to NVIDIA's AIPerf benchmarking tool [8]. A concrete gap persists, however: an open GitHub issue confirms llm-d images do not yet support AMD MI300 GPUs [7], leaving AMD behind both NVIDIA and Google TPU on the inference stack that enterprise Kubernetes operators increasingly require.

NVIDIA's dominance has rested on two pillars — raw hardware performance and a mature CUDA software ecosystem. AMD is now demonstrating cost-competitive inference hardware even while pricing below cost to gain share, and Google TPU is closing the software infrastructure gap via production-grade Kubernetes tooling donated to a neutral foundation. The asymmetry that cuts against AMD — ahead on cost-per-token, behind on open-source inference stack — is the central unresolved question for enterprise buyers choosing infrastructure in 2026.

The competition for AI accelerator dominance entered a multi-front phase in mid-May 2026, with AMD, Google TPU, and NVIDIA each posting concrete moves across hardware economics, software infrastructure, and open-source collaboration.

The headline hardware result is AMD's: its MI355 GPU is 40% cheaper than NVIDIA's B200 for single-node FP8 inference on the GLM5 architecture, using speculative decoding via SGLang v0.12 across both MTP and non-MTP configurations [1]. SemiAnalysis, which reported the finding, emphasized that it holds on both CUDA and ROCm backends and was achieved just 14 weeks after GLM5's initial launch — framing it as evidence that ROCm's software maturity has crossed a threshold where AMD hardware can realize its cost advantages in production inference. Separately, AMD's MI355X has been confirmed to carry higher manufacturing costs than NVIDIA's competing chip while selling at a substantially lower price [2], a pricing posture that trades near-term margin for market share in the inference cloud.

On the software infrastructure front, Google made two complementary moves. It added nightly continuous integration for llm-d — an open-source Kubernetes-native distributed inference framework — on its TPU hardware [3], signaling a commitment to keeping TPU compatibility production-green on each code push rather than treating OSS support as a periodic patch. Google also donated llm-d and its TPU drivers to the Cloud Native Computing Foundation (CNCF) [4], placing the framework under neutral governance and lowering the barrier for enterprise operators to standardize on it. Google has separately published 3X inference speedups on TPU using diffusion-style speculative decoding [5], adding a throughput dimension to its competitiveness claim beyond Kubernetes tooling. Commentary in the community has flagged the software moat angle as under-appreciated: one observer noted that Google wiring TPU into llm-d CI represents the software moat narrowing, not just the silicon gap closing [6].

AMD's relative position on the software stack is more complicated. A GitHub issue filed against the llm-d repository confirms that llm-d images do not yet support AMD MI300 GPUs [7], making AMD's llm-d gap concrete rather than merely relative. AMD's upstream contribution to NVIDIA's AIPerf benchmarking sub-project within the Dynamo repository [8] — believed to be the first such cross-competitor merge — suggests some willingness to collaborate on shared measurement infrastructure, but it does not close the inference-framework gap. Running against the competitive-pressure narrative is Jensen Huang's articulation at Stanford's CS153 that he would prefer to operate at low Model FLOP Utilization (MFU) at all times [9]: his argument inverts conventional framing, positioning deliberate over-provisioning as a strategic asset rather than waste — a posture that, if adopted broadly by hyperscalers, would tend to favor capacity incumbents over cost-efficiency challengers.