Version 1

Three interconnected GPU performance crises are converging around NVIDIA's ecosystem. xAI, reportedly NVIDIA's largest GPU customer, has abandoned JAX on NVIDIA GPUs after its training stack achieved model flops utilization below 10%, opting instead for a custom C-based framework built with Grok Build [1]. Separately, NVIDIA's Blackwell Confidential Computing mode lacks NVLink multicast support, causing a 61% performance regression on large-model inference workloads [3][4]. Underpinning both: SemiAnalysis argues that closing the gap between theoretical GPU peak throughput and real-world production performance is nearly impossible through manual CUDA kernel tuning, and that auto-generated kernels are now outperforming hand-written ones [5][6].

These are not isolated bugs — they represent systemic gaps between NVIDIA's marketed GPU capabilities and what leading AI labs can actually extract at scale. xAI's defection from JAX signals that even a well-resourced, NVIDIA-dependent lab can find the software stack unworkable enough to rebuild from scratch. The Blackwell Confidential Computing regression affects workloads where hardware-level isolation is required, potentially blocking enterprise GPU adoption in regulated industries.

NVIDIA's GPU ecosystem is facing a credibility challenge on multiple simultaneous fronts. The most dramatic disclosure is xAI's reported decision to abandon JAX on NVIDIA GPUs entirely. According to SemiAnalysis, xAI — widely regarded as NVIDIA's single largest GPU customer — found its JAX-based training stack achieving model flops utilization below 10%, far below the 30–50% MFU that competitive AI labs typically target [1]. Rather than continuing to work with NVIDIA's JAX engineering team, xAI moved to build a custom C-based training framework using its own Grok Build tooling. A separate Hacker News thread surfaced reports that xAI was at one point utilizing only about 11% of its 550,000 NVIDIA GPUs [2], suggesting the hardware-software gap has had real operational consequences.

A second independent failure involves NVIDIA's Blackwell Confidential Computing feature. SemiAnalysis reported that NVLink multicast — critical for high-performance multi-GPU communication — is unsupported in Blackwell's Confidential Computing mode [3]. Benchmarks on SGLang running Qwen3.5 397B inference show a 61% performance regression when the feature is enabled. SemiAnalysis characterized the implementation as 'complete slop,' noting gaps extend across both Hopper and Blackwell architectures. Confidential Computing is marketed to enterprise and regulated-industry customers who need hardware-level data isolation; the performance cost makes it practically unusable for large-model workloads [4].

Beyond these specific failures, SemiAnalysis is also surfacing a broader structural issue: the gap between theoretical GPU peak throughput and real-world production performance is growing too large for manual CUDA kernel optimization to bridge [5][6]. Community discussion and published research suggest that auto-generated CUDA kernels — written without directly programming in CUDA or Triton — are now outperforming hand-written ones in several GPU benchmarks [7][8]. HuggingFace's guide to building production-ready CUDA kernels [9] reflects the same momentum toward automated kernel generation as the dominant path forward.

Taken together, these developments sketch a picture where NVIDIA's hardware remains commercially dominant but its software ecosystem — JAX integration, Confidential Computing, and kernel tooling — is struggling to deliver the utilization rates that its largest customers require. AMD is positioning itself adjacent to this conversation with published inference performance content [10], though it has not directly addressed NVIDIA's specific failures.