Version 2

NVIDIA's GPU software ecosystem is facing failures on three simultaneous fronts, all surfaced primarily by SemiAnalysis. xAI, reportedly NVIDIA's largest GPU customer, abandoned JAX on NVIDIA GPUs after achieving model flops utilization below 10%, building a custom C-based training framework instead [1][2]. NVIDIA's Blackwell Confidential Computing mode lacks NVLink multicast support, causing a 61% performance regression on large-model inference [4][5]. Most recently, AI-assisted fuzzing uncovered 40+ miscompiles in NVIDIA's ptxas GPU compiler in under three days, with AMD's open-source AMDGPU backend patching five equivalent bugs rapidly by contrast [7].

Taken together, these failures span NVIDIA's training software (JAX/MFU), enterprise isolation features (Confidential Computing), and the compiler layer that underpins all GPU workloads. A GPU compiler that silently miscompiles code is a correctness risk affecting every customer, not just those chasing peak utilization. NVIDIA's closed toolchain makes the pace and transparency of remediation unclear in a way AMD's open-source stack does not.

NVIDIA's GPU ecosystem is facing a credibility challenge across its entire software stack simultaneously. The most operationally visible failure is xAI's reported decision to abandon JAX on NVIDIA GPUs entirely. According to SemiAnalysis, xAI — widely regarded as NVIDIA's single largest GPU customer — found its JAX-based training stack achieving model flops utilization below 10%, far below the 30–50% MFU that competitive AI labs typically target [1]. Rather than continuing to work with NVIDIA's JAX engineering team, xAI built a custom C-based training framework using its own Grok Build tooling. Community discussion has surfaced reports that xAI was at one point utilizing only around 11% of its 550,000 NVIDIA GPUs [2][3], suggesting the hardware-software gap produced significant operational waste at scale.

A second, independent failure involves NVIDIA's Blackwell Confidential Computing feature. SemiAnalysis reported that NVLink multicast — critical for high-performance multi-GPU communication — is unsupported in Blackwell's Confidential Computing mode [4]. Benchmarks running SGLang on Qwen3.5 397B inference show a 61% performance regression when the feature is enabled, leading SemiAnalysis to characterize the implementation as 'complete slop' [5]. Confidential Computing is marketed to enterprise and regulated-industry customers requiring hardware-level data isolation; the performance cost makes it practically unusable for large-model workloads. Notably, enterprise partners including NTT DATA and Fortanix have announced integrations built on NVIDIA Confidential Computing GPUs [6], suggesting the commercial ecosystem is moving forward regardless of the known performance gap.

The third and newest failure vector targets the compiler layer that underlies all GPU code. SemiAnalysis reported that a single engineer, using AI agents and a fuzzer vibe-coded without reading any code, found 40+ miscompiles in NVIDIA's ptxas GPU compiler in under three days [7]. The engineer noted spending more than $10,000 in AI agent compute in a single afternoon, characterizing the capability jump as a watershed moment for automated vulnerability discovery. Both NVIDIA's ptxas and AMD's AMDGPU backend contained bugs found at roughly the same rate — but AMD, with an open-source toolchain, had already patched five of the reported bugs by the time of disclosure [7]. NVIDIA's closed compiler stack leaves the remediation timeline opaque.

Underpinning all three crises is a broader structural argument from SemiAnalysis: the gap between theoretical GPU peak throughput and real-world production performance is too large for manual CUDA kernel optimization to bridge [8][9]. Community research and published benchmarks suggest auto-generated CUDA kernels are now outperforming hand-written expert kernels in several GPU workloads [10][11], pointing toward automated generation as the dominant path forward — though the correctness and production-reliability of that approach remains an open question.