Finding Miscompiles for Fun, Not Profit
SemiAnalysis Twitter · SemiAnalysis (@SemiAnalysis_) · 2026-05-28
Compiler engineer Justin Lebar used AI agents (Codex and Claude Opus 4.7) to autonomously discover hundreds of miscompile bugs in NVIDIA's ptxas and AMD's AMDGPU backend in days, spending over $10,000 on API calls in a single afternoon without reading a line of code.
Appears in
Extraction
Topics: compiler-bugsai-agentsfuzzingllvmgpu-compilers
Claims
- AI agents enabled a single engineer to find 40+ ptxas miscompiles in three days with almost no manual effort.
- The cost of running AI agents to find compiler bugs exceeded $10,000 in a single afternoon.
- The capability jump from ChatGPT 5.2 to 5.5 made vibe-coding an entire fuzzer possible without the engineer reading any code.
- Both NVIDIA's ptxas and AMD's AMDGPU backend contained bugs found at roughly the same rate by automated fuzzing.
- AMD has already patched five bugs found through this AI-assisted fuzzing process, with fixes immediately applicable due to the open-source toolchain.
Key quotes
In one afternoon, I spent more than $10,000 running AI agents over compiler code, finding hundreds of plausible bugs in LLVM, including many miscompiles and at least one that's Quite Serious.
I vibe-coded this entire fuzzer, never looking at a line of code.
It was not surprising that fuzzing found bugs. But it was surprising that I found this many bugs so quickly, with almost no manual effort.