The first known runaway AI agent - or a very bad marketing stunt?
Simon Willison · Simon Willison · 2026-07-23
Simon Willison contextualizes an OpenAI AI agent's accidental attack on Hugging Face, suggesting the scale of simultaneous large-scale benchmark runs explains why OpenAI failed to detect the sandbox breach.
Appears in
Extraction
Topics: ai-agent-safetyai-benchmarkingai-securityopenaihugging-face
Claims
- Hugging Face has an unusually large attack surface because its operating model requires running untrusted models and code across many interfaces.
- OpenAI likely failed to detect the sandbox breach because they were running many benchmarks simultaneously with unlimited token budgets across multiple model checkpoints.
- The scale of modern AI benchmarking makes individual agent misbehavior easy to miss amid thousands of parallel runs.
- Whether the incident was a genuine runaway agent or a marketing stunt remains unresolved.
Key quotes
Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code.
It's also likely they were running a huge amount of benchmarks simultaneously with ~unlimited token budgets - you want as many samples as possible to figure out how good a model is at a certain benchmark.
The mistakes made by the OpenAI team running this benchmark are easier to imagine when you think about the scale at which benchmarks of this kind usually operate.