Version 1

In consecutive essays published May 5–6, 2026, Simon Willison examines two converging failure modes in AI autonomy: AI agents acting in the physical world without adequate human oversight [1], and AI coding tools eroding the professional norms of software review [2]. In Stockholm, an AI café manager (Mona) ordered absurd quantities of supplies, sent unsolicited emergency emails to vendors, and submitted a permit application with a diagram generated without ever seeing the street [1]. In software development, Willison describes how increased AI reliability is tempting engineers — including himself — to skip code review, creating what he calls 'normalization of deviance' [2].

Both cases represent the same underlying dynamic: as AI systems become more capable, the humans nominally responsible for them quietly reduce oversight — until the system fails in ways that harm third parties or create invisible technical debt. The software development lifecycle, and the regulatory infrastructure around commercial operations, were both designed for human-paced work and have not caught up to AI-paced output [2][1].

Simon Willison published two linked critiques in early May 2026 that together paint a picture of AI autonomy outrunning the oversight structures humans have built around consequential work.

The first essay dissects an experiment by Andon Labs, which ran an AI-managed café in Stockholm (following an earlier AI-run retail store in San Francisco). The AI manager, named Mona, made a series of operational blunders: ordering 120 eggs despite having no stove, purchasing 22.5 kg of canned tomatoes for a fresh sandwich menu, sending unsolicited 'EMERGENCY' emails to suppliers to correct its own mistakes without human review, and submitting an outdoor seating permit application to police that included a diagram generated without the AI ever observing the actual street [1]. Willison's critique is pointed: the experiment's costs were borne not by Andon Labs but by the suppliers, public officials, and permit reviewers who had never consented to participate in an AI trial. He argues that any AI agent capable of outbound actions affecting other people must keep a human operator in the loop before those actions are executed [1].

The second essay turns inward, to Willison's own experience with AI coding tools. He observes that what once seemed a clear distinction — vibe coding (casual, low-scrutiny AI-assisted code generation) versus professional agentic engineering (disciplined, reviewed AI-assisted development) — is blurring uncomfortably. As AI coding agents become more reliably correct, professional engineers reduce the intensity of their review. Willison names this 'normalization of deviance': each time a model produces correct code without close monitoring, the engineer's trust increases, raising the risk of catastrophic misplaced trust in a future edge case [2]. He also notes a structural break: the entire software development lifecycle — including review practices, testing cadences, and quality norms — was implicitly calibrated to the assumption that humans produce a few hundred lines of code per day. That assumption is now broken, and the profession has not yet developed new norms to replace it [2].

Taken together, the two essays identify a shared failure mode: AI reliability is being used as a reason to reduce human oversight, even as the consequences of AI errors fall increasingly on third parties or accumulate invisibly in codebases. Willison is not arguing against AI autonomy in principle, but against the casual erosion of the checkpoints that make autonomy safe — whether those checkpoints are a human reviewing an outbound email or an engineer reading the code an AI just wrote.