Version 3

Simon Willison's two May 2026 essays — one on an AI-managed Stockholm café whose autonomous actions harmed unconsenting third parties [1], and one on 'normalization of deviance' eroding code-review norms in professional software development [2] — have become a focal point for a broader industry conversation. Andrej Karpathy has independently marked the same transition, cited as declaring the 'vibe coding era is ending' in favor of 'agentic engineering' [4], lending a second major voice to the framing Willison drew. Legal scholars and legislators are beginning to engage the liability gap: the AI LEAD Act (S.2937) is moving through the US Senate [5], and academic legal literature is framing AI agents as 'risky agents without intentions' that current tort frameworks cannot adequately address [6][7]. Willison's concern about normalization of deviance dates at least to December 2025 [3], suggesting his May critiques were the culmination of a developing argument rather than an ad hoc reaction.

The convergence of a practitioner critique (Willison), an influential researcher framing (Karpathy), and emerging legislative attention (AI LEAD Act, EU AI Act) suggests that the question of who is responsible when an AI agent harms a third party is moving from blog-post debate toward formal governance. The profession and the law are both racing to develop norms before a high-profile failure forces the issue.

In early May 2026, Simon Willison published two essays that together diagnosed a shared failure mode in AI autonomy: as AI systems become more reliable, the humans nominally responsible for them quietly reduce oversight — until the system fails in ways that harm third parties or accumulate invisibly in codebases.

The first essay analyzed an experiment by Andon Labs, which deployed an AI manager named Mona at a café in Stockholm (following an earlier AI-run retail store in San Francisco). Mona ordered 120 eggs despite having no stove, purchased 22.5 kg of canned tomatoes for a fresh sandwich menu, sent unsolicited 'EMERGENCY' emails to suppliers to correct its own prior mistakes without human review, and submitted an outdoor seating permit application to police that included a diagram generated without the AI ever observing the actual street [1]. Willison's critique centered on a specific ethical claim: the experiment's error costs were borne not by Andon Labs but by suppliers, public officials, and permit reviewers who had never consented to participate in an AI trial. Any AI agent capable of outbound actions affecting other people must, in his view, require a human in the loop before those actions are executed [1].

The second essay turned inward, to Willison's own experience as a professional engineer using AI coding tools. He argued that what once seemed a clear distinction — 'vibe coding' (casual, low-scrutiny AI-assisted generation) versus 'agentic engineering' (disciplined, reviewed AI-assisted development) — is blurring uncomfortably [2]. As AI coding agents become more reliably correct, professional engineers reduce the intensity of their review. Willison named this 'normalization of deviance': each incremental success without close monitoring raises the engineer's trust, increasing the risk of catastrophic misplaced trust in a future edge case [2]. He also identified a structural break: the entire software development lifecycle — review practices, testing cadences, quality norms — was implicitly calibrated to humans producing a few hundred lines of code per day. That assumption is now broken, and the profession has not yet developed replacement norms [2]. Willison had been building toward this argument since at least December 2025, when he published an earlier piece on normalization of deviance in AI systems more broadly [3].

The framing Willison introduced has since been echoed by other voices. Andrej Karpathy — a foundational figure in AI research and a key reference point for the software engineering community — has been cited as declaring that the 'vibe coding era is ending' and that 'agentic engineering' is next [4], lending a second major voice to the same conceptual boundary Willison drew. Separately, legal scholars and legislators are beginning to engage the liability gap that Willison's café essay made visceral: the AI LEAD Act (S.2937) is in the US Senate [5], academic legal literature frames AI agents as 'risky agents without intentions' that current tort frameworks cannot cleanly handle [6][7], and state-level AI legislation has been proliferating [8]. The question these frameworks face — who bears accountability when an AI agent harms a third party who never consented to interact with it — maps directly onto Willison's critique of the Stockholm café experiment.