The Information Machine

AI Agents Underperform Real-World Tasks: CAPTCHAs, Expert Benchmarks, and Memory Quality Failures

closed · v6 · 2026-06-26 · 46 items · history

What's new in v6

Three substantive new items extend the failure-mode catalog. Rohan Paul added RAG hallucination (June 25) — LLMs invent answers even when constrained to supplied documents — as a seventh structural failure mode and a new tension. He also covered LLM trading agent underperformance (June 26), where agents fail to beat naive baselines in rigorous evaluations and exhibit systematic risk miscalibration. Simon Willison, a new voice in the thread, shared a satirical piece depicting two code-review agents generating $41K in inference spend in an irresolvable loop with no technical safety mechanism intervening.

What

Research from June 2026 documents AI agents failing at real-world tasks across at least seven structurally distinct dimensions: expert-level workflow completion [1], CAPTCHA human-verification [3], post-deployment reliability degradation [7], memory quality problems [6], meta-cognitive evaluation gaps [10], world modeling under hidden-state complexity [9], and document-grounded generation where LLMs hallucinate even when explicitly constrained to supplied materials [12]. A critical evaluation of LLM trading strategies adds a financial domain case: agents appear competitive in narrow tests but fail to outperform naive market baselines in longer, fairer evaluations, and exhibit systematically miscalibrated risk behavior across market conditions [11].

Why it matters

The pattern holds across structurally unrelated domains — expert workflows, financial trading, human-verification tasks, document retrieval, agent memory — suggesting the gap between reported benchmark performance and practical utility is a general property of current LLMs, not a domain-specific artifact. Because these failure modes are structurally independent, progress on one does not address the others.

Open questions

  • If LLMs hallucinate even when explicitly constrained to supplied documents [12], what does this imply for enterprise RAG deployments that treat document grounding as a reliability guarantee?

  • Do LLM trading failures — specifically the pattern of being too cautious in bull markets and too risky in bear markets [11] — reflect a general miscalibration in sequential decision-making, or something specific to financial reasoning?

  • If LLMs cannot convert accumulating evidence into stable world models [9], what is the practical ceiling for agent performance on tasks requiring inference about hidden state?

  • Does the solve/judge dissociation [10] mean self-evaluation pipelines are structurally unreliable even when underlying task performance is high?

Narrative

Three research benchmarks published in June 2026 document large gaps between AI agent capability as measured by standard evaluations and performance on practical tasks. The Agents' Last Exam evaluates agents on genuine expert-level workflows end-to-end rather than grading discrete subtasks; frontier agents score under 10% [1]. Michael Lopez Chiesa described it as stopping the practice of grading agents on a curve [2]. The HLL benchmark tests whether agents can pass CAPTCHA human-verification checks on live websites across 10 distinct task types; current agents fail most of them, exposing integration failures in visual understanding, precise interaction, state tracking, and answer submission [3]. MBZUAI, which developed HLL, frames CAPTCHAs as a practical capability check — a task humans pass routinely — rather than a contrived obstacle [4][5].

Two diagnostic framings address agent memory failures from different angles. AGENTCL research argues the primary failure mode is not memory absence but memory quality: agents retain low-quality or irrelevant information, and evaluation should ask whether retained memory actually improves future task performance [6]. A University of Texas study adds a temporal dimension: agents can become progressively less reliable after deployment as accumulated context degrades performance through chat summarization and memory storage, a failure mode that standard benchmarks — assessed at deployment time — do not capture [7]. Research on hidden-rule environments shows that identical observable states can require different actions depending on hidden underlying rules, and LLMs often cannot convert accumulating evidence into a stable internal model of such rules, with performance degrading as hidden-world complexity increases [8][9].

Additional failure modes extend the picture across domains. Frontier models can produce correct answers to math problems but cannot reliably evaluate whether the reasoning in another solution is valid — a dissociation between solving and judging with direct implications for self-evaluation pipelines [10]. In financial domains, LLM trading strategies appear competitive in narrow tests but fail to outperform simple market baselines once evaluation windows lengthen, and agents exhibit systematically miscalibrated behavior: too cautious in bull markets, too risky in bear markets [11]. Document-grounded systems face a similar gap: LLMs frequently invent answers even when explicitly constrained to supplied documents, undermining the common assumption that retrieval-augmented generation eliminates hallucination risk [12].

A satirical incident report shared by Simon Willison illustrates the multi-agent failure mode concretely: two AI code-review agents enter an irresolvable disagreement loop, generating hundreds of comments and over $41,000 in inference spend before Finance intervenes by revoking API keys — with one vendor's marketing team subsequently framing the episode as a demonstration of advanced reasoning capability [13]. The findings collectively document at least seven structurally independent failure modes spanning expert benchmarks, human-verification, post-deployment reliability, memory quality, meta-cognitive evaluation, world modeling under hidden state, and document-grounded generation.

Timeline

  • 2026-06-09: Agents' Last Exam circulates; Michael Lopez Chiesa describes it as a benchmark that stops grading agents on a curve [2][1]
  • 2026-06-11: Rohan Paul summarizes Agents' Last Exam findings, arguing frontier agents score far below what benchmark performance implies for real-world readiness [14]
  • 2026-06-12: AGENTCL framework highlighted: agents fail due to memory quality rather than memory absence [6]
  • 2026-06-13: HLL CAPTCHA benchmark highlighted; current agents fail across 10 task types requiring integrated visual understanding and interaction [3][4][5]
  • 2026-06-14: University of Texas study reported: agents degrade post-deployment as accumulated context degrades performance, a failure mode standard benchmarks do not measure [7]
  • 2026-06-14: Research reported: iterative error-condensing pipelines produce an illusion of abstract learning rather than genuine generalization [15]
  • 2026-06-16: Rohan Paul reports solve/judge dissociation: frontier models can solve math problems correctly but cannot evaluate the validity of reasoning in others' solutions [10]
  • 2026-06-17: Elorian raises $55M, arguing visual reasoning — not code generation — is the remaining barrier to AI agents fully solving programming end-to-end [17]
  • 2026-06-19: Research highlighted: generalist agents must remember hidden environment rules, not just observable current state; identical observations can require different actions under different hidden rules [8]
  • 2026-06-22: Rohan Paul reports LLM agents degrade rapidly as hidden-world complexity increases; LLMs cannot convert accumulating evidence into a stable internal world model [9]
  • 2026-06-25: Rohan Paul reports LLMs hallucinate even when explicitly constrained to supplied documents, questioning the reliability guarantee assumed of RAG systems [12]
  • 2026-06-26: Rohan Paul reports LLM trading agents fail to outperform naive baselines in rigorous evaluations, with systematic risk miscalibration across market conditions [11]
  • 2026-06-26: Simon Willison shares satirical piece depicting two AI code-review agents generating $41K in inference spend in an irresolvable disagreement loop before Finance intervenes [13]

Perspectives

Rohan Paul (@rohanpaul_ai)

Current frontier agents are not ready for real-world automation; benchmark scores systematically overstate practical capability across task completion, human-verification, memory quality, post-deployment reliability, meta-cognitive evaluation, world modeling, document grounding, and financial trading.

Evolution: Consistent amplifier throughout; extended to RAG hallucination (June 25) and LLM trading agent underperformance (June 26).

[14][6][3][7][15][16][10][8][9][12][11]

Michael Lopez Chiesa (@PenQuester)

Agents' Last Exam improves on existing benchmarks by evaluating agents on genuine expert workflows rather than scoped subtasks.

Evolution: Consistent since initial coverage.

[2]

MBZUAI (HLL paper authors)

CAPTCHAs function as a practical capability check for agents; HLL formalizes this across 10 task types to expose integration failures in current systems.

Evolution: Consistent since HLL paper was covered.

[4][5]

University of Texas researchers (via Rohan Paul coverage)

Agents degrade post-deployment as accumulated context degrades performance; standard benchmarks do not capture this because they evaluate only at deployment time.

Evolution: Consistent since first reported on June 14.

[7]

Elorian / Andrew Dai (via The Neuron)

Visual reasoning — not code generation — is the key remaining barrier to AI agents fully solving programming end-to-end; current coding agents cannot reliably perceive and evaluate the visual output they produce.

Evolution: Consistent since first reported on June 17.

[17]

Simon Willison (simonwillison.net)

Multi-agent system failures can generate runaway costs with no technical safety mechanism intervening; financial controls — API key revocation by Finance — are the de facto circuit breaker, not AI safety systems.

Evolution: New voice this pass, endorsing satirical piece on June 26.

[13]

Tensions

  • Standard AI benchmarks report high agent performance; Agents' Last Exam and HLL researchers argue those scores do not translate to real expert tasks or basic human-verification checks. [14][1][3]
  • Prevailing evaluation practice assesses agents at deployment time; University of Texas research shows agents degrade after deployment as context accumulates, making point-in-time benchmarks misleading for operational reliability. [7]
  • The prevailing framing treats agent failure as a memory quantity problem; AGENTCL researchers argue the actual failure mode is memory quality — agents remember the wrong things rather than nothing at all. [6]
  • A model's ability to produce the correct answer implies reasoning capability; the solve/judge dissociation shows a model can reach the right answer yet still misjudge whether another solution's reasoning is valid. [10]
  • Companies assume document-grounded LLM systems are safer because the model has supplied source material; research shows LLMs still hallucinate even when explicitly constrained to those documents. [12]
  • Agent memory frameworks assume accumulating more evidence improves performance; research on hidden-rule environments shows LLMs often cannot convert accumulating evidence into a stable internal model. [8][9]

Status: active but slowing

Sources

  1. [1] New Agents' Last Exam benchmark finds frontier AI agents score under 10% on realistic expert tasks · Digg — reactive:ai-agent-benchmark-reality-gap
  2. [2] Wrote about Agents' Last Exam, the new benchmark that stops grading agents on a curve. The TL;DR: — reactive:ai-agent-benchmark-reality-gap (2026-06-09)
  3. [3] Today’s AI agents still struggle to pass real human-verification checks (CAPTCHAs) on websites. — Rohan Paul Twitter (2026-06-13)
  4. [4] HLL: Can Agents Cross Humanity's Last Line of Verification? — reactive:ai-agent-benchmark-reality-gap
  5. [5] CAPTCHAs aren’t just annoying, they’re a reality check for AI agents - MBZUAI — reactive:ai-agent-benchmark-reality-gap
  6. [6] Most AI agents do not forget because they lack memory; they fail because they remember badly. — Rohan Paul Twitter (2026-06-12)
  7. [7] Univ of Texas paper shows AI agents can slowly become less reliable after deployment, even when the model itself does no… — Rohan Paul Twitter (2026-06-14)
  8. [8] This paper shows that a good generalist agent must remember hidden environment rules, not just observe the current state… — Rohan Paul Twitter (2026-06-19)
  9. [9] Can LLM agents actually discover hidden rules by interacting? — Rohan Paul Twitter (2026-06-22)
  10. [10] This paper shows a strange weakness in AI reasoning: models can solve math, yet fail to judge reasoning. — Rohan Paul Twitter (2026-06-16)
  11. [11] LLM trading agents mostly fail when stock-market tests become long, broad, and fair. — Rohan Paul Twitter (2026-06-26)
  12. [12] This study tests how often LLMs invent answers when they should rely only on supplied documents. — Rohan Paul Twitter (2026-06-25)
  13. [13] Incident Report: CVE-2026-LGTM — Simon Willison (2026-06-26)
  14. [14] Today’s frontier agents are far less ready for real-world automation than their benchmark scores suggest. — Rohan Paul Twitter (2026-06-11)
  15. [15] Researchers found our current approach to making AI smarter over time has a giant blind spot. — Rohan Paul Twitter (2026-06-14)
  16. [16] Long-running language agents may work better if they periodically stop to consolidate memory. — Rohan Paul Twitter (2026-06-14)
  17. [17] 😺 Watch: Elorian wants to fix AI's toddler vision — The Neuron (2026-06-17)