The Information Machine

AI-Generated Content: Hallucination, Deliberate Misuse, and Undetectability · history

Version 4

2026-05-25 11:07 UTC · 83 items

What

Three documented failure modes of AI-generated content — fabricated quotes in published books, AI noise overwhelming security vulnerability disclosure, and hallucinated citations in court filings — have developed substantial institutional and legal accountability tracks simultaneously. The core cases are Ron Rosenbaum's book about AI misinformation containing AI-hallucinated quotes denied by named subjects [1]; curl's bug bounty program formally terminated after AI slop made it unworkable [13][14]; and courts sanctioning lawyers under Rule 11 for submitting AI-hallucinated citations [6][5]. The Mythos security-scanning thread now has new factual detail: a Reddit discussion records that curl's maintainer used what it describes as 'Anthropic's Mythos scan' and found 1 confirmed vulnerability and approximately 20 bugs [17], while an AISLE analytical piece examines what the Mythos case means for AI in cybersecurity under the 'jagged frontier' framing [19]. Thomson Reuters has added major institutional weight to the legal accountability track with a formal 2026 report on responsible AI use in courts [8], and Drug & Device Law published practitioner commentary on AI hallucinations in March 2026 [9].

Why it matters

Legal and institutional accountability for AI fabrications is now converging across publishing, law, and security infrastructure simultaneously, while the corrective tools (AI detectors, bug bounty triage) have been documented as either inequitable or unworkable. The Mythos thread suggests that structured AI security scanning may operate in a fundamentally different success regime from bounty-hunting slop — but that distinction is not yet institutionally codified, and the factual record about what Mythos found and under whose auspices remains unresolved.

Open questions

  • A Reddit post describes 'Anthropic's Mythos scan' finding 1 confirmed vulnerability and ~20 bugs in curl [17], while prior reporting indicated Mythos (described then as a separate company) found five real vulnerabilities [18] — are these the same engagement with discrepant counts, different scans, or does Mythos belong to Anthropic rather than operating independently?

  • Thomson Reuters published a 2026 report specifically on responsible AI use in courts and hallucinations [8] — do its recommendations include concrete verification standards that bar associations could adopt, potentially closing the gap between active Rule 11 sanctions and the absence of agreed professional guidance?

  • With Drug & Device Law adding sector-specific commentary on AI hallucinations [9], is pharmaceutical and medical device litigation developing specific AI liability exposure distinct from general legal malpractice, and if so, what makes that sector distinctive?

  • With AI detectors condemned by Stanford HAI, NIH, and CDT as structurally inequitable [20][21][22], and with Thomson Reuters now adding institutional weight on the court-hallucination side [8], will academic institutions face new pressure to revise enforcement policies built on tools their own evidence base condemns?

Narrative

Three high-profile collapses of AI-generated content quality — in publishing, cybersecurity, and legal proceedings — are documented with named actors, specific incidents, verifiable primary sources, and accumulating legal consequences, while the tools meant to detect AI-generated text have been shown by multiple major research institutions to fail inequitably.

The most prominent publishing case is journalist Ron Rosenbaum's book The Future of Truth: How AI Reshapes Reality — a work explicitly about AI's distortions of fact — which a New York Times investigation found to contain fabricated quotes attributed to real named individuals [1]. Tech journalist Kara Swisher stated she never said the attributed quote; Northeastern University professor Lisa Feldman Barrett said the quotes attributed to her do not appear in her book and are also factually wrong, a two-level repudiation covering both attribution and substance. Rosenbaum acknowledged the error and launched a citation audit while continuing to defend AI tools in his research workflow. Legal practitioners have framed such incidents under a doctrine called 'defamation by hallucination': a November 2025 article documented courts testing this frontier [2], Quinn Emanuel published a client alert on defamation liability in the AI era [3], and Thompson Coburn LLP warned writers that AI hallucinations of attributed statements create personal legal exposure [4].

The legal system's encounter with AI hallucination extends beyond defamation into the courtroom itself. A 2025 review by Sterne Kessler documented an accumulating record of court sanctions for AI-hallucinated citations in legal filings across jurisdictions [5], and in February 2026, AI-hallucinated case citations prompted sanctions and materially delayed a class action settlement [6]. A Helsell Fetterman review published in April 2026 continued tracking ongoing costs to lawyers [7]. These are Rule 11 violations and professional misconduct findings — an enforcement track already active, running in parallel to the defamation doctrine developing in publishing contexts. Thomson Reuters' January 2026 report on responsible AI use in courts [8] adds major legal-industry institutional weight to this accountability track, and Drug & Device Law's March 2026 practitioner commentary [9] signals that sector-specific legal communities are engaging the hallucination liability question on their own terms.

In cybersecurity, the problem of AI-generated false vulnerability reports moved from an operational complaint to a formally terminated program with primary-source documentation. Curl founder Daniel Stenberg had publicly criticized AI slop bug submissions as early as May 2025 [10][11][12], and on January 26, 2026, he formally announced the end of curl's bug bounty program [13]. A GitHub gist published by Stenberg provides documentary evidence of the actual AI slop submissions curl received [14]. Bugcrowd reported that bug bounty submissions more than quadrupled over a three-week period in March 2026, with most proving false [15], and RedMonk framed the dynamic as a 'vulnerability treadmill' threatening the economics of coordinated vulnerability disclosure [16]. A complicating thread concerns Mythos, a scanning tool described in a Reddit post as 'Anthropic's Mythos scan': Stenberg apparently used it and found 1 confirmed vulnerability and approximately 20 bugs in curl [17] — both qualifying his prior claim that no valid AI-assisted security report had ever been received, and raising the question of whether Mythos belongs to Anthropic rather than operating as an independent company as previously framed [18]. AISLE's analytical piece applying the 'jagged frontier' framework to AI in cybersecurity after the Mythos case [19] suggests this distinction — between structured AI security scanning and bounty-hunting noise — is already being theorized in industry commentary.

The tools positioned to address AI content verification — AI writing detectors — are condemned by a multi-institutional body of evidence as systematically biased against non-native English speakers. Stanford HAI documented that AI detectors produce false positives at higher rates for non-native English writers [20]; an NIH-published peer-reviewed study confirmed the finding for GPT detectors specifically [21]; the Center for Democracy and Technology documented disproportionate effects on English learners [22]; and Turnitin's own blog engaged with the bias question regarding its own tool [23]. Research arguing that false positives are structurally unavoidable due to the statistical variation of human writing — not merely a classifier quality problem — adds a fundamental technical limit to the institutional and equity critiques [24]. The cumulative picture is not a technical gap that better tools might close but a structural limitation that equitable enforcement cannot be built on top of.

Timeline

  • 2025-05-07: Curl founder Daniel Stenberg publicly criticizes AI slop bug submissions; curl states it has still not seen a single valid AI-assisted security report [10][11][12]
  • 2025-07-24: TechCrunch reports AI slop and fake reports are exhausting bug bounty programs industry-wide, before curl's formal program termination [30]
  • 2025-11-17: Minnesota Lawyer reports courts are beginning to test a new frontier of defamation law as AI-hallucinated content enters litigation [2]
  • 2026-01: Thomson Reuters publishes 'Hallucinations Report 2026' on responsible AI use for courts [8]
  • 2026-01-26: Daniel Stenberg formally announces the end of curl's bug bounty program; a GitHub gist documenting actual AI slop submissions received is published [13][14]
  • 2026-02-03: AI-hallucinated case citations prompt court sanctions and delay a class action settlement [6]
  • 2026-03: Bugcrowd reports bug bounty submissions more than quadrupled over three weeks, with most proving to be false AI-generated reports; some companies suspend programs entirely [15]
  • 2026-03: Drug & Device Law blog publishes practitioner commentary on AI hallucinations in pharmaceutical and medical device litigation contexts [9]
  • 2026-04-24: Helsell Fetterman review documents ongoing costs to lawyers from AI hallucinations in court filings [7]
  • 2026-05-05: RedMonk analyst publishes 'AI Slop & the Vulnerability Treadmill,' framing AI-generated false reports as a structural economic threat to coordinated vulnerability disclosure [16]
  • 2026-05-22: New York Times investigation finds AI-hallucinated quotes in Rosenbaum's The Future of Truth; Kara Swisher and Lisa Feldman Barrett publicly deny the attributed quotes [1]
  • 2026-05-23: Rohan Paul summarizes research arguing AI writing detectors fail due to fundamental statistical limitations of human writing variation, not classifier quality alone [24]
  • 2026-05-24: Social commentary notes AI misinformation is outpacing fact-checking by hours and that detection methods journalists rely on 'keep expiring' [31]
  • 2026-05: AISLE publishes analytical piece on AI in cybersecurity after Mythos using the 'jagged frontier' framework; Reddit discussion documents curl maintainer's use of 'Anthropic's Mythos scan' yielding 1 confirmed vulnerability and approximately 20 bugs [19][17]

Perspectives

Ron Rosenbaum (author, The Future of Truth)

Acknowledges AI-hallucinated quotes were an error and is conducting a citation audit, but continues to defend AI research tools and their role in his workflow

Evolution: Consistent through the incident: apologetic about the specific failure, not about the broader practice

Kara Swisher (tech journalist, attributed quote subject)

Direct denial — states the quote attributed to her is something she never said

Evolution: Consistent; victim of fabricated attribution with no stated prior position on AI research tools

Lisa Feldman Barrett (Northeastern University professor, attributed quote subject)

States the attributed quotes do not appear in her book and are also factually wrong — repudiating both the attribution and the substance

Evolution: Consistent; victim of fabricated attribution

Daniel Stenberg / curl project

Formally ended curl's bug bounty program in January 2026 after AI slop made it unworkable; published a GitHub gist as verifiable documentary evidence of the fake submissions; subsequently used what a Reddit post describes as 'Anthropic's Mythos scan' and found 1 confirmed vulnerability and approximately 20 bugs

Evolution: Hardened from initial public criticism to formal program termination with primary-source documentation; the Mythos scan result further qualifies his prior claim that no valid AI-assisted security report had ever been received, and may clarify Mythos's affiliation with Anthropic rather than as a separate independent company

AISLE (industry analyst / blog)

Frames the post-Mythos AI cybersecurity landscape through the 'jagged frontier' concept — implying AI has genuine capability in some structured security research contexts while failing badly in others

Evolution: New voice in the thread; adds an analytical industry framing that distinguishes between AI security research contexts rather than treating AI-generated security content as uniformly problematic

Bugcrowd (bug bounty platform)

Reports empirical data showing a dramatic spike in AI-generated false submissions and signals severe operational burden

Evolution: Consistent industry concern; March 2026 scale data shows the problem has grown substantially since the curl case in 2025

RedMonk (technology analyst firm)

Frames AI-generated false reports as a 'vulnerability treadmill' that fundamentally threatens the economics of coordinated vulnerability disclosure

Evolution: Consistent since first appearance in thread; adds an industry analyst framing to what had been primarily platform-operator and maintainer complaints

Legal scholars and law firm practitioners (defamation and AI liability)

AI hallucinations fabricating attributed statements are actionable defamation; courts are testing this frontier; Quinn Emanuel, Thompson Coburn, Helsell Fetterman, and Drug & Device Law practitioners warn of legal exposure across publishing and sector-specific legal contexts alike

Evolution: Expanded this pass: the pharmaceutical and medical device litigation sector has joined the practitioner commentary track via Drug & Device Law, and Thomson Reuters' institutional report adds major legal-information-industry weight to the court accountability framing

Thomson Reuters (legal information industry)

Published a formal 2026 report specifically on responsible AI use for courts and AI hallucinations, adding institutional industry weight to calls for structured AI verification in legal contexts

Evolution: New voice in the thread; represents the legal information publishing sector weighing in institutionally rather than individual practitioners

Stanford HAI / NIH / CDT researchers (AI detector bias)

AI writing detectors produce false positives at systematically higher rates for non-native English speakers — a finding now confirmed across multiple major research institutions, with Turnitin's own blog engaging the question about its tool

Evolution: Consistent multi-institutional consensus; reinforced by structural critique from NLP researchers that the limitation is inherent to classifier design, not correctable with better training data

Rohan Paul and NLP researchers (AI detector structural limits)

AI detection tools fail for a reason that cannot be engineered away: human writing is statistically too varied, making false positives structurally unavoidable regardless of classifier quality

Evolution: Consistent skepticism; reinforced by the multi-institutional bias documentation as independent corroboration from a different angle

Tensions

  • Rosenbaum defends continued AI tool use in research even after his book about AI misinformation was found to contain AI-hallucinated fabrications — a position Swisher and Barrett implicitly contest by naming concrete reputational harm done to them as real individuals falsely quoted, and that legal practitioners contest by framing such fabrications as potentially actionable defamation with personal legal exposure [1][2][3][25][26][4]
  • Institutions enforcing academic integrity via AI detectors assume the tools are reliable enough to act on, while Stanford HAI, NIH, and CDT research documents that false positives are structurally unavoidable and disproportionately burden non-native English speakers — making detector-based enforcement both technically and ethically contested [24][20][21][22][23][29]
  • Daniel Stenberg terminated curl's bug bounty because AI slop made it unworkable and had claimed no valid AI-assisted security report had ever been received — but a Reddit post reports he subsequently used 'Anthropic's Mythos scan' and found 1 confirmed vulnerability and approximately 20 bugs, while prior reporting described Mythos as an independent company that found five real vulnerabilities; the factual record on Mythos's affiliation and findings remains unresolved [13][11][18][17]
  • AISLE's 'jagged frontier' framing implies AI has genuine structured security research capability in some contexts — validating a distinction between bounty-hunting slop and disciplined scanning — while the broader platform record (Bugcrowd, RedMonk, Stenberg's program termination) suggests the dominant operational reality is noise, not signal [19][15][16][13][17]
  • Courts are already sanctioning lawyers under existing Rule 11 mechanisms for AI-hallucinated citations — an active enforcement track with documented sanctions and settlement delays — while the legal profession has no agreed standard for AI verification before filing, a gap Thomson Reuters' 2026 court-hallucination report implicitly addresses but formal bar association guidance has not yet filled [6][5][7][8]
  • Legal scholars frame 'defamation by hallucination' as structurally inevitable and actionable — implying ongoing legal exposure for AI vendors, authors, and publishers — while AI tool advocates (represented implicitly by Rosenbaum's continued defense) frame individual errors as correctable edge cases rather than systemic liabilities [25][26][1][3][4]

Sources

  1. [1] AI put "synthetic quotes" in his book. But this author wants to keep using it. — Ars Technica AI (2026-05-22)
  2. [2] Courts test new frontier of defamation law as AI enters mix — reactive:ai-content-integrity
  3. [3] Client Alert: Defamation in the AI Era — reactive:ai-content-integrity
  4. [4] AI Hallucinated Me. If You Are a Writer, It May Hallucinate You | Thompson Coburn LLP — reactive:ai-content-integrity
  5. [5] AI IP Year in Review - AI Hallucinations in Court Filings and Orders — reactive:ai-legal-hallucination
  6. [6] AI Hallucinated Case Citations Prompt Sanctions And Delay Class Action Settlement – Class Action Defense — reactive:ai-content-web-degradation
  7. [7] AI Hallucinations Keep Costing Lawyers in Court | Helsell Fetterman — reactive:ai-content-integrity
  8. [8] [PDF] Responsible AI use for courts - Thomson Reuters — reactive:ai-content-integrity
  9. [9] A Modest Proposal Concerning AI Hallucinations | Drug & Device Law — reactive:ai-content-integrity
  10. [10] Curl eliminates bug bounty program due to AI slop - CSO Online — reactive:ai-content-integrity
  11. [11] Curl: We still have not seen a valid security report done with AI help | Hacker News — reactive:ai-content-integrity
  12. [12] Curl project founder snaps over deluge of time-sucking AI slop bug ... — reactive:ai-content-integrity
  13. [13] The end of the curl bug-bounty | daniel.haxx.se — reactive:ai-content-integrity
  14. [14] AI slop security reports submitted to curl · GitHub — reactive:ai-content-integrity
  15. [15] Bug bounty businesses bombarded with AI slop — Ars Technica AI (2026-05-18)
  16. [16] AI Slop & the Vulnerability Treadmill – console.log() — reactive:ai-content-integrity
  17. [17] 1 confirmed vulnerability and ~20 bugs : r/ClaudeAI — reactive:ai-content-integrity
  18. [18] curl founder Daniel Stenberg says Mythos identified five ... - Facebook — reactive:ai-content-integrity
  19. [19] AI Cybersecurity After Mythos: The Jagged Frontier - AISLE — reactive:ai-content-integrity
  20. [20] AI-Detectors Biased Against Non-Native English Writers | Stanford HAI — reactive:ai-content-integrity
  21. [21] GPT detectors are biased against non-native English writers — reactive:ai-content-integrity
  22. [22] Disproportionate Effects of Generative AI-Detectors on English ... — reactive:ai-content-integrity
  23. [23] Is Turnitin's AI Detector Biased Against Non-Native English Writers? — reactive:ai-content-integrity
  24. [24] AI detectors fail because student writing is too varied to judge from 1 document. — Rohan Paul Twitter (2026-05-23)
  25. [25] Inevitable Errors: Defamation by Hallucination in AI Reasoning Models — reactive:ai-content-integrity
  26. [26] Courts Navigating AI Defamation Opens Legal Risks for Companies — reactive:ai-content-integrity
  27. [27] Study Reveals AI Detectors' False Positives on Non-Native Writers — reactive:ai-content-integrity
  28. [28] ESL students are getting falsely flagged by AI detectors and ... - Reddit — reactive:ai-content-integrity
  29. [29] The Problem with False Positives: AI Detection Unfairly Accuses ... — reactive:ai-content-integrity
  30. [30] AI slop and fake reports are coming for your bug bounty programs — reactive:ai-content-integrity
  31. [31] AI misinformation now outpaces fact-checking by hours, the detection methods journalists rely on keep expiring, and the ... — reactive:ai-content-integrity (2026-05-24)