The Information Machine

AI-Generated Content: Hallucination, Deliberate Misuse, and Undetectability · history

Version 3

2026-05-25 04:33 UTC · 79 items

What

Three documented failures of AI-generated content are reshaping publishing, cybersecurity, and law simultaneously. Journalist Ron Rosenbaum's book explicitly about AI misinformation was found to contain AI-fabricated quotes denied by their named subjects [1]; curl's bug bounty program was formally shut down in January 2026 after AI slop reports made it unworkable [12], with curl's founder publishing a GitHub gist documenting the actual fake submissions received [13]; and courts have begun sanctioning lawyers for submitting AI-hallucinated case citations in filings [6][5][7], adding a Rule 11 enforcement track alongside the emerging 'defamation by hallucination' doctrine [2][3]. AI writing detectors, the presumed corrective tool, are now documented by Stanford HAI [17] and NIH-published research [18] to produce false positives at systematically higher rates against non-native English speakers.

Why it matters

Legal accountability for AI fabrications is materializing on two parallel tracks simultaneously — defamation doctrine for attributed content in publishing, and existing court sanction mechanisms for hallucinated citations in legal filings — suggesting accountability structures are taking shape faster than detection or prevention tools. The institutional corrective (AI detectors) is now condemned by enough major research institutions that detector-based enforcement can no longer be defended as a merely imperfect technical measure; it is an inequitably applied one.

Open questions

  • Curl's founder acknowledged that a company called Mythos separately identified five real vulnerabilities in curl [16] — does this represent legitimate AI-assisted security research working where bug bounty submissions failed, and if so, what distinguishes it from the slop?

  • With courts already sanctioning lawyers under existing Rule 11 mechanisms for AI-hallucinated citations [6][7] but no agreed professional standard for pre-filing AI verification in place, will bar associations formalize guidance before the next wave of sanctions?

  • With AI detector bias against non-native English speakers now documented by Stanford HAI [17], CDT [19], and NIH-published peer-reviewed research [18], will academic institutions actually revise enforcement policies — or continue relying on tools their own evidence base condemns as inequitable?

  • Does the curl case — terminating a program rather than reforming it [12] — represent the terminal response pattern for open-source security infrastructure under AI noise, or will a successor model for coordinated vulnerability disclosure emerge?

Narrative

Three high-profile collapses of AI-generated content quality — in publishing, cybersecurity, and legal proceedings — are now documented with named actors, specific incidents, verifiable primary sources, and emerging legal consequences, while the tools meant to detect AI-generated text have been shown by multiple major research institutions to fail inequitably.

The most prominent publishing case is journalist Ron Rosenbaum's book The Future of Truth: How AI Reshapes Reality — a work explicitly about AI's distortions of fact — which a New York Times investigation found to contain fabricated quotes attributed to real named individuals [1]. Tech journalist Kara Swisher stated she never said the attributed quote; Northeastern University professor Lisa Feldman Barrett said the quotes attributed to her do not appear in her book and are also factually wrong — a two-level repudiation covering both attribution and substance. Rosenbaum has acknowledged the error and launched a citation audit while continuing to defend AI tools in his research workflow. Legal practitioners have begun framing such incidents under a doctrine called 'defamation by hallucination': a November 2025 article documented courts testing this new frontier of defamation law [2], Quinn Emanuel published a client alert on defamation liability in the AI era [3], and Thompson Coburn LLP warned writers specifically that AI hallucinations of attributed statements create personal legal exposure [4].

The legal system's encounter with AI hallucination extends beyond defamation into the courtroom itself. A 2025 review by Sterne Kessler documented an accumulating record of court sanctions for AI-hallucinated citations in legal filings across jurisdictions [5], and in February 2026, AI-hallucinated case citations prompted sanctions and materially delayed a class action settlement [6]. A Helsell Fetterman review published in April 2026 continued tracking ongoing costs to lawyers [7]. These are not defamation cases but Rule 11 violations and professional misconduct findings — a distinct enforcement track that is already active, not emergent, running in parallel to the attribution-and-defamation doctrine developing in publishing contexts.

In cybersecurity, the problem of AI-generated false vulnerability reports moved from an operational complaint to a formally terminated program with primary-source documentation. TechCrunch reported in July 2025 that AI slop and fake reports were already 'exhausting' bug bounty programs industry-wide [8]. Curl founder Daniel Stenberg had publicly criticized the AI slop problem as early as May 2025 [9][10][11], and on January 26, 2026, he formally announced the end of curl's bug bounty program on his blog [12]. A GitHub gist published by Stenberg provides documentary evidence of the actual AI slop submissions curl received, moving the claim from assertion to verifiable record [13]. Bugcrowd reported that bug bounty submissions more than quadrupled over a three-week period in March 2026, with most proving false [14], and RedMonk framed the dynamic as a 'vulnerability treadmill' threatening the economics of coordinated vulnerability disclosure [15]. A complicating thread: a post attributed to Stenberg indicates that a company called Mythos separately identified five real vulnerabilities in curl [16] — raising the question of whether structured AI-assisted security research can succeed where bounty-hunting submissions failed, and whether Stenberg's prior claim that no valid AI-assisted report had ever been received requires qualification.

The tools positioned to address AI content verification — AI writing detectors — are now condemned by a multi-institutional body of evidence as systematically biased against non-native English speakers. Stanford HAI's research documented that AI detectors produce false positives at higher rates for non-native English writers [17]; an NIH-published peer-reviewed study confirmed the finding specifically for GPT detectors [18]; the Center for Democracy and Technology documented disproportionate effects on English learners in a policy brief [19]; and Turnitin's own blog engaged with the bias question regarding its own tool [20]. Reddit discussions document students directly experiencing wrongful accusations [21]. The cumulative picture is not a technical gap that better tools might close but a structural limitation — a conclusion now corroborated by enough institutional sources that 'AI detector bias' has moved from a research finding to a documented policy problem that equitable enforcement cannot be built on top of.

Timeline

  • 2025-05-07: Curl founder Daniel Stenberg publicly criticizes AI slop bug submissions; curl states it has still not seen a single valid AI-assisted security report [9][10][11]
  • 2025-07-24: TechCrunch reports AI slop and fake reports are exhausting bug bounty programs industry-wide, before curl's formal program termination [8]
  • 2025-11-17: Minnesota Lawyer reports courts are beginning to test a new frontier of defamation law as AI-hallucinated content enters litigation [2]
  • 2026-01-26: Daniel Stenberg formally announces the end of curl's bug bounty program on his blog; a GitHub gist documenting actual AI slop submissions received is published [12][13]
  • 2026-02-03: AI-hallucinated case citations prompt court sanctions and delay a class action settlement [6]
  • 2026-03: Bugcrowd reports bug bounty submissions more than quadrupled over three weeks, with most proving to be false AI-generated reports; some companies suspend programs entirely [14]
  • 2026-04-24: Helsell Fetterman review documents ongoing costs to lawyers from AI hallucinations in court filings [7]
  • 2026-05-05: RedMonk analyst publishes 'AI Slop & the Vulnerability Treadmill,' framing AI-generated false reports as a structural economic threat to coordinated vulnerability disclosure [15]
  • 2026-05-22: New York Times investigation finds AI-hallucinated quotes in Rosenbaum's The Future of Truth; Kara Swisher and Lisa Feldman Barrett publicly deny the attributed quotes [1]
  • 2026-05-23: Rohan Paul summarizes research arguing AI writing detectors fail due to fundamental statistical limitations of human writing variation, not classifier quality alone [26]
  • 2026-05-24: Social commentary notes AI misinformation is outpacing fact-checking by hours and that detection methods journalists rely on 'keep expiring' [27]

Perspectives

Ron Rosenbaum (author, The Future of Truth)

Acknowledges AI-hallucinated quotes were an error and is conducting a citation audit, but continues to defend AI research tools and their role in his workflow

Evolution: Consistent through the incident: apologetic about the specific failure, not about the broader practice

Kara Swisher (tech journalist, attributed quote subject)

Direct denial — states the quote attributed to her is something she never said

Evolution: Consistent; victim of fabricated attribution with no stated prior position on AI research tools

Lisa Feldman Barrett (Northeastern University professor, attributed quote subject)

States the attributed quotes do not appear in her book and are also factually wrong — repudiating both the attribution and the substance

Evolution: Consistent; victim of fabricated attribution

Daniel Stenberg / curl project

Formally ended curl's bug bounty program in January 2026 after AI slop made it unworkable; published a GitHub gist as verifiable documentary evidence of the fake submissions; separately acknowledged that Mythos identified five real vulnerabilities in curl

Evolution: Hardened from initial public criticism to formal program termination with primary-source documentation; the Mythos acknowledgment adds nuance to his prior blanket claim that no valid AI-assisted security report had ever been received

Bugcrowd (bug bounty platform)

Reports empirical data showing a dramatic spike in AI-generated false submissions and signals severe operational burden

Evolution: Consistent industry concern; March 2026 scale data shows the problem has grown substantially since the curl case in 2025

RedMonk (technology analyst firm)

Frames AI-generated false reports as a 'vulnerability treadmill' that fundamentally threatens the economics of coordinated vulnerability disclosure

Evolution: Consistent since first appearance in thread; adds an industry analyst framing to what had been primarily platform-operator and maintainer complaints

Legal scholars and law firm practitioners (defamation and AI liability)

AI hallucinations fabricating attributed statements are actionable defamation; courts are testing this frontier; Quinn Emanuel, Thompson Coburn, and Helsell Fetterman warn of legal exposure for authors and lawyers alike

Evolution: Expanded this pass: previously framed primarily around defamation doctrine for published attributed content; now encompasses active court sanctions (Rule 11 violations) for hallucinated citations in filings as a parallel already-live enforcement track

Stanford HAI / NIH / CDT researchers (AI detector bias)

AI writing detectors produce false positives at systematically higher rates for non-native English speakers — a finding now confirmed across multiple major research institutions, with Turnitin's own blog engaging the question about its tool

Evolution: Evolved from a single research study (previous pass) to a multi-institutional consensus with peer-reviewed journal publication, Stanford HAI report, CDT policy brief, and direct student experience documentation

Rohan Paul and NLP researchers (AI detector structural limits)

AI detection tools fail for a reason that cannot be engineered away: human writing is statistically too varied, making false positives structurally unavoidable regardless of classifier quality

Evolution: Consistent skepticism; now reinforced by the multi-institutional bias documentation as independent corroboration from a different angle

Tensions

  • Rosenbaum defends continued AI tool use in research even after his book about AI misinformation was found to contain AI-hallucinated fabrications — a position Swisher and Barrett implicitly contest by naming concrete reputational harm done to them as real individuals falsely quoted, and that legal practitioners contest by framing such fabrications as potentially actionable defamation with personal legal exposure [1][2][3][22][23][4]
  • Institutions enforcing academic integrity via AI detectors assume the tools are reliable enough to act on, while Stanford HAI, NIH, and CDT research documents that false positives are structurally unavoidable and disproportionately burden non-native English speakers — making detector-based enforcement both technically and ethically contested [26][17][18][19][20][25]
  • Daniel Stenberg terminated curl's bug bounty because AI slop made it unworkable and stated no valid AI-assisted security report had ever been received — but separately acknowledged that Mythos identified five real curl vulnerabilities, suggesting a tension between the bounty-hunting context that produces slop and structured security research that may operate differently [12][10][16]
  • Legal scholars frame 'defamation by hallucination' as structurally inevitable and actionable — implying ongoing legal exposure for AI vendors, authors, and publishers — while AI tool advocates (represented implicitly by Rosenbaum's continued defense) frame individual errors as correctable edge cases rather than systemic liabilities [22][23][1][3][4]
  • Courts are already sanctioning lawyers under existing Rule 11 mechanisms for AI-hallucinated citations — an active enforcement track with documented sanctions and settlement delays — while the legal profession has no agreed standard for AI verification before filing, creating a gap between accountability and guidance [6][5][7]

Sources

  1. [1] AI put "synthetic quotes" in his book. But this author wants to keep using it. — Ars Technica AI (2026-05-22)
  2. [2] Courts test new frontier of defamation law as AI enters mix — reactive:ai-content-integrity
  3. [3] Client Alert: Defamation in the AI Era — reactive:ai-content-integrity
  4. [4] AI Hallucinated Me. If You Are a Writer, It May Hallucinate You | Thompson Coburn LLP — reactive:ai-content-integrity
  5. [5] AI IP Year in Review - AI Hallucinations in Court Filings and Orders — reactive:ai-legal-hallucination
  6. [6] AI Hallucinated Case Citations Prompt Sanctions And Delay Class Action Settlement – Class Action Defense — reactive:ai-content-web-degradation
  7. [7] AI Hallucinations Keep Costing Lawyers in Court | Helsell Fetterman — reactive:ai-content-integrity
  8. [8] AI slop and fake reports are coming for your bug bounty programs — reactive:ai-content-integrity
  9. [9] Curl eliminates bug bounty program due to AI slop - CSO Online — reactive:ai-content-integrity
  10. [10] Curl: We still have not seen a valid security report done with AI help | Hacker News — reactive:ai-content-integrity
  11. [11] Curl project founder snaps over deluge of time-sucking AI slop bug ... — reactive:ai-content-integrity
  12. [12] The end of the curl bug-bounty | daniel.haxx.se — reactive:ai-content-integrity
  13. [13] AI slop security reports submitted to curl · GitHub — reactive:ai-content-integrity
  14. [14] Bug bounty businesses bombarded with AI slop — Ars Technica AI (2026-05-18)
  15. [15] AI Slop & the Vulnerability Treadmill – console.log() — reactive:ai-content-integrity
  16. [16] curl founder Daniel Stenberg says Mythos identified five ... - Facebook — reactive:ai-content-integrity
  17. [17] AI-Detectors Biased Against Non-Native English Writers | Stanford HAI — reactive:ai-content-integrity
  18. [18] GPT detectors are biased against non-native English writers — reactive:ai-content-integrity
  19. [19] Disproportionate Effects of Generative AI-Detectors on English ... — reactive:ai-content-integrity
  20. [20] Is Turnitin's AI Detector Biased Against Non-Native English Writers? — reactive:ai-content-integrity
  21. [21] ESL students are getting falsely flagged by AI detectors and ... - Reddit — reactive:ai-content-integrity
  22. [22] Inevitable Errors: Defamation by Hallucination in AI Reasoning Models — reactive:ai-content-integrity
  23. [23] Courts Navigating AI Defamation Opens Legal Risks for Companies — reactive:ai-content-integrity
  24. [24] Study Reveals AI Detectors' False Positives on Non-Native Writers — reactive:ai-content-integrity
  25. [25] The Problem with False Positives: AI Detection Unfairly Accuses ... — reactive:ai-content-integrity
  26. [26] AI detectors fail because student writing is too varied to judge from 1 document. — Rohan Paul Twitter (2026-05-23)
  27. [27] AI misinformation now outpaces fact-checking by hours, the detection methods journalists rely on keep expiring, and the ... — reactive:ai-content-integrity (2026-05-24)