AI-Generated Content: Hallucination, Deliberate Misuse, and Undetectability · history
Version 2
2026-05-24 04:29 UTC · 59 items
What
Three converging failures of AI-generated content are reshaping publishing, cybersecurity, and education simultaneously. In publishing, journalist Ron Rosenbaum's The Future of Truth — explicitly about AI misinformation — was found to contain fabricated quotes attributed to real named individuals, both of whom publicly denied the statements [1]. In cybersecurity, AI-generated false vulnerability reports have not only overwhelmed bug bounty platforms (Bugcrowd reported submissions quadrupling over three weeks in March 2026 [5]) but already drove one high-profile termination: the curl open-source project eliminated its bug bounty in May 2025 after its founder described an unmanageable deluge of AI junk, with curl stating it had still not seen a single valid AI-assisted security report [6][7][8]. Underpinning both: AI writing detectors are structurally limited by human writing variation [11], while legal scholars have begun constructing a formal 'defamation by hallucination' liability framework as courts start to navigate who bears legal exposure for AI fabrications [2][3].
Why it matters
The harms are no longer hypothetical — fabricated quotes damage named individuals' reputations, security infrastructure is being dismantled rather than patched, and the legal system is beginning to formalize liability. If 'defamation by hallucination' becomes enforceable doctrine, it would shift AI content failures from reputational problems into legal ones, potentially restructuring accountability for AI vendors, authors, and publishers alike.
Open questions
Will courts develop workable standards for 'defamation by hallucination' — and would liability fall on AI tool vendors, authors, or publishers? [2][3]
As named projects like curl have already eliminated their bug bounties [6][7], will other open-source maintainers follow, and what mechanism realistically replaces coordinated vulnerability disclosure at scale?
If AI detectors are structurally incapable of reliable detection [11] and disproportionately flag non-native English speakers and certain writing styles [15], what alternative institutional mechanisms will schools and publishers adopt that are both reliable and equitable?
Does Rosenbaum's self-described 'citation audit' establish a replicable publisher standard, or remain a one-off correction with no systemic effect on AI-assisted nonfiction? [1]
Narrative
Three incidents in May 2026 together reveal a widening gap between the pace of AI-generated content and society's ability to verify or filter it — a gap now visible across publishing, cybersecurity, and education simultaneously, and increasingly entering legal territory.
The most striking publishing case centers on journalist Ron Rosenbaum's book The Future of Truth: How AI Reshapes Reality — a work explicitly about how AI 'bends, blurs, and synthesizes' truth — which was found by a New York Times investigation to contain fabricated quotes attributed to real named individuals [1]. Tech journalist Kara Swisher stated she 'never said' the quote attributed to her; Northeastern University professor Lisa Feldman Barrett noted the quotes attributed to her 'don't appear in [my] book, and they are also wrong' — a two-level repudiation covering both attribution and substance [1]. Rosenbaum has acknowledged the error and launched a 'citation audit' ahead of future editions while continuing to defend AI tools in his research workflow. That defense is now being scrutinized not only reputationally but legally: scholars have begun framing such fabrications under a concept they call 'defamation by hallucination,' with one Florida Law Review article titled 'Inevitable Errors: Defamation by Hallucination in AI Reasoning Models' arguing that such errors are structurally predictable rather than exceptional [2]. Bloomberg Law has reported that courts are actively navigating what legal risks this creates for companies [3], and legal observers at Thompson Coburn LLP specifically warned writers that AI hallucination of attributed statements creates personal legal exposure [4].
In cybersecurity, the problem of AI-generated false vulnerability reports has moved from a warning into a documented operational crisis with named casualties. Bug bounty platforms — which pay researchers to find and report software vulnerabilities — have been swamped by AI-generated submissions that prove false or near-worthless. Bugcrowd reported that its submission volume more than quadrupled over a three-week period in March 2026, with most reports proving false [5], and some companies suspended their programs entirely rather than absorb the triage burden. The curl open-source project — a foundational internet tool used by billions of devices — had already reached this breaking point in May 2025, when its founder publicly criticized AI slop submissions and curl eliminated its bug bounty program, stating it had 'still not seen a valid security report done with AI help' [6][7][8]. Industry analysts at RedMonk characterized the dynamic as a 'vulnerability treadmill,' warning that the noise burden is fundamentally altering the economics of coordinated vulnerability disclosure [9]. A parallel observation notes that while many researchers are now using AI tools for bug bounty work, almost none are using them in ways that produce valid output [10] — suggesting the problem is not simply volume but a mismatch between AI capability and the demands of genuine security research.
Meanwhile, research summarized by AI commentator Rohan Paul argues that tools meant to detect AI-generated text are failing for a reason that cannot be engineered away: human writing itself is statistically too varied [11]. The paper reframes AI detection as a hypothesis-testing problem rather than a classification task, concluding that many real students write in ways statistically indistinguishable from AI output — making false positives structurally unavoidable regardless of classifier quality [11]. Multiple academic and institutional sources have echoed this conclusion [12][13][14], and a ResearchGate study has specifically documented that the false-positive problem is not evenly distributed: scholars from non-native English-speaking backgrounds and those with certain writing styles are disproportionately flagged, adding an equity dimension to what had been framed primarily as a technical limitation [15]. The cumulative implication is that better models alone cannot solve the detection problem — the institutional enforcement mechanisms currently relying on these tools are built on a foundation that research now calls structurally flawed.
Timeline
- 2025-05-07: curl project founder publicly criticizes AI slop bug submissions; curl eliminates its bug bounty program, stating it has still not seen a single valid AI-assisted security report [6][7][8]
- 2026-03: Bugcrowd reports bug bounty submissions more than quadrupled over three weeks, with most proving to be false AI-generated reports; some companies suspend programs entirely [5]
- 2026-05-05: RedMonk analyst publishes 'AI Slop & the Vulnerability Treadmill,' framing AI-generated false reports as a structural economic threat to coordinated vulnerability disclosure [9]
- 2026-05-18: Ars Technica reports on AI-generated false vulnerability reports overwhelming bug bounty platforms industry-wide [5]
- 2026-05-22: New York Times investigation finds AI-hallucinated 'synthetic quotes' in Rosenbaum's The Future of Truth; Kara Swisher and Lisa Feldman Barrett publicly deny the attributed quotes [1]
- 2026-05-23: Rohan Paul summarizes research arguing AI writing detectors fail due to fundamental statistical limitations of human writing variation, not classifier quality alone [11]
- 2026-05-24: Social commentary notes AI misinformation is outpacing fact-checking by hours and that detection methods journalists rely on 'keep expiring' [22]
Perspectives
Ron Rosenbaum (author, The Future of Truth)
Acknowledges AI-hallucinated quotes were an error and is conducting a citation audit, but continues to defend AI research tools and their role in his workflow
Evolution: Consistent through the incident: apologetic about the specific failure, not about the broader practice
Kara Swisher (tech journalist, attributed quote subject)
Direct denial — states the quote attributed to her is something she never said
Evolution: Consistent; victim of fabricated attribution with no stated prior position on AI research tools
Lisa Feldman Barrett (Northeastern University professor, attributed quote subject)
States the attributed quotes do not appear in her book and are also factually wrong — repudiating both the attribution and the substance
Evolution: Consistent; victim of fabricated attribution
Bugcrowd (bug bounty platform)
Reports empirical data showing a dramatic spike in AI-generated false submissions and signals severe operational burden
Evolution: Consistent industry concern; the March 2026 scale data shows the problem has grown substantially since the curl case in 2025
curl project / Daniel Stenberg (open-source maintainer)
Eliminated curl's bug bounty program entirely due to AI slop; states no valid AI-assisted security report has ever been received
Evolution: First named specific case in thread; represents a harder terminus than platform complaints — a full program termination by a foundational open-source project
RedMonk (technology analyst firm)
Frames AI-generated false reports as a 'vulnerability treadmill' that fundamentally threatens the economics of coordinated vulnerability disclosure
Evolution: First appearance in thread; adds an industry analyst framing to what had been primarily platform-operator and maintainer complaints
Rohan Paul and paper authors (AI/NLP researchers)
Skeptical of AI detection tools as a viable institutional solution; argues the failure is a structural statistical limit, not a technical gap that better classifiers can close
Evolution: Consistent skepticism; now corroborated by multiple institutional and academic sources, and an equity dimension has been added by separate research on disparate false-positive rates
Legal scholars (defamation and AI liability researchers)
Argue that AI hallucination fabricating statements attributed to real people constitutes actionable defamation; courts are beginning to navigate liability questions for vendors, authors, and publishers
Evolution: New voice in thread; converts the previously open question about legal liability into an active area of legal scholarship and court navigation
Thompson Coburn LLP (legal advisory)
Warns writers specifically that AI hallucinating attributed statements creates personal legal exposure — framing the Rosenbaum-type incident as a legal risk, not merely a reputational one
Evolution: New voice in thread; bridges the publishing and legal domains
Tensions
- Rosenbaum defends continued AI tool use in research even after his book about AI misinformation was found to contain AI-hallucinated fabrications — a position Swisher and Barrett implicitly contest by naming the concrete reputational harm done to them as real individuals falsely quoted, and that legal scholars contest by framing such fabrications as potentially actionable defamation [1][2][3]
- Institutions enforcing academic integrity via AI detectors assume the tools are reliable enough to act on, while researchers argue false positives are structurally unavoidable and separately document that the burden falls disproportionately on non-native English speakers — making detector-based enforcement both technically and ethically contested [11][15][20]
- Bug bounty platforms and open-source maintainers like curl have responded to AI slop by terminating programs, while security analysts and researchers argue this eliminates legitimate vulnerability disclosure channels with no clear replacement proposed [5][6][7][21][9]
- Legal scholars frame 'defamation by hallucination' as inevitable and actionable — implying AI vendors and authors bear ongoing legal exposure — while AI tool advocates (represented implicitly by Rosenbaum's continued defense) frame individual errors as correctable edge cases rather than systemic liabilities [2][3][1][4]
Sources
- [1] AI put "synthetic quotes" in his book. But this author wants to keep using it. — Ars Technica AI (2026-05-22)
- [2] Inevitable Errors: Defamation by Hallucination in AI Reasoning Models — reactive:ai-content-integrity
- [3] Courts Navigating AI Defamation Opens Legal Risks for Companies — reactive:ai-content-integrity
- [4] AI Hallucinated Me. If You Are a Writer, It May Hallucinate You | Thompson Coburn LLP — reactive:ai-content-integrity
- [5] Bug bounty businesses bombarded with AI slop — Ars Technica AI (2026-05-18)
- [6] Curl eliminates bug bounty program due to AI slop - CSO Online — reactive:ai-content-integrity
- [7] Curl: We still have not seen a valid security report done with AI help | Hacker News — reactive:ai-content-integrity
- [8] Curl project founder snaps over deluge of time-sucking AI slop bug ... — reactive:ai-content-integrity
- [9] AI Slop & the Vulnerability Treadmill – console.log() — reactive:ai-content-integrity
- [10] Everyone Is Using AI for Bug Bounty in 2026. Almost Nobody Is ... — reactive:ai-content-integrity
- [11] AI detectors fail because student writing is too varied to judge from 1 document. — Rohan Paul Twitter (2026-05-23)
- [12] The Imperfection of AI Detection Tools - HumTech - UCLA — reactive:ai-content-integrity
- [13] Can we trust academic AI detective? Accuracy and limitations of AI-output detectors - PMC — reactive:ai-content-integrity
- [14] Detecting AI May Be Impossible. That's a Big Problem For Teachers. — reactive:ai-content-integrity
- [15] The Problem with False Positives: AI Detection Unfairly Accuses ... — reactive:ai-content-integrity
- [16] "Defamation in the Age of Artificial Intelligence" by Leslie Y. Garfield Tenzer — reactive:ai-content-integrity
- [17] Redefining Defamation: Establishing Proof of Fault for Libel and Slander in AI Hallucinations — Columbia Undergraduate Law Review — reactive:ai-content-integrity
- [18] Defamation by Hallucination - Torts - Jotwell — reactive:ai-content-integrity
- [19] AI Hallucinated Me. If You Are a Writer, It May Hallucinate You | Thompson Coburn LLP - JDSupra — reactive:ai-content-integrity
- [20] Why AI Detectors Fail: The False Positive Crisis in Education | Dr. Yusuf Akhter posted on the topic | LinkedIn — reactive:ai-content-integrity
- [21] Is Bug Bounty Dead? AI, Slop Reports, and the Future of Security ... — reactive:ai-content-integrity
- [22] AI misinformation now outpaces fact-checking by hours, the detection methods journalists rely on keep expiring, and the ... — reactive:ai-content-integrity (2026-05-24)