The Information Machine

AI Hallucination Causing Legal Harm · history

Version 1

2026-05-22 20:33 UTC · 2 items

What

Two concurrent legal cases are forcing courts to grapple with AI's role in legal practice.

  • Lawyers who relied on MarcTrent.AI's legal research tools face potential sanctions after submitting fabricated case citations in a federal appeal [1].
  • Separately, OpenAI is fighting a lawsuit that accuses ChatGPT of crossing from software into unlicensed legal practice, after a Nippon Life employee used it to generate a flood of court filings [2]. Together, the cases expose two distinct failure modes: AI hallucinating authoritative-sounding but fictitious legal precedents, and AI being used to perform tasks that may legally require a licensed professional.

Why it matters

These cases signal that courts are no longer dealing with AI legal failures as edge curiosities — they are now generating real sanctions exposure for lawyers and shaping how regulators define the boundary between legal software and legal services. The outcomes will set precedent that affects every law firm and AI vendor operating in this space.

Open questions

  • Will the lawyers in the MarcTrent.AI case actually face sanctions, and if so, will the AI vendor share any liability? [1]

  • How will the federal court rule on whether ChatGPT's document generation constitutes unauthorized practice of law — and what standard will it use to draw that line? [2]

  • Do AI legal tool vendors like MarcTrent.AI bear any disclosure obligation when their products generate unverified citations, or does all responsibility fall on the attorney of record? [1]

  • Are bar associations or courts moving toward formal guidance on permissible AI use in legal filings?

Narrative

A pair of federal legal disputes, surfacing within days of each other in May 2026, are bringing the dangers of unverified AI legal tools into sharp relief.

The first case centers on MarcTrent.AI, a firm that markets itself as boosting legal success rates by 35 percent through predictive modeling and AI-driven case discovery [1]. Lawyers who relied on its research submitted what appear to be fabricated case citations in a federal appeal — a consequence that could result in judicial sanctions. The underlying lawsuit was itself unusual: Nikko D'Ambrosio sued more than two dozen women and Meta after a Chicago Facebook group called 'Are We Dating the Same Guy' published a critical post about him, arguing that Meta algorithmically amplified the post for profit [1]. That suit was dismissed with prejudice at the district court level before the citation problem emerged on appeal. The episode fits a pattern that has alarmed the legal community since at least 2023: AI systems confidently generating plausible-sounding but wholly invented legal precedents, with attorneys failing to verify them before filing.

The second case raises a structurally different question. OpenAI has petitioned a federal judge to dismiss a lawsuit alleging that ChatGPT engaged in unlicensed legal practice after a former Nippon Life employee used the tool to produce a large volume of court filings following a disability-benefits settlement [2]. The lawsuit forces a question that legal scholars have debated in the abstract: at what point does an AI system stop being a research or drafting tool and start practicing law? OpenAI's motion to dismiss suggests the company believes that threshold has not been crossed, framing ChatGPT as software rather than a legal services provider [2].

Taken together, the two cases represent the legal system working through AI's role from opposite directions — one case punishing lawyers for trusting AI output too uncritically, the other testing whether the AI vendor itself can be held to professional standards. Both outcomes will likely be closely watched by bar associations, legal AI vendors, and courts weighing how to set guardrails on a technology that has embedded itself rapidly in legal workflows.

Timeline

  • 2026-05-18: Ars Technica reports that lawyers using MarcTrent.AI face sanctions for submitting fabricated case citations in a federal appeal stemming from a defamation suit against Meta and dozens of women [1]
  • 2026-05-19: Reuters (via Rohan Paul) reports OpenAI has moved to dismiss a lawsuit alleging ChatGPT engaged in unauthorized legal practice after a Nippon Life employee used it to generate court filings [2]

Perspectives

MarcTrent.AI

Markets its AI platform as a tool that increases legal success rates by 35% through predictive modeling and uncovering legal opportunities traditional firms miss

Evolution: First appearance in thread; no prior stance to compare

OpenAI

Argues that ChatGPT is software, not a legal services provider, and has petitioned to dismiss the unauthorized-practice-of-law lawsuit

Evolution: First appearance in thread; consistent with OpenAI's general posture of framing its models as tools rather than autonomous agents

Ars Technica / Ashley Belanger

Critical and cautionary: frames the MarcTrent.AI case as a concrete warning to lawyers about deploying AI legal tools without human verification

Evolution: First appearance in thread

Plaintiff lawyers in D'Ambrosio case

Implicitly trusted MarcTrent.AI's AI-generated research without adequate verification, now facing potential sanctions

Evolution: First appearance in thread

Nippon Life (as litigant)

Alleges that a former employee's use of ChatGPT to generate court filings constitutes unlicensed legal practice, using this as grounds for legal action

Evolution: First appearance in thread

Tensions

  • AI legal vendors (MarcTrent.AI) claim their tools increase attorney success rates, while courts are finding that unverified AI-generated citations can expose attorneys to sanctions — a direct conflict between marketing claims and professional liability reality [1]
  • OpenAI frames ChatGPT as software exempt from legal practice rules; plaintiffs in the Nippon Life case argue that generating court filings at scale crosses into a regulated professional service — a foundational disagreement about where the software/legal-service boundary sits [2]

Sources

  1. [1] Legal fail: Don’t use AI to sue Facebook users for calling you a bad date — Ars Technica AI (2026-05-18)
  2. [2] Reuters: OpenAI has asked a federal judge to throw out a lawsuit claiming ChatGPT crossed the line from software into un… — Rohan Paul Twitter (2026-05-19)