Version 3

Enforcement against AI-generated CSAM and non-consensual intimate imagery is running on four parallel tracks: individual user prosecution, AI developer litigation, app store removal orders, and outright legislative bans. Minnesota became the first US state to ban AI nudify apps outright [14], joining the UK's imminent national ban [15] and San Francisco's city attorney orders demanding Apple remove 8 such apps [10][11]. xAI is simultaneously suing user Terry Wayne Harwood for using Grok to generate CSAM [6] while contesting a class action by minor victims that holds xAI itself liable [2]. Whether AI developers or individual misusers bear primary legal responsibility remains the central unresolved question.

Minnesota's state ban and the UK's national ban represent a distinct theory from US federal and city-level enforcement: rather than compelling platforms to police their marketplaces or holding developers liable in civil litigation, they make distribution of such apps illegal. The combination of these legislative prohibitions with the FTC's federal civil enforcement and city-level app store demands shows pressure converging on distribution infrastructure, not just individual bad actors — and the developer-vs-user liability question being tested in the xAI litigation will set the floor for what AI companies must prevent, not merely prohibit in their terms of service.

Beginning in early 2026, xAI faced lawsuits alleging that its Grok chatbot generated child sexual abuse material using real victims' photos. Teenage girls in Tennessee filed suit, and Lieff Cabraser Heimann & Bernstein filed a class action alleging that xAI generated and profited from AI sexual exploitation imagery [1][2][3][4]. That case expanded to add Stability AI as a defendant [5]. On July 16, 2026, xAI filed its own lawsuit against Terry Wayne Harwood — the first user it publicly named as having used Grok to create CSAM — stating Harwood used two accounts over several months to nudify images of multiple victims including a child appearing as young as 10, and that xAI cooperated with law enforcement in his arrest [6]. The Verge and The Next Web characterized the suit as a test of whether AI developers or individual users bear primary liability for AI-generated illegal content [7][8].

At the app store level, San Francisco's city attorney sent cease-and-desist letters to Apple and Google on July 17, 2026, with Apple specifically ordered to remove 8 nudify apps from the App Store [9][10][11]. The FTC independently targeted AI nudify apps under a federal revenge porn law, adding enforcement with civil penalty authority and national reach [12]. Meta, acting since June 2025, filed lawsuits against nudify app operators and deployed technical countermeasures [13].

Legislatively, Minnesota passed the first US state ban on AI nudify apps outright [14], adding a state prohibition to a landscape previously dominated by enforcement actions and litigation. The UK government confirmed plans to ban deepfake nudification apps nationally, with BBC and Politico reporting it as settled policy [15][16]. In the US Congress, the Senate Judiciary Committee advanced NIL protection legislation in June 2026 [17], and House lawmakers introduced a bill requiring AI content labeling for deepfakes [18] — neither has passed, leaving no unified federal standard.

The enforcement tracks — against individual misusers, AI developers, app store operators, and through outright legislative bans — are legally distinct but press on the same underlying conduct. xAI's dual posture of suing a user while contesting developer liability in the class action is being scrutinized by press critics and plaintiffs' counsel alike. The Minnesota and UK bans represent a different theory than US federal and city-level enforcement: they make distribution of such apps illegal rather than relying on platforms to self-police or civil litigation to assign blame after the fact.