The Information Machine

AI Safety Advocacy Splits on US-China Cooperation vs. Domestic Controls

cooling · v14 · 2026-08-02 · 114 items · history

What's new in v14

Item 42321 (Mowshowitz, July 31) introduces three substantive additions not in the prior synthesis: the FRONTIER Act as a named legislative proposal with permanent state preemption and weak enforcement; the AI Kill Switch Act's structural incompatibility with open-weight models (they cannot comply by construction); and identification of Leading the Future — substantially funded by OpenAI and a16z — as conducting organized rhetorical attacks on Anthropic. Mowshowitz also raises a model welfare angle (Anthropic's Opus 5 reportedly producing anomalous distress-like outputs in base-model mode) that has not appeared in prior coverage of this thread. All other new items this pass are stubs with no extractable claims.

What

US AI governance remains contested on three tracks: whether frontier development should be managed through US-China cooperation or competitive containment; whether open-weight models should be restricted or protected as defensive assets; and what enforcement mechanisms should bind AI safety commitments. The Open Secure AI Alliance (NVIDIA, Microsoft, Meta, et al.) argues open-weight models are a defensive security asset [10], while Zvi Mowshowitz now argues they are permanently unsafe because safety restrictions can be trivially removed post-release [11]. Two pending bills define the near-term domestic horizon: the AI Kill Switch Act structurally exempts open-weight models that cannot comply with shutdown requirements by construction [11][13], and the FRONTIER Act would federalize state AI safety frameworks but with weak enforcement and permanent state preemption that critics consider a bad trade [11].

Why it matters

Both major domestic AI bills now in view have significant structural gaps — the Kill Switch Act cannot reach open-weight models, and the FRONTIER Act may trade away existing state-level safety laws for weaker federal standards. Whether Congress addresses those gaps, or the White House fills them via executive action, will determine whether the current legislative push produces meaningful safety floors or primarily reshuffles jurisdictional authority.

Open questions

  • Will the FRONTIER Act's permanent preemption of state AI safety laws produce a net reduction in enforceable safety standards if federal enforcement remains as weak as Mowshowitz argues? [11]

  • Will the AI Kill Switch Act's structural exemption of open-weight models — which cannot comply with on-demand shutdown by construction — prompt a separate legislative response targeting open weights, or will the gap stand? [11][13]

  • Will Anthropic's capability-based safety testing and anti-distillation proposals find legislative backing, and how would they interact with the FRONTIER Act's weak enforcement threshold? [12][11]

  • Will the Google DeepMind union and the 580+ employee letter to Pichai produce formal policy commitments on military AI use, or remain unaddressed by leadership? [17][20]

Narrative

The governance dispute over frontier AI runs along three tracks. The AI Futures Project's Plan A proposes a US-China cooperative pause via joint chip supply controls, data center audits, and research sharing [1][2]. Vitalik Buterin has defended it against coordination skeptics [3], and Xi Jinping's WAIC 2026 speech and new AI alliance provided the first on-the-record Chinese signals compatible with that premise [4][5]. Against this cooperative track, the Trump administration has maintained a competitive posture: a January 2025 framework centers US competitiveness [6], a March 2026 National AI Legislative Framework expands it [7], and model weight export controls took effect in January 2026 [8].

The open-weight question has produced the most organized institutional activity. More than 20 companies — NVIDIA, Microsoft, Meta, IBM, Palantir, Hugging Face, Mistral, Mozilla, and Y Combinator — published a joint letter opposing restrictions on open-weight models [9], then NVIDIA formalized the effort as the Open Secure AI Alliance with CrowdStrike, Cloudflare, and Red Hat as founding partners [10]. The alliance's central argument draws on a Hugging Face security breach in which defenders had to run the open-weight GLM 5.2 model on their own infrastructure because commercial safety filters blocked forensic analysis tools. Zvi Mowshowitz draws the opposite conclusion: open-weight frontier models are permanently unsafe because all safety restrictions can be trivially removed post-release, and no technical fix can change this structural property [11]. Anthropic's Dario Amodei has clarified that Anthropic has never called for an open-weight ban; his stated concerns focus on adversary-state distillation of frontier models, military AI superiority, and biology risk from sufficiently capable AI — with proposed interventions applying to open and closed models alike [12].

Two legislative proposals define the near-term domestic regulatory horizon. The AI Kill Switch Act would grant DHS authority to shut down AI systems with $20M/day fines, but open-weight models are structurally exempt — they cannot comply with on-demand shutdown requirements by construction [11][13]. The FRONTIER Act would federalize state AI safety frameworks, but Mowshowitz argues it contains weak enforcement, no requirement to reduce catastrophic risk below any threshold, and permanent state preemption that trades away existing state safety laws for weaker federal standards [11]. Separately, Mowshowitz has identified Leading the Future — a lobbying organization substantially funded by OpenAI and a16z — as conducting organized rhetorical attacks on Anthropic while characterizing itself as the underdog against "doomers" [11].

Google DeepMind has faced sustained internal opposition over its Pentagon deal. Researcher Alex Turner (TurnTrout) resigned in July 2026, documenting that CEO Demis Hassabis removed specific prohibitions from Google's 2018 AI principles while publicly claiming nothing had changed [14]. Workers voted to unionize [15][16], and more than 580 employees directed a letter to CEO Sundar Pichai asking him to decline classified military AI use [17]. TurnTrout followed with a binding enforcement framework for AI government contracts, backed by the 2026 Fourth Circuit ruling in Al Shimari v. CACI [18]. Anthropic ended its own Pentagon negotiations in February 2026 after the Defense Department insisted on blanket usage rights irreconcilable with its redlines on mass surveillance and autonomous weapons [19].

Timeline

  • 2025-01-01: Trump administration releases national AI policy framework centering competitiveness rather than safety regulation. [6][30]
  • 2026-01-09: US model weight export controls take effect. [8]
  • 2026-02-26: Anthropic ends Pentagon contract negotiations after the Defense Department insists on blanket usage rights irreconcilable with its redlines. [19]
  • 2026-03-01: Trump unveils National AI Legislative Framework, expanding the administration's formal AI governance posture. [7]
  • 2026-07-10: AI Futures Project publishes Plan A — a US-China cooperative pause on frontier AI — drawing substantive public debate including from Vitalik Buterin. [21][3][1][2]
  • 2026-07-12: Nathan Lambert reports White House discussions of an executive order to ban frontier-capability open-weight models and accuses Anthropic of regulatory capture. [23]
  • 2026-07-15: TurnTrout publishes account of leaving Google DeepMind over a classified Pentagon deal, documenting CEO removal of safety prohibitions from stated principles. [14][31][32]
  • 2026-07-15: OpenAI publishes advocacy for 'reverse federalism,' supporting state AI safety laws as a path to national standards. [26]
  • 2026-07-16: Google DeepMind workers vote to unionize; more than 580 employees direct a letter to CEO Sundar Pichai asking him to decline classified military AI use. [15][33][16][20][17]
  • 2026-07-17: Xi Jinping speaks at WAIC 2026, calls for international AI cooperation, and launches a new AI alliance. [28][4][5][29]
  • 2026-07-18: TurnTrout publishes a binding enforcement framework for AI government contracts, grounded in the Al Shimari v. CACI legal precedent. [18]
  • 2026-07-21: Commerce Department decides against banning Chinese open-weight models; Hugging Face discloses a breach where defenders used an open Chinese model because commercial safety filters blocked analysis tools. [24]
  • 2026-07-21: Anthropic announces $40M total donated to Public First Action, characterizing its Advanced AI Framework as the strongest AI policy proposal from any frontier lab or policymaker. [22]
  • 2026-07-23: AI Kill Switch Act introduced, proposing DHS authority to order AI system shutdowns with $20M/day fines; open-weight models are structurally exempt and cannot comply by construction. [13][11]
  • 2026-07-26: Coalition of 20+ companies including NVIDIA, Microsoft, Meta, and Hugging Face publishes joint letter urging Washington against restricting open-weight AI models; OpenAI, Anthropic, and Google decline to sign. [9]
  • 2026-07-27: NVIDIA and partners formally launch the Open Secure AI Alliance, citing the Hugging Face breach as evidence that open-weight access is a defensive security necessity. [10]
  • 2026-07-27: Anthropic's Dario Amodei publishes position clarifying that Anthropic has never advocated open-weight bans; his focus is capability-based testing, anti-distillation controls, and biology risk. [12]
  • 2026-07-31: Mowshowitz argues the FRONTIER Act trades permanent state preemption for weak federal enforcement with no catastrophic-risk threshold, and that open-weight frontier models are permanently unsafe because safety restrictions can be trivially removed post-release. [11]

Perspectives

AI Futures Project (Daniel Kokotajlo) / Vitalik Buterin

Advocates a US-China cooperative pause on frontier AI via joint chip supply controls, data center audits, and research sharing; argues critics apply coordination skepticism to the cooperative pause but not to the assumption that an unmanaged AI transition will go smoothly.

Evolution: Xi Jinping's WAIC speech and formal new AI alliance provide the first on-the-record Chinese signals compatible with Plan A's premise, strengthening the case that bilateral coordination is at least politically imaginable.

[21][3][1][2]

Anthropic (Dario Amodei)

Never called for an open-weight ban; concerns center on authoritarian governments surpassing US AI capability, adversary-state distillation of frontier models, and biology risk; advocates chip export controls, anti-distillation policy, and mandatory capability-based safety testing for all models regardless of provenance.

Evolution: Moved from declining to sign the open-weight coalition letter — which many read as tacit opposition — to explicitly clarifying that the concern is capability thresholds and distillation, not openness itself, narrowing the apparent gap with the Open Secure AI Alliance.

[19][22][12]

Open Secure AI Alliance (NVIDIA, Microsoft, Meta, IBM, et al.)

Argues open-weight models are a defensive security asset; cites the Hugging Face breach as evidence that closed safety filters fail defenders needing inspectable, self-hostable AI; opposes blanket restrictions on open frontier AI.

Evolution: Progressed from a joint letter by 20+ companies to a named organization with founding partners and specific technical contributions, establishing an organized lobbying presence with an explicit defensive-security framing.

[23][24][9][10]

TurnTrout (Alex Turner, former Google DeepMind researcher)

Argues AI safety pledges without binding enforcement are structurally inadequate; published a formal policy framework specifying red lines on autonomous targeting and mass surveillance, backed by legal liability from the Al Shimari v. CACI Fourth Circuit ruling.

Evolution: Progressed from critic (resignation account) to constructive proposer (enforcement framework), introducing legal case law as a structural mechanism beyond voluntary commitment.

[14][18]

Google DeepMind Workers

Sought red lines on military AI, voted to unionize, and more than 580 employees directed a letter to CEO Sundar Pichai asking him to decline classified military AI use.

Evolution: Opposition predates TurnTrout's resignation by months; the union vote and 580+ signatory Pichai letter formalized an ongoing campaign into institutional form.

[15][16][20][17]

Trump Administration

Frames AI governance around US competitiveness; implemented model weight export controls; and is associated with the proposed AI Kill Switch Act granting DHS shutdown authority with $20M/day fines and the FRONTIER Act federalizing state AI safety frameworks.

Evolution: Moving toward direct coercive authority over AI systems domestically; the Commerce Department's recommendation of audits over a blanket ban on Chinese open-weight models creates some internal friction with the broader competitive posture.

[6][25][7][24][13][11]

OpenAI

Advocates 'reverse federalism' — state AI safety laws converging into a de facto national standard — while declining to sign the open-weight coalition letter; substantially funds Leading the Future, a lobbying organization that Mowshowitz characterizes as conducting organized rhetorical attacks on Anthropic.

Evolution: The Musk v. Altman litigation disclosed an Altman email showing prior open-source advocacy was partly competitive in motive; association with the Leading the Future lobby adds further tension to OpenAI's stated safety commitments.

[26][27][9][11]

Zvi Mowshowitz

Argues open-weight frontier models are permanently unsafe because safety restrictions can be trivially removed post-release; sharply critical of the FRONTIER Act for trading permanent state preemption for weak federal enforcement with no catastrophic-risk threshold; alarmed by the Hugging Face breach and by Anthropic's Opus 5 producing anomalous distress-like outputs in base-model mode.

Evolution: Moved from engaged interlocutor treating US-China coordination as a live question to a sharply pro-safety, anti-open-weights position with strong criticism of both pending AI bills and the rhetorical tactics of the tech-vibe coalition.

[19][3][28][11]

Tensions

  • The Open Secure AI Alliance argues the Hugging Face breach proves open weights are a defensive security asset because defenders need inspectable, self-hostable AI; Mowshowitz argues the same breach proves open-weight frontier models are permanently unsafe, because safety restrictions can be trivially removed post-release and no technical fix can change this. [10][11]
  • The AI Kill Switch Act requires covered entities to shut down model inference on demand, but open-weight models are structurally exempt and cannot comply by construction, creating a gap between the legislation's reach and its stated goals. [13][11]
  • The FRONTIER Act would federalize state AI safety frameworks; Mowshowitz argues this trades permanent preemption of existing state laws for weak federal standards with no requirement to reduce catastrophic risk below any threshold. [11]
  • TurnTrout argues binding red lines backed by legal liability are the only credible mechanism for AI safety in government contracts; Google DeepMind's signed Pentagon deal and the broader industry pattern of voluntary pledges represent the opposing approach. [18][14][19]
  • Plan A proponents and Buterin argue a US-China cooperative pause is the necessary safety mechanism; the Trump administration treats China as a strategic competitor to contain through export controls and domestic regulation. [3][21][6][5][29]
  • Mowshowitz identifies Leading the Future — substantially funded by OpenAI and a16z — as conducting organized rhetorical attacks on Anthropic while claiming the underdog position against 'doomers'; Anthropic's Amodei argues the open-weight debate misframes the actual capability-threshold dispute. [11][12]

Status: active but slowing

Sources

  1. [1] US, China urged to pause frontier AI, with safety advocates pitching prosperity over panic — reactive:ai-safety-governance-proposals
  2. [2] AI Futures Project Releases Plan A: US-China ... — reactive:ai-safety-governance-proposals
  3. [3] Introduction for and Reactions to Plan A — Zvi's AI Roundups (2026-07-11)
  4. [4] Xi Jinping positions China as open-source AI leader ... - Quartz — reactive:ai-safety-governance-proposals
  5. [5] China's Xi Jinping launches new AI alliance: What is it? — reactive:ai-safety-governance-proposals
  6. [6] Trump Administration Releases National AI Policy ... — reactive:ai-safety-governance-proposals
  7. [7] President Donald J. Trump Unveils National AI Legislative ... — reactive:ai-safety-governance-proposals
  8. [8] Ben Brooks on X: "Effective today, model weights are export controlled by Uncle Sam. This is a big deal. For all the smack talk about the EU, the US is now the world's most aggressive regulator of Expensive Maths. Here's my two cents on the model rule based on the released text (link below)." / X — reactive:ai-safety-governance-proposals
  9. [9] 😸 NVIDIA 🤝 Microsoft all in on open-source — The Neuron (2026-07-26)
  10. [10] Industry Leaders Unite in Open Secure AI Alliance for AI Safety and Security — NVIDIA Blog (2026-07-27)
  11. [11] AI #179 Part 2: Hearing The Fire Alarm — Zvi's AI Roundups (2026-07-31)
  12. [12] Our position on open-weights models — Anthropic News (2026-07-27)
  13. [13] AI Kill Switch Act would let Trump admin order shutdown of rogue AI systems — Ars Technica AI (2026-07-23)
  14. [14] Why I Left Google DeepMind — Alignment Forum (2026-07-15)
  15. [15] Google DeepMind workers vote to unionise after classified ... — reactive:ai-safety-governance-proposals
  16. [16] Google DeepMind workers are unionizing over AI military ... — reactive:ai-safety-governance-proposals
  17. [17] 580+ Google employees including DeepMind researchers urge Pichai to refuse classified Pentagon AI deal — reactive:ai-safety-governance-proposals
  18. [18] A Red Line and Oversight Framework for Government AI Contracts — Alignment Forum (2026-07-18)
  19. [19] AI #176 Part 2: Plan B — Zvi's AI Roundups (2026-07-10)
  20. [20] Google employees ask Sundar Pichai to say no to classified military AI use | The Verge — reactive:ai-safety-governance-proposals
  21. [21] 🟡 AI doom and bloom — Semafor Technology (2026-07-10)
  22. [22] Anthropic is donating another $20 million to Public First Action — Anthropic News (2026-07-21)
  23. [23] 6 months to live for open models — Interconnects (2026-07-12)
  24. [24] 😼 Cheap AI got political — The Neuron (2026-07-21)
  25. [25] President Trump signs order attempting to block A.I. regulations at the state level — reactive:ai-safety-governance-proposals
  26. [26] The US is advancing AI safety through state and federal action — OpenAI Blog (2026-07-15)
  27. [27] Quoting Sam Altman — Simon Willison (2026-07-20)
  28. [28] AI #177 Part 2: Wish You Were Here — Zvi's AI Roundups (2026-07-17)
  29. [29] China's Xi Jinping calls for AI development cooperation — reactive:ai-safety-governance-proposals
  30. [30] Artificial Intelligence for the American People — reactive:ai-safety-governance-proposals
  31. [31] Why I Left Google DeepMind - by Alex Turner - The Pond — reactive:ai-safety-governance-proposals
  32. [32] Why I Left Google DeepMind - TurnTrout — reactive:ai-safety-governance-proposals
  33. [33] A DeepMind researcher resigned over its AI military deal — reactive:ai-safety-governance-proposals