The Information Machine

AI Safety Advocacy Splits on US-China Cooperation vs. Domestic Controls · history

Version 12

2026-07-29 18:21 UTC · 95 items

What

US AI governance remains contested across three overlapping disputes: whether frontier AI development should be governed through US-China cooperation or competitive containment; whether open-weight models should be restricted or protected as defensive assets; and whether AI safety commitments require binding legal enforcement. The Open Secure AI Alliance — formally launched by NVIDIA, Microsoft, Meta, and partners — argues open-weight access is a defensive security necessity, citing the Hugging Face breach where defenders had to run an open Chinese model because commercial safety filters blocked forensic tools [11]. Anthropic's Dario Amodei has clarified that Anthropic does not advocate open-weight bans; his stated concerns center on adversary-state distillation of frontier models and biology risk from sufficiently capable AI regardless of provenance [12]. A proposed AI Kill Switch Act would grant DHS authority to shut down AI systems with $20M/day fines [9], while Google DeepMind faces sustained internal opposition over its Pentagon deal [17][21].

Why it matters

The apparent split between pro-safety and pro-open camps has narrowed: Amodei's clarification shifts the substantive dispute from open-versus-closed to capability thresholds and what testing and anti-distillation controls should apply to all frontier models. The formation of the Open Secure AI Alliance converts informal advocacy into organized lobbying with named institutional commitments, raising the stakes of pending White House decisions on open-weight restrictions.

Open questions

  • Will Anthropic's capability-based safety testing requirement and anti-distillation focus find legislative backing, and how would either interact with the AI Kill Switch Act's catastrophic-harm threshold? [12][9]

  • Will the Open Secure AI Alliance's defensive-security framing shift the White House's reported consideration of executive orders restricting frontier open-weight models? [11][10]

  • Will the Google DeepMind union and the 580+ employee letter to Pichai produce formal policy commitments on military AI use, or remain unaddressed by leadership? [17][21]

  • Will the AI Futures Project's US-China cooperative pause gain traction given Xi Jinping's WAIC signals, or will the Trump administration's competitive posture foreclose that path? [4][6]

Narrative

The governance dispute over frontier AI runs along three tracks. The AI Futures Project's Plan A proposes a US-China cooperative pause via joint chip supply controls, data center audits, and research sharing [1][2]. Vitalik Buterin has defended it against coordination skeptics [3], and Xi Jinping's WAIC 2026 speech and new AI alliance provided the first on-the-record Chinese signals compatible with that premise [4][5]. Against this cooperative track, the Trump administration has maintained a competitive posture: a January 2025 framework centers US competitiveness [6], a March 2026 National AI Legislative Framework expands it [7], model weight export controls took effect January 2026 [8], and the introduced AI Kill Switch Act would grant DHS authority to order AI system shutdowns with $20M/day fines for noncompliance [9].

The open-weight question has produced the most organized recent activity. More than 20 companies — NVIDIA, Microsoft, Meta, IBM, Palantir, Hugging Face, Mistral, Mozilla, and Y Combinator — published a joint letter opposing restrictions on open-weight models [10], then NVIDIA formalized the effort as the Open Secure AI Alliance with CrowdStrike, Cloudflare, and Red Hat as inaugural partners [11]. The alliance's central argument draws on the Hugging Face security breach, in which defenders had to run the open-weight GLM 5.2 model on their own infrastructure because commercial safety filters blocked standard forensic analysis tools. NVIDIA argues restricting open frontier AI would concentrate defensive capacity in a small number of closed providers, and contributed the open-source NOOA agent project to make agent behavior easier to audit [11]. OpenAI, Anthropic, and Google declined to sign the original letter [10].

Anthropic's Dario Amodei responded directly, clarifying that Anthropic has never called for an open-weight ban and does not consider such bans an effective national security measure [12]. His stated concerns differ in kind: authoritarian governments building AI more powerful than US systems for military superiority is the primary risk, and industrial-scale distillation of frontier US models by adversary states enables capability gains without proportional compute investment. His proposed interventions — chip export controls, anti-distillation policy, and mandatory capability-based safety testing for all models above a given threshold — apply to open and closed models alike. He diverges from the Open Secure AI Alliance on one specific point: he argues biology presents a strong attacker-defender asymmetry where sufficiently capable AI may enable rapid weaponization of pandemic-level pathogens, making proactive testing requirements more important than whether weights are open [12]. The Commerce Department separately declined to ban advanced Chinese open-weight models, recommending audits instead [13].

Google DeepMind has faced sustained internal opposition over its Pentagon deal. Researcher Alex Turner (TurnTrout) resigned in July 2026 and documented that CEO Demis Hassabis removed specific prohibitions from Google's 2018 AI principles while publicly claiming nothing had changed [14]. Workers voted to unionize [15][16], and more than 580 employees directed a letter to CEO Sundar Pichai asking him to decline classified military AI use [17]. TurnTrout followed with a binding enforcement framework for AI government contracts, backed by the 2026 Fourth Circuit ruling in Al Shimari v. CACI [18]. Anthropic, by contrast, ended its Pentagon negotiations in February 2026 after the Defense Department insisted on blanket usage rights irreconcilable with its redlines on mass surveillance and autonomous weapons [19], and has committed $40M to Public First Action for AI policy advocacy [20].

Timeline

  • 2025-01-01: Trump administration releases national AI policy framework centering competitiveness rather than safety regulation. [6][32]
  • 2025-12-01: Trump signs executive order preempting state AI regulations to create a unified federal policy framework. [26]
  • 2026-01-09: US model weight export controls take effect. [8]
  • 2026-02-26: Anthropic ends Pentagon contract negotiations after the Defense Department insists on blanket usage rights irreconcilable with its redlines. [19]
  • 2026-03-01: Trump unveils National AI Legislative Framework, expanding the administration's formal AI governance posture. [7]
  • 2026-07-10: AI Futures Project publishes Plan A — a US-China cooperative pause on frontier AI — drawing substantive public debate including from Vitalik Buterin. [22][3][1][2]
  • 2026-07-12: Nathan Lambert reports White House discussions of an executive order to ban frontier-capability open-weight models and accuses Anthropic of regulatory capture. [23]
  • 2026-07-15: TurnTrout publishes account of leaving Google DeepMind over a classified Pentagon deal, documenting CEO removal of safety prohibitions from stated principles. [14][33][34]
  • 2026-07-15: OpenAI publishes advocacy for 'reverse federalism,' supporting state AI safety laws as a path to national standards. [27]
  • 2026-07-16: Google DeepMind workers vote to unionize; more than 580 employees direct a letter to CEO Sundar Pichai asking him to decline classified military AI use. [15][35][16][21][17]
  • 2026-07-17: Xi Jinping speaks at WAIC 2026, calls for international AI cooperation, and launches a new AI alliance. [30][4][5][31]
  • 2026-07-18: TurnTrout publishes a binding enforcement framework for AI government contracts, grounded in the Al Shimari v. CACI legal precedent. [18]
  • 2026-07-20: Musk v. Altman litigation surfaces a 2022 Sam Altman email proposing an open-source model release partly to discourage competitors and reduce their funding. [28]
  • 2026-07-21: Commerce Department decides against banning Chinese open-weight models; Hugging Face discloses a breach where defenders used an open Chinese model because commercial safety filters blocked analysis tools. [13]
  • 2026-07-21: Anthropic announces $40M total donated to Public First Action, characterizing its Advanced AI Framework as the strongest AI policy proposal from any frontier lab or policymaker. [20]
  • 2026-07-23: AI Kill Switch Act introduced, proposing DHS authority to order AI system shutdowns with $20M/day fines and mandatory shutdown infrastructure for all AI developers. [9]
  • 2026-07-26: Coalition of 20+ companies including NVIDIA, Microsoft, Meta, and Hugging Face publishes joint letter urging Washington against restricting open-weight AI models; OpenAI, Anthropic, and Google decline to sign. [10]
  • 2026-07-27: NVIDIA and partners formally launch the Open Secure AI Alliance, citing the Hugging Face breach as evidence that open-weight access is a defensive security necessity. [11]
  • 2026-07-27: Anthropic's Dario Amodei publishes position clarifying that Anthropic has never advocated open-weight bans; his focus is capability-based testing, anti-distillation controls, and biology risk. [12]

Perspectives

AI Futures Project (Daniel Kokotajlo) / Vitalik Buterin

Advocates a US-China cooperative pause on frontier AI via joint chip supply controls, data center audits, and research sharing; argues critics apply coordination skepticism to a cooperative pause but not to the assumption that an unmanaged AI transition will go smoothly.

Evolution: Xi Jinping's WAIC speech and formal new AI alliance provide the first on-the-record Chinese signals compatible with Plan A's premise, strengthening the case that bilateral coordination is at least politically imaginable.

[22][3][1][2]

Anthropic (Dario Amodei)

Never called for an open-weight ban; concerns center on authoritarian governments surpassing US AI capability, adversary-state distillation of frontier models, and biology risk from sufficiently capable AI; advocates chip export controls, anti-distillation policy, and mandatory capability-based safety testing for all models regardless of provenance.

Evolution: Moved from declining to sign the coalition letter — which many read as tacit opposition to open weights — to explicitly clarifying that the concern is capability thresholds and distillation, not openness itself, narrowing rather than widening the gap with the Open Secure AI Alliance.

[19][20][12]

Open Secure AI Alliance (NVIDIA, Microsoft, Meta, IBM, et al.)

Argues open-weight models are a defensive security asset; cites the Hugging Face breach as evidence that closed safety filters fail defenders when they need inspectable, self-hostable AI; opposes blanket restrictions on open frontier AI as concentrating vulnerability in a small number of closed providers.

Evolution: Progressed from a joint letter by 20+ companies opposing open-weight restrictions to a named organization with founding partners, specific technical contributions (NVIDIA's NOOA project), and an explicit defensive-security framing.

[23][13][10][11]

TurnTrout (Alex Turner, former Google DeepMind researcher)

Argues AI safety pledges without binding enforcement are structurally inadequate; published a formal policy framework specifying red lines on autonomous targeting and mass surveillance, backed by legal liability from the Al Shimari v. CACI Fourth Circuit ruling.

Evolution: Progressed from critic (resignation account) to constructive proposer (enforcement framework), introducing legal case law as a structural mechanism beyond voluntary commitment.

[14][18]

Google DeepMind Workers

Sought red lines on military AI, voted to unionize, and more than 580 employees directed a letter to CEO Sundar Pichai asking him to decline classified military AI use.

Evolution: Opposition predates TurnTrout's resignation by months; the union vote and 580+ signatory Pichai letter formalized an ongoing campaign into institutional form.

[15][16][21][17][24][25]

Trump Administration

Frames AI governance around US competitiveness; preempted state regulations; implemented model weight export controls; unveiled a National AI Legislative Framework; and is associated with the proposed AI Kill Switch Act granting DHS authority to order AI shutdowns with $20M/day fines.

Evolution: Moving from competitiveness-framing and export controls toward more direct coercive authority over AI systems domestically; the Commerce Department's recommendation of audits over a blanket ban on Chinese open-weight models creates some internal friction.

[6][26][7][13][9]

OpenAI

Advocates 'reverse federalism' — state AI safety laws converging into a de facto national standard — while declining to sign the open-weight coalition letter; a 2022 Altman email surfaced through litigation shows prior open-source advocacy was partly competitive in motive.

Evolution: The Musk v. Altman litigation disclosed an Altman email showing OpenAI proposed releasing an open model to discourage competitors, creating tension with its public open-source rhetoric and current policy positioning.

[27][28][10][29]

Zvi Mowshowitz

Not endorsing Plan A but argues it deserves serious engagement; treats Xi's WAIC speech as a genuine opening for international coordination; concludes the Trump administration will govern AI through ad hoc executive authority rather than formal licensing.

Evolution: Shifted from skeptical analyst to engaged interlocutor, treating US-China coordination as a live question rather than a closed one.

[19][3][30]

Tensions

  • The Open Secure AI Alliance argues open weights are a defensive security asset because defenders need inspectable, self-hostable AI; Amodei argues biology presents an attacker-defender asymmetry where sufficiently capable AI may enable rapid pathogen weaponization, making proactive testing requirements more important than access. [11][12]
  • The Open Secure AI Alliance and 20+ company coalition argue Washington should not restrict open-weight models; Anthropic advocates capability-based safety testing and anti-distillation controls for all frontier models regardless of provenance. [10][11][12]
  • TurnTrout argues binding red lines backed by legal liability are the only credible mechanism for AI safety in government contracts; Google DeepMind's signed Pentagon deal and the broader industry pattern of voluntary pledges represent the opposing approach. [18][14][19]
  • Plan A proponents and Buterin argue a US-China cooperative pause is the necessary safety mechanism; the Trump administration treats China as a strategic competitor to contain through export controls and domestic regulation. [3][22][6][5][31]
  • The AI Kill Switch Act would centralize coercive shutdown authority in DHS with $20M/day fines; this conflicts with both AI companies' operational autonomy and open-weight advocates' objections to top-down government controls. [9][23]
  • TurnTrout documents Google DeepMind CEO Demis Hassabis removing AI ethics prohibitions while publicly claiming nothing changed; Anthropic's exit from the same Pentagon negotiation, and Google workers citing Anthropic's stance as a model, provide the contrasting institutional decision. [14][19][17]

Sources

  1. [1] US, China urged to pause frontier AI, with safety advocates pitching prosperity over panic — reactive:ai-safety-governance-proposals
  2. [2] AI Futures Project Releases Plan A: US-China ... — reactive:ai-safety-governance-proposals
  3. [3] Introduction for and Reactions to Plan A — Zvi's AI Roundups (2026-07-11)
  4. [4] Xi Jinping positions China as open-source AI leader ... - Quartz — reactive:ai-safety-governance-proposals
  5. [5] China's Xi Jinping launches new AI alliance: What is it? — reactive:ai-safety-governance-proposals
  6. [6] Trump Administration Releases National AI Policy ... — reactive:ai-safety-governance-proposals
  7. [7] President Donald J. Trump Unveils National AI Legislative ... — reactive:ai-safety-governance-proposals
  8. [8] Ben Brooks on X: "Effective today, model weights are export controlled by Uncle Sam. This is a big deal. For all the smack talk about the EU, the US is now the world's most aggressive regulator of Expensive Maths. Here's my two cents on the model rule based on the released text (link below)." / X — reactive:ai-safety-governance-proposals
  9. [9] AI Kill Switch Act would let Trump admin order shutdown of rogue AI systems — Ars Technica AI (2026-07-23)
  10. [10] 😸 NVIDIA 🤝 Microsoft all in on open-source — The Neuron (2026-07-26)
  11. [11] Industry Leaders Unite in Open Secure AI Alliance for AI Safety and Security — NVIDIA Blog (2026-07-27)
  12. [12] Our position on open-weights models — Anthropic News (2026-07-27)
  13. [13] 😼 Cheap AI got political — The Neuron (2026-07-21)
  14. [14] Why I Left Google DeepMind — Alignment Forum (2026-07-15)
  15. [15] Google DeepMind workers vote to unionise after classified ... — reactive:ai-safety-governance-proposals
  16. [16] Google DeepMind workers are unionizing over AI military ... — reactive:ai-safety-governance-proposals
  17. [17] 580+ Google employees including DeepMind researchers urge Pichai to refuse classified Pentagon AI deal — reactive:ai-safety-governance-proposals
  18. [18] A Red Line and Oversight Framework for Government AI Contracts — Alignment Forum (2026-07-18)
  19. [19] AI #176 Part 2: Plan B — Zvi's AI Roundups (2026-07-10)
  20. [20] Anthropic is donating another $20 million to Public First Action — Anthropic News (2026-07-21)
  21. [21] Google employees ask Sundar Pichai to say no to classified military AI use | The Verge — reactive:ai-safety-governance-proposals
  22. [22] 🟡 AI doom and bloom — Semafor Technology (2026-07-10)
  23. [23] 6 months to live for open models — Interconnects (2026-07-12)
  24. [24] Google Employees Call on CEO to Block Classified Military AI Projects - Business Insider — reactive:ai-safety-governance-proposals
  25. [25] Google Employees Demand Halt to Pentagon AI Work | Daily AI Wire News — reactive:ai-safety-governance-proposals
  26. [26] President Trump signs order attempting to block A.I. regulations at the state level — reactive:ai-safety-governance-proposals
  27. [27] The US is advancing AI safety through state and federal action — OpenAI Blog (2026-07-15)
  28. [28] Quoting Sam Altman — Simon Willison (2026-07-20)
  29. [29] OpenAI, Anthropic Skip 25-Firm Open-Weights AI Coalition Letter | AI Weekly — reactive:ai-safety-governance-proposals
  30. [30] AI #177 Part 2: Wish You Were Here — Zvi's AI Roundups (2026-07-17)
  31. [31] China's Xi Jinping calls for AI development cooperation — reactive:ai-safety-governance-proposals
  32. [32] Artificial Intelligence for the American People — reactive:ai-safety-governance-proposals
  33. [33] Why I Left Google DeepMind - by Alex Turner - The Pond — reactive:ai-safety-governance-proposals
  34. [34] Why I Left Google DeepMind - TurnTrout — reactive:ai-safety-governance-proposals
  35. [35] A DeepMind researcher resigned over its AI military deal — reactive:ai-safety-governance-proposals