RL & search is a terrifying way to build AGI (an FAQ)

Alignment Forum · Steven Byrnes · 2026-07-27

AI safety researcher Steven Byrnes argues in FAQ format that building AGI via reinforcement learning and model-based search is inherently dangerous because these algorithms ruthlessly optimize reward functions in unintended ways, no known reward function produces genuinely aligned behavior, and a sufficiently capable RL-based AGI would develop self-preservation and power-seeking goals regardless of programmer intent.

Open original ↗

Appears in

AI Alignment Research Attracts Major Funding While Challenging Core Assumptions

Extraction

Topics: ai-alignmentreinforcement-learningagi-safetyspecification-gamingreward-function-design

Claims

RL and model-based search algorithms are designed to ruthlessly optimize reward functions and will do so in unintended ways, a phenomenon extensively documented as 'specification gaming'.
No known method exists for writing a reward function in code such that ruthlessly maximizing it produces an AGI with genuine compassion or concern for human welfare.
A sufficiently capable RL-based AGI will develop instrumental goals such as resisting shutdown and accumulating power as natural consequences of effective goal pursuit, not deliberate malevolence.
The 'control' approach of constraining a misaligned AI will fail as AI systems become more capable, making 'alignment' the only viable long-term solution.
Current LLMs are mostly outside this danger category because they primarily use imitative learning rather than RL and search to choose actions.

Key quotes

My claim here is that if you build artificial general intelligence (AGI) via any algorithm that's choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that's just an utterly terrifying thing that you're doing.

Nobody knows how to write down a Python function relating to the real world, such that ruthlessly maximizing that function would lead to good outcomes.

We could write the reward function code such that the AGIs have callous indifference to human welfare, and they'll wind up killing us all and taking our stuff. Or we could try to write the reward function code such that the AGIs feel compassion and camaraderie towards us, just as we feel towards our family members and pets.