Notes on technical alignment via human-like social drives
Alignment Forum · Steven Byrnes · 2026-07-08
Steven Byrnes proposes using human innate social drives—"Sympathy Reward" and "Approval Reward"—as the foundation for brain-like AGI alignment, tentatively advocating for a "truth-seeking disagreeable nerd AGI" as the most viable approach to navigating the acute AI risk period.
Appears in
Extraction
Topics: ai-alignmentbrain-like-agisocial-drivesagi-safetyreward-function-design
Claims
- Human innate social drives—specifically Sympathy Reward and Approval Reward—can serve as a jumping-off point for designing prosocial AGI motivation systems.
- Pure consequentialist goals derived from Sympathy Reward alone lead to a "ruthless bliss-maxxing" failure mode in which an AGI tiles the universe with hedonium.
- Approval Reward introduces virtue-ethics-y, non-consequentialist motivations that can resist instrumental convergence even under extreme power imbalances, provided the setup is correct.
- A "desire-first" pathway—where innate drives like aversion to confusion lead organically to pride in honesty and truth-sharing—is more promising for alignment than a "person-first" pathway of social mimicry.
- The most viable alignment approach is a "truth-seeking disagreeable nerd AGI" motivated to understand the strategic situation regarding ASI and share its findings with humans, satisfying ethical and buy-in constraints better than corrigibility or autonomous action.
Key quotes
The premise of this project is: we assume that someone has figured out the secret sauce of how the human cortex (in conjunction with related brain areas) creates human intelligence, and then they ask me what code they should write for the reward function, training setup, test protocols and so on.
I think we need to get away from pure ruthless optimization, by mixing in some scrupulous honesty, sincere helpfulness, and so on. And I think that the only way to get those is via virtue-ethics-y dispositions, tied to pride in one's self-image.
If somehow I knew a lot about algorithms and AI but nothing about humans, and you asked me how to make an AGI that sincerely wants to figure out true and insightful things, and share them, then what would I propose? Well, I'll tell you what I wouldn't propose: 'let's make a social system by labeling concepts corresponding to thinking about humans and specific human feelings...'