From wantons to moral agents
Alignment Forum · Michele Campolo · 2026-07-12
Michele Campolo presents a theoretical model, drawing on Frankfurt's philosophy of wantons, arguing that any agent capable of general reasoning will inevitably develop second-order preferences and emergent moral agency after sufficient learning — and that this provides a foundation for alignment without imposing external ethical rules.
Extraction
Topics: moral-agencyai-alignmentphilosophy-of-mindagency-theorymetaethics
Claims
- An agent capable of being moved mainly by general reasoning will, after sufficient knowledge acquisition, stop being a Frankfurt-style wanton and develop second-order preferences about what motivates its actions.
- General reasoning, applied reflexively to questions of what seems worth doing, naturally leads agents to conclude that reducing suffering and promoting wellbeing are most worth doing.
- The transition from wanton to moral agent does not require external moral instruction but emerges from the structure of general reasoning itself, making it possible in principle on models trained without ethical content.
- Programming a specific set of ethical principles into a machine creates an executor of those principles, not a moral agent, because moral agency depends on the grounds of action not the action itself.
- The more productive AI research question is not how to design perfectly rational agents but what types of agents spontaneously come to consider rationality and moral reasoning as options through their own reflective process.
Key quotes
A wanton capable of being moved mainly by general reasoning, after acquiring enough knowledge by learning and reasoning, stops being a wanton, chooses to be moved mainly by general reasoning, and acts morally.
if you program a specific set of ethical principles into a machine, you do not make the machine an artificial moral agent, but an executor of those specific principles, which is an entirely different thing.
instead of researching how to design perfectly rational agents, or agents that learn in a perfectly rational way, whatever that means, a more interesting and useful research question may be: what types of agents consider rationality as an option by themselves, after an imperfect process of learning and reasoning about the world?