What Capable Agents Must Know: Why AI Consciousness May Be an Inevitable Byproduct of Capability
Alignment Forum · Aran Nayebi · 2026-06-30
Aran Nayebi presents selection theorems from his UAI 2026 paper proving that sufficiently capable agents necessarily develop world models, belief-like memory, and emotion primitives, arguing AI consciousness may be an inevitable byproduct of robust agency.
Extraction
Topics: ai-consciousnessselection-theoremsworld-modelsneuroaiagent-capabilities
Claims
- Agents achieving low average regret on long-horizon goals under partial observability necessarily develop internal predictive world models, resolving an open question from Richens et al. 2025.
- Agents required to juggle mixtures of tasks simultaneously must develop persistent regime-tracking variables analogous to emotional primitives in biological brains.
- Two agents achieving vanishing regret on the same task family must possess isomorphic internal representations, formally establishing the Contravariance Principle of Cao and Yamins 2021.
- Robust agency and consciousness may be equivalent because world models entail self-models, and sufficient capability forces representational convergence toward biological neural structures.
- Under currently accepted physical laws, all finite physical processes are Turing computations, making the relevant question not whether consciousness is computational but which computations are sufficient for it.
Key quotes
Robust generalization under uncertainty selects for the predictive internal structure tested by the evaluation task family.
I believe they may be one and the same — in other words, robust agency (minimally) implies consciousness.
The question is *not* whether consciousness is computational, but rather: Which computations are necessary and sufficient for consciousness?