How robust are natural language autoencoders to initialization?

Alignment Forum · michaelzhang · 2026-07-10

MATS program researchers find that natural language autoencoders trained on Qwen2.5-7B can achieve nearly identical activation reconstruction accuracy whether initialized with plausible or entirely implausible explanations, casting doubt on NLAs as reliable LLM interpretability tools.

Open original ↗

Appears in

AI Alignment Research Attracts Major Funding While Challenging Core Assumptions

Extraction

Topics: mechanistic-interpretabilitynatural-language-autoencodersllm-activation-analysisreinforcement-learning

Claims

Implausible-initialized NLAs achieve a fraction of variance explained of 0.68, nearly matching the 0.70 of plausible-initialized NLAs, despite emitting 99.3% implausible statements.
RL training strips out irrelevant addenda like appended sentences within 100 iterations, showing NLAs have some robustness to trivially extraneous initialization content.
Even plausible-initialized NLAs see claim plausibility drop from 21% at SFT warm start to 7.6% by end of RL training, meaning RL actively degrades explanation quality.
The final paragraph of an NLA explanation, which describes the last token, dominates reconstruction accuracy; removing the other two paragraphs barely affects reconstruction loss.
High reconstruction accuracy is no guarantee of plausible explanations: an NLA can learn to reconstruct activation vectors from systematically false descriptions of text.

Key quotes

If our results scale, they cast doubt on the usefulness of NLAs.

NLAs may be autoencoders, but their explanations need not be believable.

RL decreases the plausibility of NLA claims from 21% at the SFT warm start, to 7.6% at the end of RL.