Data filtering works a lot worse than you would expect
Alignment Forum · Dohun Lee · 2026-07-07
Alignment Forum researchers find that standard training data attribution methods — including LLM judges, probes, EKFAC, and activation-based approaches — largely fail to remove broad assistant-like behaviors from supervised fine-tuned language models, with refusal being the main filterable exception, suggesting these behaviors are elicited from the base model rather than taught during fine-tuning.
Extraction
Topics: supervised-fine-tuningtraining-data-attributionalignmentdata-filteringsft-behaviors
Claims
- Filtering the top 10% of training documents identified by attribution methods does not outperform random document removal for most broad SFT behaviors including bold formatting, both-sides framing, liberal lean, and feelings validation.
- Training a model on only coding or reasoning data still produces the same broad assistant-like behaviors, suggesting these behaviors are not caused by specific documents but are elicited when a model is shifted into an assistant-like mode.
- Refusal is the main exception to the general finding — it appears to be genuinely filterable via data attribution and removal, with probes and LLM judges being the most effective methods.
- All tested attribution methods (EKFAC, probes, activation-based, LLM judge) work well on targeted fine-tuning testbeds like emergent misalignment where the source of the bad data is known, but fail on broad naturalistic SFT behaviors.
- Many broad SFT behaviors appear to pre-exist in the mid-trained base model as assistant personas and are elicited rather than instilled during supervised fine-tuning.
Key quotes
filtering out 10% of documents chosen across TDA methods does not lead to the model saying 'Your feelings are valid' any less
many behaviors are bundled together into a persona, and training on enough of those traits will teach all of the others, even if there is not a clear relationship
many of these behaviors are not taught by a small number of responsible examples, but are instead elicited as a consequence of shifting the model into an assistant-like mode