Should we benchmark conceptual capabilities using judgment prediction tasks?
Alignment Forum · Alex Mallen · 2026-07-17
Alignment Forum researcher Alex Mallen analyzes whether AI models should be benchmarked on conceptual reasoning by predicting a specific expert's judgment rather than objective truth, identifying both promise and significant methodological risks including noise, data leakage, and cutoff validity.
Appears in
Extraction
Topics: ai-benchmarkingconceptual-reasoningalignment-researchevaluation-methodology
Claims
- Many conceptual reasoning tasks involve hard-to-resolve subjective disagreements that make standard benchmarking unreliable as a measure of model capability.
- Judgment prediction benchmarks—where models predict a specific named expert's answer—could reduce unresolvable disagreement noise and better isolate model capability.
- A model called Mythos preview reportedly matched Ryan Greenblatt's own accuracy on MCQs about Greenblatt's unpublished views.
- Training data leakage is a major risk when benchmarks are built from experts' existing public or unpublished writing, as the model may pattern-match rather than reason.
- A lightweight version of judgment prediction benchmarking, where models are prompted with a judge's worldview and context, is cheap and likely worthwhile.
Key quotes
For conceptual tasks with hard-to-resolve disagreement, it's unclear if judgment prediction is a suitable methodology for benchmarking—i.e., for tracking progress of frontier models over time.
I heard that Mythos preview was about as good at answering MCQs about Ryan Greenblatt's views than Ryan was. The MCQs were generated based on some of Ryan's unpublished writing.
I overall think that it's quite good and cheap to do the milquetoast version of this benchmark methodology, in which you basically make whatever conceptual reasoning benchmark you were going to make anyways but open up to questions with harder-to-resolve disagreements and you prompt the model with some background worldview and context on the judge.