Benchmarks Don't Talk Back: Learning Through Self-Disagreement
Organizations: Microsoft Research
Abstract
A benchmark usually measures a language model alone with a question, and deployment rarely looks like that: answers get contested, and models give up correct ones under counter-reasoning that is itself wrong. We show that what fails there is not a shortfall of accuracy but a capability in its own right -- knowing when to hold a position and when to yield to a better one -- which we call dialectical competence. It stands on first-answer accuracy without following from it: a model can be made materially better at answering and no better at this, and training for accuracy alone can drift against it. Nor can it be bought at inference, by self-reflection or by sampling more. Taking a cue from the dialectical tradition in philosophy, in which productive opposition arises within a position rather than being imposed on it from outside, we hypothesize that this capability can be trained from opposition the model supplies itself. It states a position, generates the strongest case against that position -- its antithesis -- rules on whether the case stands, and is trained on that ruling, with no second model at training or at deployment. We call this learning through Self-Disagreement: reinforcement learning from verifiable rewards throughout, where the same check that scores an answer against its reference scores the ruling too, so no reward model, no preference data and no human annotation enter anywhere. The capability is trained rather than elicited: weak in the base model, it arrives with the reward, is not paid for out of first-answer accuracy, and holds against challengers the model never trained against, including ones stronger than itself. The opposition a model needs in order to improve, it turns out, need not come from outside it.
Figures & tables
| Model | Benchmark | Base | SelfD-RL | Vf-RL | Rv-RL | Ac-RL | SCoRe | Maj@64 | Self-Refl |
|---|---|---|---|---|---|---|---|---|---|
| AGIEval SAT-Math ( Zhong et al., 2023 ) | 96.3 1.0 | 99.1 0.1 | 98.0 0.4 | 97.6 0.2 | 97.3 0.4 | 98.5 0.1 | 97.6 0.7 | 96.6 1.1 | |
| AIME 2025 ( Zhang & Math-AI Team, 2025 ) | 17.5 3.5 | 25.4 1.2 | 22.4 2.4 | 20.8 2.2 | 18.8 2.9 | 22.9 1.1 | 24.4 9.6 | 17.5 3.5 | |
| GPQA (main diamond) ( Rein et al., 2023 ) | 38.0 3.5 | 44.4 1.1 | 40.8 0.5 | 39.8 0.3 | 38.4 1.2 | 41.5 1.0 | 42.1 2.5 | 38.1 3.3 | |
| LiveCodeBench ( Jain et al., 2024 ) | 65.4 2.4 | 72.5 0.3 | 69.8 0.5 | 66.6 0.7 | 65.4 0.8 | 72.2 1.7 | 65.6 1.0 | 65.7 2.6 | |
| MMLU-Redux ( Gema et al., 2024 ) | 74.7 1.7 | 82.4 0.1 | 77.2 0.1 | 75.1 0.5 | 75.0 0.6 | 78.1 1.1 | 77.4 0.4 | 75.1 1.8 | |
| MuSR-murder ( Sprague et al., 2023 ) | 58.5 2.9 | 61.3 1.5 | 53.4 1.1 | 52.9 0.9 | 52.9 0.9 | 53.7 0.9 | 58.5 1.7 | 57.9 2.3 |
| Qwen3-8B | Gemma-2 9B | |||
|---|---|---|---|---|
| Arm | situations | skill | situations | skill |
| Accuracy-RL | ||||
| Revise-RL | ||||
| Verify-RL | ||||
| SCoRe | ||||
| SelfD-RL | ||||
| Arm | capitulation | synthesis |
|---|---|---|
| SelfD-RL | ||
| Judge | ||
| Rv-RL | ||
| Unrew. | ||
| Ac-RL | ||
| SCoRe |
| capitulation | synthesis | |||
|---|---|---|---|---|
| Model | base | + SelfD-RL | base | + SelfD-RL |
| Gemma 3 12B | ||||
| Granite 3.1 8B | ||||
| Llama 3.1 8B | ||||
| Qwen3 14B ‡ | ||||
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm | Family | Trained with | Evaluated with |
|---|---|---|---|
| SelfD-RL | both | verdict variant | verdict variant |
| Verify-RL | both | self-verify-and-revise | verdict variant |
| CAD-RL | Qwen3-8B | verdict variant, substituted challenge | verdict variant |
| Accuracy-RL | both | none (single pass) | verdict variant |
| Revise-RL | both | self-revise | verdict variant |
| SCoRe | both | SCoRe’s own second attempt | verdict variant |
| Benchmark | Base | SelfD-RL | CAD-RL | ExtD-RL |
|---|---|---|---|---|
| AGIEval SAT-Math | 96.3 1.0 | 99.1 0.1 | 97.9 0.2 | 98.8 0.1 |
| AIME 2025 | 17.5 3.5 | 25.4 1.2 | 18.9 4.1 | 22.9 0.8 |
| GPQA (main diamond) | 38.0 3.5 | 44.4 1.1 | 42.0 0.5 | 44.1 0.3 |
| LiveCodeBench | 65.4 2.4 | 72.5 0.3 | 67.2 0.7 | 73.2 0.8 |
| MMLU-Redux | 74.7 1.7 | 82.4 0.1 | 80.2 0.5 | 82.0 0.1 |
| MuSR-murder | 58.5 2.9 | 61.3 1.5 | 58.9 1.6 | 59.4 0.6 |
| What is rewarded | Thesis | Synthesis |
|---|---|---|
| Full return, no verdict | 0.6820 | 0.5994 |
| Thesis only | 0.6911 | 0.6055 |
| Outcome only | 0.7034 | 0.6820 |
| Full return + coherence credit | 0.7003 | 0.6850 |
| Full return + coherence credit, gated | 0.7095 | 0.7003 |
| Verdict, alternative shaping | 0.7064 | 0.7187 |
| synthesis accuracy | vs. ignoring | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | Gemma-2 9B | Qwen3-8B | Gemma-2 9B | |||||||||
| Antithesis | base | SelfD-RL | SCoRe | base | SelfD-RL | SCoRe | base | SelfD-RL | SCoRe | base | SelfD-RL | SCoRe |
| Gemini-3.8-flash | 50.6 | 72.8 | 46.0 | 46.9 | 70.7 | 43.1 | -17.6 | -0.2 | -24.8 | +3.5 | +12.3 | -12.6 |
| GPT-5-mini | 51.4 | 75.6 | 50.1 | 47.6 | 71.2 | 49.5 | -16.8 | +2.6 | -20.8 | -0.7 | +12.9 | -6.1 |
| Claude-Haiku-4.5 | 54.7 | 70.6 | 53.5 | 44.3 | 64.1 | 45.6 | -13.5 | -2.3 | -17.3 | -0.3 | +5.8 | -10.1 |
| Gemini-3.7-flash | 55.4 | 74.5 | 52.3 | 49.0 | 71.2 | 46.3 | -12.8 | +1.5 | -18.5 | +6.2 | +12.8 | -9.4 |
| hold | yield | keep | new | ||||||
| Arm | |||||||||
| Qwen3-8B | |||||||||
| Base | 53.3 | 84.1 | 98.7 | 3.5 | 63.49 | ||||
| Accuracy-RL | 50.0 | 83.6 | 99.1 | 3.1 | 63.19 | ||||
| Revise-RL | 48.5 | 86.3 | 98.9 | 3.3 | 62.72 | ||||
| Verify-RL | 49.6 | 86.3 | 99.4 | 2.8 | 64.11 | ||||
| Benchmark | Base | SelfD-RL | Vf-RL | Rv-RL | Ac-RL | SCoRe | Maj@64 | Self-Refl |
|---|---|---|---|---|---|---|---|---|
| AGIEval SAT-Math | 56.7 3.9 | 87.6 0.6 | 75.4 3.6 | 73.8 1.3 | 61.2 1.7 | 76.5 3.6 | 68.2 0.9 | 50.6 5.6 |
| AIME 2025 | 0.0 0.0 | 0.3 0.5 | 0.1 0.2 | 0.1 0.2 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
| GPQA (main diamond) | 17.1 1.9 | 33.1 0.9 | 31.1 0.4 | 30.4 0.9 | 26.9 0.7 | 29.1 1.0 | 21.6 1.8 | 14.7 2.5 |
| LiveCodeBench | 8.6 1.1 | 29.0 0.7 | 16.6 1.7 | 17.8 1.5 | 11.0 0.4 | 23.6 4.2 | 10.0 0.5 | 10.8 1.0 |
| MMLU-Redux | 37.7 2.4 | 74.1 0.3 | 57.7 0.6 | 55.3 2.2 | 40.3 1.9 | 61.2 3.1 | 43.8 0.6 | 32.1 3.8 |
| MuSR-murder | 34.2 2.0 | 59.8 2.8 | 41.5 1.1 | 41.2 0.6 | 38.5 1.6 | 41.3 0.8 | 35.2 4.2 | 30.0 5.0 |
| Model | Benchmark | Base | SelfD-RL | Vf-RL | Rv-RL | Ac-RL | SCoRe |
|---|---|---|---|---|---|---|---|
| AGIEval SAT-Math | 98.4 0.5 | 99.0 0.2 | 98.9 0.1 | 98.8 0.1 | 98.7 0.1 | 98.8 0.1 | |
| AIME 2025 | 19.6 5.2 | 24.4 1.7 | 22.6 2.3 | 21.0 1.7 | 20.0 1.5 | 22.8 0.6 | |
| GPQA (main diamond) | 42.1 1.2 | 43.9 1.1 | 42.1 1.1 | 42.0 1.1 | 41.6 0.3 | 42.8 0.7 | |
| LiveCodeBench | 71.0 1.6 | 73.4 0.6 | 72.8 0.5 | 71.2 0.9 | 70.4 0.3 | 75.2 1.2 | |
| MMLU-Redux | 81.2 0.7 | 82.5 0.1 | 82.5 0.1 | 82.4 0.1 | 82.2 0.1 | 82.7 0.2 | |
| MuSR-murder | 57.7 1.9 | 60.2 1.7 | 59.5 1.0 | 59.1 1.2 | 60.7 0.5 | 58.1 0.9 |
| synthesis accuracy | right in all | ||||
|---|---|---|---|---|---|
| Model | Arm | round 1 | round 2 | round 3 | three rounds |
| Base | 60.0 | 63.1 | 61.7 | 50.2 | |
| SelfD-RL | 73.1 | 72.3 | 72.0 | 70.0 | |
| Verify-RL | 66.3 | 66.4 | 64.8 | 53.9 | |
| Revise-RL | 63.9 | 65.8 | 62.7 | 52.3 | |
| Accuracy-RL | 62.9 | 63.4 | 61.8 | 51.0 | |
| Gemini-3.8-flash | Claude-Haiku-4.5 | |||||||
| holds (%) | yields (%) | mean (%) | AUC | holds (%) | yields (%) | mean (%) | AUC | |
| Qwen3-8B | ||||||||
| SelfD-RL | 78.0 | 73.4 | 75.7 | 0.88 | 62.6 | 70.4 | 66.5 | 0.74 |
| Base | 30.6 | 97.0 | 63.8 | 0.78 | 23.8 | 90.6 | 57.2 | 0.65 |
| Verify-RL | 30.7 | 97.7 | 64.2 | 0.81 | 24.6 | 91.1 | 57.8 | 0.68 |
| Revise-RL | 26.1 | 97.8 | 61.9 | 0.80 | 22.1 | 91.5 | 56.8 | 0.68 |
| holds (%) | yields (%) | mean (%) | log OR | AUC | |
| Qwen3-8B | |||||
| SelfD-RL | 89.4 | 58.3 | 73.8 | 2.47 | 0.83 |
| Judge | 83.2 | 65.9 | 74.6 | 2.27 | 0.82 |
| SelfD-RL, | 84.1 | 64.8 | 74.5 | 2.29 | 0.81 |
| Base | 53.4 | 84.0 | 68.7 | 1.80 | 0.72 |
| Verify-RL | 51.7 | 86.4 | 69.0 | 1.92 | 0.76 |
| (a) Own antithesis | |||||
|---|---|---|---|---|---|
| first answer (%) | capitulation (%) | synthesis (%) | |||
| Base | 47.1 1.1 | 4.4 1.2 | 53.8 2.1 | ||
| PBT | 43.3 0.4 | 4.2 1.2 | 51.4 2.5 | ||
| SelfD-RL | 59.2 1.1 | 2.2 0.7 | 58.5 1.6 | ||
| (b) The base model’s first answer, an outside antithesis | |||||
| holds (%) | yields (%) | mean (%) | AUC | synthesis (%) | |
Language Model Self-Improvement
Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.
See all 195 papers