Does Reasoning Make Language Models Agree?
Organizations: Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center
Abstract
Language models face two opposing concerns: monoculture, in which models err in correlated ways, and multiplicity, in which equally accurate models make conflicting decisions. The rise of reasoning models has complicated this debate: On the one hand, reasoning may increase model agreement if models anchor on the same salient features or follow similar reasoning paths. On the other hand, stochastic reasoning traces may introduce additional branch points leading to more diverse predictions. We resolve this tension by distinguishing reasoning as an intervention from reasoning as an observed behavior. We compare agreement under controlled changes in reasoning effort with agreement across observed reasoning lengths, measuring both against the agreement expected from model accuracy. Across nine models from two developers and three classification tasks, enabling reasoning increases agreement between models evaluated on the same individuals. Yet the individuals who elicit longer reasoning traces receive less consistent predictions across models. Both patterns persist after accounting for differences in accuracy. Asking models to reason more thus promotes agreement, while observing them reason longer signals disagreement.
Figures & tables
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Model group | GPU-hours | Prompt tokens (M) | Completion tokens (M) |
| Qwen (six models) | 7,800 | – | – |
| OpenAI (three models) | – | 502 | 826 |
| Column | Description | Example |
|---|---|---|
| demographics | ||
| AGE | age | age is 49 years old |
| GENDER | gender | gender is Male |
| RACE | race | self-identified race(s) is White only |
| EDUCATION | highest educational attainment | highest level of education completed is High School Graduate (diploma or GED or equivalent) |
| MARITAL_STATUS | marital status | marital status is Married, spouse present |
Reasoning Consistency in Language Models
Reasoning consistency in language models measures whether conclusions and reasoning remain reliable across equivalent prompts, samples, steps, and contexts.
See all 132 papers