cs.LGOct 30, 2026

Does Reasoning Make Language Models Agree?

Authors: Mila Gorecki, Moritz Hardt

Organizations: Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center

Abstract

Language models face two opposing concerns: monoculture, in which models err in correlated ways, and multiplicity, in which equally accurate models make conflicting decisions. The rise of reasoning models has complicated this debate: On the one hand, reasoning may increase model agreement if models anchor on the same salient features or follow similar reasoning paths. On the other hand, stochastic reasoning traces may introduce additional branch points leading to more diverse predictions. We resolve this tension by distinguishing reasoning as an intervention from reasoning as an observed behavior. We compare agreement under controlled changes in reasoning effort with agreement across observed reasoning lengths, measuring both against the agreement expected from model accuracy. Across nine models from two developers and three classification tasks, enabling reasoning increases agreement between models evaluated on the same individuals. Yet the individuals who elicit longer reasoning traces receive less consistent predictions across models. Both patterns persist after accounting for differences in accuracy. Asking models to reason more thus promotes agreement, while observing them reason longer signals disagreement.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning

    May 22, 2026Muhammad Usama, Dong Eui ChangRepresentational Similarity AnalysisLLM Interpretability

  2. Hidden Language Consistency Phenomena in Reasoning LLMs

    Aug 9, 2026Muhammad Ali Shafique, Kelly MarchisioMultilingual Language ModelsMultilingual Language Model Evaluation

  3. Know Thy Reasoner: Not All Language Models Explore Alike

    Apr 12, 2026Moulik Choraria, Argyrios Gerogiannis, Anirban Das +4LLM ReasoningEfficient Language Model Reasoning