Correlated Errors in Language Models

Follow this topic Ask about this topic

Latest papers

8 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Reasoning Concentrates Errors, and Self-Consistency Never Notices
Sep 25, 2026Asaad AlthoubiCorrelated Errors in Language Models Self-Consistency in Language Models
Department of Computer Science Oklahoma State University Stillwater, OK, USA
Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models
Sep 3, 2026Ross Tieman, Evan MarkouLLM Reliability Language Model Ensembles
Fenner School of Environment & Society · Australian National University · School of Computing
JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification
Aug 20, 2026Tianxin Zhou, Ruixi LinCorrelated Errors in Language Models LLM-as-a-Judge
Independent Researcher
Minority Sentinel: When to Overturn Majority Voting in Multi-Agent LLM Debates
Jun 28, 2026Chuan He, Zebin Chen, Zhengyi Yang +5Multi-Agent Debate Multi-Agent Consensus
University of New South Wales · Euler AI · Macquarie University
Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
May 28, 2026Guneet KohliLLM Evaluation LLM-as-a-Judge
Apple
A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges
Apr 8, 2026Chenchen Kuai, Jiwan Jiang, Zihao Zhu +8Correlated Errors in Language Models LLM Auditing
Texas A&M University · Marquette University · University of Utah
The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission
Apr 7, 2026Theodor SpiroCorrelated Errors in Language Models LLM Reliability
Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness
Feb 20, 2026Yegor Denisov-Blanch, Joshua Kazdan, Jessica Chudnovsky +4Correlated Errors in Language Models Inference-Time Scaling
Stanford Computer Science · Stanford Statistics · Stanford University +1

Sep 25, 2026J/K move · Enter open · S save

Reasoning Concentrates Errors, and Self-Consistency Never Notices

Asaad Althoubi

Department of Computer Science Oklahoma State University Stillwater, OK, USA

Open paper AskPDF arXiv

Self-consistency assumes that independent samples disagree when a model is unsure, so agreement is evidence of correctness. Holding weights fixed and toggling only a reasoning mode, over five benchmarks and 74,944 samples, we show that reasoning concentrates a model's errors: the probability that two independently drawn wrong answers coincide rises in all ten dataset-scale comparisons (p = 0.00098), and in nine of nine after restricting both arms to the problems each gets wrong. Where the answer space is unbounded, reasoning cuts the distinct answers produced to 0.43-0.65 of the non-reasoning count; where it is bounded, both arms hold an identical option set and reasoning concentrates mass on it instead, which no positional prior can explain at fixed weights. The aggregate cost is smaller than the mechanism predicts, because reasoning also shrinks the set of problems where answer diversity can decide anything, in ten of ten cells and by 2.7x; normalized for available headroom, both arms convert a quarter of it in domain. Confidence weighting does not recover what is left. Across 280 method-dataset-model combinations on eight models and five benchmarks, not one beats plain majority voting after correction; weighted voting agrees with it on 98.5% of problem-method pairs and is right 56.3% of the time on the rest; and a signal's direction can invert within fixed weights, with answer log-probability predicting correctness when reasoning is off and error when it is on. A learned six-signal combination gains nothing out of domain. Confidence signals should be evaluated on decisions, not on discrimination.