Correlated Errors in Language Models

Follow this topic Ask about this topic

Latest papers

12 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors
Nov 9, 2026Paul LitvakAutomated Peer Review Language Model Error Detection
Robyn Dawes Institute
Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models
Nov 4, 2026Joan C. Timoneda, Sebastián Vallejo VeraScientific Reproducibility LLM Reliability
Assistant Professor, Department of Political Science, Purdue University. · Assistant Professor, Department of Political Science, University of Western Ontario.
Wrong Together: Shared Bias in User Simulation
Oct 31, 2026Saber ZerhoudiUser Simulation Correlated Errors in Language Models
University of Passau Passau, Germany
Reasoning Concentrates Errors, and Self-Consistency Never Notices
Sep 25, 2026Asaad AlthoubiCorrelated Errors in Language Models Self-Consistency in Language Models
Department of Computer Science Oklahoma State University Stillwater, OK, USA
Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models
Sep 3, 2026Ross Tieman, Evan MarkouLLM Reliability Language Model Ensembles
Fenner School of Environment & Society · Australian National University · School of Computing
JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification
Aug 20, 2026Tianxin Zhou, Ruixi LinCorrelated Errors in Language Models LLM-as-a-Judge
Independent Researcher
Minority Sentinel: When to Overturn Majority Voting in Multi-Agent LLM Debates
Jun 28, 2026Chuan He, Zebin Chen, Zhengyi Yang +5Multi-Agent Debate Multi-Agent Consensus
University of New South Wales · Euler AI · Macquarie University
Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
May 28, 2026Guneet KohliLLM Evaluation LLM-as-a-Judge
Apple
Where Cross-Question Structure in LLM Errors Comes From: Shared Seeds, Shared Prompts, and Answer-Letter Preference
May 18, 2026Izhar AliCorrelated Errors in Language Models Multiple-Choice Question Answering
Rowan University.
A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges
Apr 8, 2026Chenchen Kuai, Jiwan Jiang, Zihao Zhu +8Correlated Errors in Language Models LLM Auditing
Texas A&M University · Marquette University · University of Utah
The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission
Apr 7, 2026Theodor SpiroCorrelated Errors in Language Models LLM Reliability
Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness
Feb 20, 2026Yegor Denisov-Blanch, Joshua Kazdan, Jessica Chudnovsky +4Correlated Errors in Language Models Inference-Time Scaling
Stanford Computer Science · Stanford Statistics · Stanford University +1

Nov 9, 2026J/K move · Enter open · S save

How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors

Paul Litvak

Robyn Dawes Institute

Automated Peer Review Language Model Error Detection Scholarly Peer Review Correlated Errors in Language Models

Open paper AskPDF arXiv

One hundred known errors were planted into 10 open-access psychology papers, and the modified papers were then run through frontier models and two commercial AI review tools. The best single system caught 71 of the 100 errors, while the worst caught 30. Pooling every system's output caught 93. Models are only partly correlated in the errors they find, which makes ensembling a big lever for finding issues in papers. Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it. Refine.ink contributes more unique catches than any other single system, though it is expensive on a per-error basis. False positives were not measured, and how this error distribution compares to the distribution of errors in real papers remains unknown. The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.