Self-Consistency in Language Models

Follow this topic Ask about this topic

This topic studies how language models’ answers and reasoning vary across repeated generations, prompts, and related inputs, and whether consistency can improve reliability. Common approaches sample multiple reasoning paths and aggregate them with majority voting, semantic clustering, ranking, or verification; adaptive sampling and stopping rules aim to reduce the cost. Other work trains models to align their explanations and behavior or tests consistency in settings such as safety, preferences, and evaluation. A central tension is that agreement can improve performance but may also reinforce correlated errors, so consistency alone is not proof of correctness.

Latest papers

49 papers10 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Which Answers Agree? Evaluating Self-Consistency from Incomplete Logs
Oct 19, 2026Zhengtao Gui, Zexi Chen, Weining ShenSelf-Consistency in Language Models Language Model Reasoning Evaluation
University of California, Irvine
Think More or Read More? A Token-Budget Theory of Reasoning with Shared Context
Oct 16, 2026Manh Nguyen, Sunil Gupta, Hung LeEfficient Language Model Reasoning Multi-Agent Debate
Applied Artificial Intelligence Initiative, Deakin University, Australia
When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation
Oct 15, 2026Xiang Zhang, Lovedeep Gondara, Jekaterina NovikovaReference-Free Evaluation LLM Reliability
The Vanguard Group, Inc.
Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL
Oct 14, 2026Dylan P. Wilson, Chengsong Huang, Jiaxin HuangRL for Language Model Reasoning Noisy-Label Learning
Dartmouth College · Washington University in St. Louis
Beyond Answer Voting: Fine-Grained Correction of LLM Reasoning through Graph Alignment across Sampled Traces
Oct 13, 2026Haotian Liu, Yingjian Chen, Xiaohang Xu +4LLM Self-Correction Reasoning Verification
School of Informatics, Xiamen University · The University of Tokyo · National Institute of Informatics (NII)
ACED: Reference-Anchored Aggregation of Natural-Language and Executable Evidence for Mathematical Reasoning
Oct 12, 2026Yiyi Lu, Jiaxiang Geng, Bing LuoLLM Mathematical Reasoning LLM Reasoning
1Duke Kunshan University · 2The University of Hong Kong
An Investigation of Model Coherence: Narrow Finetunes Contradict Themselves Under Resampling
Oct 8, 2026Robert Graham, Yariv Barsheshat, Phil Blandfort +1Reasoning Consistency in Language Models Self-Consistency in Language Models
Independent · Predictably Weird · Amazon
Self-Repulsive Sampling for Diffusion Language Models
Sep 30, 2026Michael Helcig, Martin JaggiDiffusion Language Model Decoding Diverse Text Generation
ETH Zürich · EPFL
Adaptive Self-Consistency: From Black-Box Sampling to Distribution-Valued Feedback
Sep 30, 2026Jingkai Huang, Yunfan Zhang, Will Ma +2LLM Inference Efficiency Adaptive Inference
Stern School of Business New York University · School of Management Zhejiang University
VAA-CSEC: Vote-guided Advantage Allocation for Chinese Semantic Error Correction
Sep 29, 2026Yitong Han, Nankai Lin, Juan Luo +3RL for Language Models Self-Consistency in Language Models
School of Information Science and Technology, Guangdong University of Foreign Studies · Guangdong Engineering Research Center of Data Security Governance and Privacy Computing · College of Computer Science and Technology, National University of Defense Technology
Reasoning Concentrates Errors, and Self-Consistency Never Notices
Sep 25, 2026Asaad AlthoubiCorrelated Errors in Language Models Self-Consistency in Language Models
Department of Computer Science Oklahoma State University Stillwater, OK, USA
Enhancing Assessment of Self-Consistency in LLM Explanations using Perturbation Strength
Sep 25, 2026Phuong Q. Le, Kemal Kurniawan, Jey Han LauSelf-Consistency in Language Models Language Model Robustness
School of Computing and Information Systems The University of Melbourne, Melbourne, Australia · School of Computer Science and Engineering University of New South Wales, Sydney, Australia · The University of Melbourne
Strategic Self-Consistency
Sep 24, 2026Tori Qiu, Ander Artola Velasco, Manuel Gomez-RodriguezStrategic Reasoning Risks in Language Models Self-Consistency in Language Models
Carnegie Mellon University, Pittsburgh, USA · Max Planck Institute for Software Systems, Kaiserslautern, Germany
Route-Verify-Vote: Procedure-Conditioned Self-Consistency for Mixed-Domain Reasoning
Sep 23, 2026Xinchen XiaoCommonsense Reasoning Self-Consistency in Language Models
Xinjiang University
An Analysis of Training-Free Self-Reported Confidence in Language Models
Sep 17, 2026Lukas Meyer, Sofia Rossi, Wei Chen +2Confidence Estimation in Language Models Language Model Self-Assessment
DreamAI
When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
Sep 12, 2026Aakash Kumar TiwariLLM-as-a-Judge Bias in LLM-as-a-Judge
Department of Mathematics Indian Institute of Technology Kharagpur
Prefix-Denoising Consistency: Test-Time Verification for Diffusion Language Models
Aug 26, 2026Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1Diffusion Language Models Diffusion Language Model Inference
MBZUAI · Nagoya University · RIKEN AIP
Is Discrete Difficulty Sufficient? Leveraging Continuous Difficulty for Efficient Self-Consistency in LLMs
Aug 25, 2026Sihyeong Yeom, Geon Park, Geunyeong Jeong +3Efficient Language Model Reasoning Self-Consistency in Language Models
Konkuk University · DATUMO INC.
Decomposing Wrong-Consensus Agreement in LLM Self-Consistency
Aug 19, 2026Lizhuo Zhang, Mengmeng Tang, Chenfeng Long +2LLM Evaluation LLM Reliability
College of Information and Intelligence, Hunan Agricultural University, Changsha 410128, China · Yuelushan Laboratory, Changsha 410128, China · School of Computer Science and Technology, Changsha University of Science and Technology, Changsha 410114, China +1
LLM-Derived Preference Judgments Are Not Self-Consistent
Aug 18, 2026Matthew T. Ford, Francis Bahk, Jingjing Wang +4Preference Modeling User Preference Modeling
Cornell University · Georgia Institute of Technology
Incoherent by Design? On the Moral Self-Consistency of LLMs
Aug 15, 2026Pegah Nokhiz, Aravinda Kanchana Ruwanpathirana, Helen NissenbaumMoral Reasoning in Language Models Reasoning Consistency in Language Models
Cornell University, Cornell Tech · Cornell Tech · Nanyang Technological University
On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3On-Policy Self-Distillation Language Model Self-Improvement
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1
Position: It's Time to Optimize LLMs for Self-Consistency
Jul 31, 2026Itamar Pres, Belinda Z. Li, Laura Ruis +6Reasoning Consistency in Language Models Language Model Robustness
MIT CSAIL · Goodfire AI
Controlled Reformulation Testing for Logical Consistency in Large Language Models
Jul 16, 2026Alexander Gu, Alan ChenLLM Evaluation Reasoning Consistency in Language Models
University of Texas at Austin
When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
Jul 9, 2026Kaihua DingLLM-as-a-Judge LLM Reliability
University of Pennsylvania
Self-CTRL: Self-Consistency Training with Reinforcement Learning
Jun 16, 2026Itamar Pres, Laura Ruis, Melat Ghebreselassie +2Self-Consistency in Language Models Faithfulness of Language Model Explanations
MIT CSAIL
The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes
Jun 16, 2026Marina Mancoridis, Zoë HitzigLLM Evaluation Reasoning Consistency in Language Models
Massachusetts Institute of Technology · Harvard Society of Fellows
Agreement in Representation Space for Open-Ended Self-Consistency
Jun 10, 2026Paula Ontalvilla, Gorka Azkune, Aitor OrmazabalOpen-Ended Generation Self-Consistency Decoding
HiTZ Center - Ixa, University of the Basque Country (UPV/EHU)
Boosting Self-Consistency with Ranking
Jun 3, 2026Maria Marina, Daniil Moskovskiy, Sergey Pletenev +3Self-Consistency Decoding LLM Reranking
AIRI · Skoltech · EPFL
Self-Consistency via Marginal Sharpening
May 27, 2026Aleksei Arzhantsev, Otmane Sakhi, Nicolas ChopinEfficient Language Model Reasoning Reasoning Consistency in Language Models
Criteo AI Lab, CREST IP Paris · Criteo AI Lab · ENSAE, Institut Polytechnique de Paris

Oct 19, 2026J/K move · Enter open · S save

Which Answers Agree? Evaluating Self-Consistency from Incomplete Logs

Zhengtao Gui, Zexi Chen, Weining Shen

University of California, Irvine

Self-Consistency in Language Models Language Model Reasoning Evaluation

Open paper AskPDF arXiv

Self-consistency answers a question by generating multiple responses and selecting the most frequent answer. Its evaluation requires both correctness labels and information about which responses agree; correctness labels alone generally determine neither voting accuracy nor the benefit of additional votes. For tasks in which all correct responses map to the same normalized answer, we derive sharp accuracy bounds using classical stochastic majorization, refined by abstention counts and answer-category limits. We also derive sharp bounds on accuracy changes across voting budgets that share a response pool. When answer-category counts are available, we introduce Plurality-U, a count-based dynamic program that exactly averages voting accuracy over all fixed-size subsets, eliminating finite-pool Monte Carlo error. On 1,403,520 stored outputs from recent reasoning models in Scorio Lite and Scorio Trace, recording abstention counts and answer options cuts the width of SuperGPQA accuracy intervals by more than half. For moving from 9 to 17 votes, our bounds reduce interval width by a median of 77.7% on SuperGPQA and 72.3% on 14 GPQA and Adaptive-Consistency pools, compared with a baseline that combines separate accuracy bounds. Exact computation has lower paired median runtime than 2,000-subset Monte Carlo in 14 of 15 tested pool-budget settings. These results clarify what incomplete logs can establish and provide an exact evaluation procedure when answer counts are retained.