Reasoning Consistency in Language Models

Follow this topic Ask about this topic

The field studies whether language models preserve conclusions and sound reasoning when prompts are paraphrased, evidence changes, answers are resampled, or conversations continue. Evaluations use controlled reformulations and multi-turn tests, repeated-sample agreement, and checks of reasoning traces with formal solvers, causal interventions, or evidence-grounded verifiers across math, logic, coding, and medical tasks. Methods include self-consistency and uncertainty-based answer selection, consistency-aware training and rewards, and test-time verification, repair, or early-exit systems. A central tension is that consistency can signal stability without correctness or faithful reasoning: models may repeat the same error, abstain, or agree while relying on invalid reasoning, so reliability must be assessed alongside accuracy, faithfulness, and cost.

Latest papers

123 papers24 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

STEP: Soft Trajectory Estimation of Prediction Correctness via Per-Step Probability Scoring
Oct 25, 2026Chenyang Yu, Zifan Zeng, Qunli Zhang +1Confidence Estimation in Language Models Language Model Performance Prediction
Technical University of Munich · RAMS Lab, Huawei Heisenberg Research Center
Scoped Semantic Invariance as a Foundation for LLM Reliability
Oct 23, 2026Rosario Uceda-Sosa, Maria Chang, Karthikeyan Natesan Ramamurthy +1Reasoning Consistency in Language Models LLM Reliability
IBM T. J. Watson Research Center, New York, USA
LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs
Oct 23, 2026Zenghui Zhou, Xiaoke Fang, Zheng ZhengReasoning Consistency in Language Models Language Model Reasoning Evaluation
School of Automation Science and Electrical Engineering, Beihang University Xueyuan Road, Haidian District, Beijing, 100191, China
Latent Coherence: Do Frontier Language Models' Probability Judgments Obey the Law of Total Probability?
Oct 22, 2026Peter PotashReasoning Consistency in Language Models Language Model Reasoning Evaluation
Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable
Oct 21, 2026Marius De Kuthy Meurers, Aaron Parisi, Bernd BohnetMoral Reasoning in Language Models Reasoning Consistency in Language Models
Google DeepMind
Same Answer, Different Reasons: Trace-Level Divergence in Multilingual Large Language Models
Oct 21, 2026Rui Sun, Peizhao Mei, Xiewei Cheng +4Multilingual Language Model Evaluation Reasoning Consistency in Language Models
University of Illinois Urbana-Champaign · Texas A&M University
CALM: Consistency-Aware Logit Masking in Decoding Space for Mitigating Overthinking of Large Reasoning Models
Oct 19, 2026Shengjing Liu, Feiyang Li, Yuxuan Yang +2Overthinking in Language Models Efficient Language Model Reasoning
College of Computer Science and Software Engineering, Shenzhen University
Consistency Training Suppresses Cue Following Without Teaching Cue Selectivity
Oct 19, 2026Prakhar Gupta, Terry Jingchen Zhang, Zhijing JinReasoning Consistency in Language Models Cognitive Bias in Language Models
University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford +2
When Form Changes but Logic Doesn't: Building Logic-invariant LLMs through Logical Structure
Oct 18, 2026Xuyuan Liu, Xinshuai Dong, Elynn Chen +1LLM Reasoning Structured Reasoning in Language Models
Dartmouth College · Carnegie Mellon University · New York University
Not All Test-Time Compute Is Evidence: Model-Specific Bias in LLM Reasoning
Oct 16, 2026Ming Shen, Tianjun YaoTest-Time Scaling Efficient Language Model Reasoning
Shanghai Jiao Tong University · Shenzhen Loop Area Institute (SLAI)
Competence-Based Evaluation of Logical Reasoning in Large Language Models
Oct 16, 2026Jize Jiang, Chengxiang ZhaiLLM Evaluation LLM Reasoning
University of Illinois Urbana-Champaign
When Protection Overrides Consequences: Understanding Protection Mode in Large Language Model Decision-Making
Oct 14, 2026Yuanfan Li, Junyi Chen, Qi Zhou +3LLM Decision-Making Language Model Safety Evaluation
Xiaohongshu · Shanghai Jiao Tong University · Xi’an Jiaotong University
Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
Oct 14, 2026Yuanbo Xie, Mingli Zhu, Tianyun Liu +3LLM Reliability Reasoning Consistency in Language Models
System Security Team, Li Auto Inc, China · Institute of Information Engineering, Chinese Academy of Sciences, China · School of Cyber Security, University of Chinese Academy of Sciences, China
Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives
Oct 12, 2026Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim +9Temporal Reasoning in Language Models Relation Extraction
Islamic University of Technology, Gazipur, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh
An Investigation of Model Coherence: Narrow Finetunes Contradict Themselves Under Resampling
Oct 8, 2026Robert Graham, Yariv Barsheshat, Phil Blandfort +1Reasoning Consistency in Language Models Self-Consistency in Language Models
Independent · Predictably Weird · Amazon
Training Language Models To Be Coherent Decision-Makers
Oct 6, 2026Khurram Yamin, Xavier Fernandes, Paul Koch +2LLM Decision-Making Decision-Making under Uncertainty
Carnegie Mellon University · Microsoft
TCMClinicalReason-Bench: Can Language Models Reason from Pathogenesis to Prescription over Real-World Clinical Cases?
Oct 3, 2026Jirui Dai, Chenkai Zhang, Yan Jia +4Traditional Chinese Medicine Clinical Language Model Evaluation
the Department of Computer Science, Johns Hopkins University, Baltimore, USA · the School of Pharmacy, Nanjing University of Chinese Medicine, Nanjing, China · the School of Information Engineering, Huzhou University, Huzhou, China +3
Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability
Oct 2, 2026Samuel Lewis-Lim, Xingwei Tan, Mario Sanger +2Faithfulness of Language Model Reasoning Efficient Language Model Reasoning
School of Computer Science, University of Sheffield · AstraZeneca
Silent Dissent: LLM Agents That Yield to the Majority Still Represent Their Original Premise
Oct 2, 2026Ziang Ni, Peng ZouNormative Conformity in Language Models Multi-Agent Debate
Sun Yat-sen University Guangzhou, China
Understanding as No-Arbitrage: Bounded Dutch Books as a Definition and Training Objective for Language Models
Sep 30, 2026Daniel DragonevskiyAdversarial Training Reasoning Consistency in Language Models
Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
Sep 30, 2026Guang Yang, Homa Hosseinmardi, Fengchen Liu +1LLM Sycophancy LLM Reliability
OASIS Lab, University of California, Los Angeles · University of California, Berkeley
When Scientific Contradictions Are Lost in Translation
Sep 29, 2026Tal Zeevi, Trey W. Jensen, Maxwell StromeReasoning Consistency in Language Models Scientific Reasoning in Language Models
ACTR: Aligning Thoughts and Responses for Multilingual Safety in Reasoning LLMs
Sep 29, 2026Xianhui Zhang, Jian Yu, Chengyu Xie +6Adversarial Attacks on LLMs LLM Safety Alignment
Nanjing University of Science and Technology, Nanjing, China · National University of Singapore, Singapore · Beihang University, Beijing, China
Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning
Sep 28, 2026Yingjin Song, Denis Paperno, Albert GattVLM Interpretability Spatial Reasoning
Utrecht University Utrecht, The Netherlands
Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces
Sep 27, 2026Han Chen, Yingrui LiReasoning Consistency in Language Models LLM Reliability
Independent Researcher
Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?
Sep 27, 2026Keyi Li, Yihao He, Quanyi LiReasoning Consistency in Language Models
Software College, Northeastern University Shenyang, China
Diagnosing Sampled LLM Reasoning in Formal Geometry: Coverage, Realization, and Validity Evidence
Sep 26, 2026Xiao Yue, Guangzhi QuLLM Evaluation Reasoning Consistency in Language Models
Oakland University, Rochester, United States
Right Answer, Wrong Reason: Accuracy, Consistency, and Consensus Are Misleading Indicators of LLM Faithfulness in Clinical Decision Support
Sep 26, 2026Bharath Kumar Bolla, Chaitali Deb Purkayastha, Vishnu Surya Reddy NandiFaithfulness of Language Model Explanations Clinical Language Model Evaluation
Institute of Product Leadership, Bengaluru, India · Independent Researcher, Bengaluru, India · Amazon, Inc, Seattle, USA
Reasoning Concentrates Errors, and Self-Consistency Never Notices
Sep 25, 2026Asaad AlthoubiCorrelated Errors in Language Models Self-Consistency in Language Models
Department of Computer Science Oklahoma State University Stillwater, OK, USA
Not What You Meant: Can LLMs Follow a Specified Negation Semantics?
Sep 23, 2026Qiming Bao, Agnieszka Mensfelt, Michael J. Witbrock +1Nonmonotonic Reasoning Reasoning Consistency in Language Models
University of Auckland, New Zealand · Royal Holloway, University of London, United Kingdom

Oct 25, 2026J/K move · Enter open · S save

STEP: Soft Trajectory Estimation of Prediction Correctness via Per-Step Probability Scoring

Chenyang Yu, Zifan Zeng, Qunli Zhang, Feng Liu

Technical University of Munich · RAMS Lab, Huawei Heisenberg Research Center

Confidence Estimation in Language Models Language Model Performance Prediction Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Estimating the correctness of large language model (LLM) predictions without access to ground truth is critical for reliable deployment in high-stakes settings. Existing approaches, such as entropy and perplexity, operate solely on final output statistics, discarding the rich information encoded in intermediate reasoning steps. To address this limitation, we propose STEP (Soft Trajectory Estimation of Prediction correctness), a training-free framework that scores each reasoning step via lightweight forward passes to obtain a soft probability distribution qi over candidate answers, from which we derive a set of soft trajectory features capturing per-step confidence and its progression toward the final answer. Extensive experiments on various benchmarks demonstrate consistent and substantial improvements, particularly in AUROC, over output-based baselines and hard-indicator trajectory methods. Notably, Temporally-Weighted Consistency, which assigns higher importance to later reasoning steps where answer distributions have stabilized, contributes the largest gains, highlighting the importance of temporal structure in reasoning trajectories. These results further validate the advantage of preserving continuous probability signals over binary discretization.