CoT Evaluation

CoT: Chain-of-Thought

Chain-of-thought evaluation studies not only whether a model gives the right answer, but whether its intermediate reasoning is valid, faithful to the computation, and useful for solving or checking a task. Approaches include verifying steps against executable math or domain rules, perturbing or removing parts of a trace, and using judges, probes, process metrics, or statistical models to assess reasoning quality. Benchmarks span mathematical and general question answering as well as medical and financial tasks, where answer accuracy alone can conceal flawed or unsupported rationales. A central open tension is that readable traces may not reveal which steps the model actually used, while current faithfulness detectors and trace-quality metrics can be unreliable across tasks and answer-correctness regimes.

Latest papers

21 papers3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Does Latent Chain-of-Thought Actually Reason? A Single, Unifying Taxonomy for Latent Reasoning
Oct 15, 2026Jingjie Qiu, Sheldon Yu, Xunyi Jiang +6Latent CoT Reasoning CoT Evaluation
University of California, San Diego · New York University · University of Virginia +1
Certified by Abstention: Distribution-Free Guarantees for Chain-of-Thought Verifiers at Small Calibration Budgets
Oct 7, 2026Arjun BalajiSelective Prediction Conformal Prediction
Columbia University
Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces
Sep 29, 2026Ratish Puduppully, Pranabendu Misra, Paarth Iyer +3Faithfulness of Language Model Explanations CoT Evaluation
IT University of Copenhagen · Chennai Mathematical Institute · Indian Institute of Technology Jammu +1
Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning
Sep 3, 2026Kevin Du, Alexander Hoyle, Laura Ruis +1CoT Faithfulness Process Reward Models
ETH Zürich · Work done during an internship at Cohere. · MIT +1
Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking
Aug 31, 2026Xiaoyang Chen, Jie Liu, Haijin Liang +5Learning to Rank CoT Evaluation
University of Chinese Academy of Sciences · Tencent · University of Chinese Academy of Sciences, 2Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences
What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Content
Aug 26, 2026Marios Papamichalis, Regina RuaneCoT Evaluation
Human Nature Lab, Yale University, New Haven, CT 06511, USA. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA.
Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought
Aug 25, 2026Mengzhu Xu, Jifan Gao, Xia Jiang +2CoT Faithfulness CoT Evaluation
Eindhoven University of Technology, Eindhoven, The Netherlands · Dana-Farber Cancer Institute, Boston, MA, USA
Two Regimes of Chain-of-Thought Unfaithfulness: Behavioral Detection Fails Where Models Are Wrong
Jul 26, 2026Suramya R. Angdembay, Dikshant Aryal, Nick RahimiCoT Faithfulness CoT Evaluation
The University of Southern Mississippi
RusFinChain: A Russian Benchmark for Verifiable Chain-of-Thought Reasoning in Finance with Fuzzy-Aligned Evaluation
Jul 1, 2026M. K. ArabovLogical Reasoning LLM Evaluation
Kazan Federal University · Institute of Computational Mathematics and Information Technologies · Department of Data Analysis +1
Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation
Jun 13, 2026Xian Sun, Wei Gao, Yingshuo Wang +9Language Model Generation Evaluation CoT Evaluation
Duke University · Northeastern University · University of California, Berkeley +5
TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
May 28, 2026Yundong Kim, Heyoung YangStructured Reasoning Language Model Generation Evaluation
Applied Agent Research Center, Korea Institute of Science and Technology Information (KISTI), Republic of Korea · Department of Computer Science and Engineering, University of Seoul, Republic of Korea.
ReasonOps: Operator Segmentation for LLM Reasoning Traces
May 28, 2026Daniel Lee, Owen Queen, James ZouLLM Reasoning Reasoning Trace Analysis
Stanford University
The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure
May 27, 2026Yubo Li, Ramayya Krishnan, Rema PadmanCoT Faithfulness Reasoning Evaluation
Carnegie Mellon University
Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation
May 27, 2026Zhaoyang Jiang, Xuanqi Peng, Fei Teng +5CoT Distillation CoT Evaluation
School of Health & Wellbeing, University of Glasgow, Glasgow, UK · Department of Respiratory and Critical Care Medicine, Shanghai Sixth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China +1
When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channel
May 12, 2026Wenkai Li, Fan Yang, Ananya Hazarika +2CoT Faithfulness LLM Auditing
Carnegie Mellon University · Fujitsu Research of America Inc.
The Last Word Often Wins: A Format Confound in Chain-of-Thought Corruption Studies
May 11, 2026Gabriel GarciaCoT Faithfulness CoT Evaluation
Independent Researcher
The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits
May 8, 2026Wenhua Nie, Junlin Liu, Jianan Wu +5CoT Evaluation CoT Reasoning
National Taiwan University
FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning
Apr 12, 2026Yuxi Sun, Aoqi Zuo, Haotian Xie +3CoT Evaluation Faithfulness of Language Model Explanations
Hong Kong Baptist University · The University of Melbourne · Singapore Management University
When does Chain-of-Thought Help: A Markovian Perspective
Feb 27, 2026Zihan Wang, Yijun Dong, Qi LeiMarkov Chains CoT Reasoning
New York University
Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability
Feb 19, 2026Shashank Aggarwal, Ram Vikas Mishra, Amit AwekarCoT Evaluation LLM Reasoning
Indian Institute of Technology Guwahati, Assam, India
Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length
Dec 7, 2025Zhiyu Xu, Jia Liu, Yixin Wang +1LLM Evaluation Latent Variable Models
Department of Statistics, Columbia University. · Department of Statistics, University of Michigan

Oct 15, 2026J/K move · Enter open · S save

Does Latent Chain-of-Thought Actually Reason? A Single, Unifying Taxonomy for Latent Reasoning

Jingjie Qiu, Sheldon Yu, Xunyi Jiang, Jiajin Liu+5

University of California, San Diego · New York University · University of Virginia+1

Latent CoT Reasoning CoT Evaluation LLM Interpretability

Open paper AskPDF arXiv

Latent chain-of-thought replaces written reasoning steps with continuous hidden states, giving models additional steps to compute before answering. It remains unclear what computation these steps actually perform. High task accuracy and decodable intermediate answers do not establish that a model uses its latent chain to reach an answer. We study this question across twenty-one model-task configurations covering three training recipes, three reasoning tasks, and two backbones. We combine chain deletion and content replacement with probes and trajectory analysis to test whether the chain affects the answer, what it encodes, and how its states change. On logical tasks, COCONUT often carries an answer that is already readable before the chain begins. Its states settle early, and deleting the chain leaves predictions almost unchanged. Per-step supervision produces an ordered layout of reasoning steps and can make the model depend on the chain. Yet even an ordered chain can act as a placeholder, with deletion hurting performance while content replacement leaves answers unchanged. On arithmetic tasks, all tested recipes show dependence on chain content. These results show why readable latent states alone are insufficient evidence of reasoning. We introduce a taxonomy that separates what a chain represents from how the model uses it. Seen this way, earlier reports no longer compete: placeholder tokens, collapsed superposition and step-dependent causal roles are each consistent with a different kind of chain, each obtained on a different task, training recipe or measurement. Once a study says which property it tested, whether the model uses the chain, what the chain holds, or how its state moves, claims about latent reasoning can be compared.