Language Model Reasoning Evaluation

Follow this topic Ask about this topic

Latest papers

36 papers2 papers in the last four weeks, down 60% on the four weeks before. 0.0% of all new papers.

Reasoning Models Can Control Their Chains of Thought When They Tell Themselves To
Oct 16, 2026Uriel Dolev, Idan Schwartz, Yoav GoldbergLanguage Model Reasoning Evaluation LLM Prompting
Bar-Ilan University · Allen Institute for AI
Are Large Reasoning Models Good Everyday Reasoners? A Comparison with Non-Reasoning Models on Logical Reasoning Tasks
Oct 15, 2026Kentaro Ozeki, Risako Ando, Takanobu Morishita +3Language Model Reasoning Evaluation LLM Reasoning
Keio University
RCCM: Diagnosing Gains and Harms in Reasoning-Prompt Transfer
Oct 14, 2026Yuqi Zhang, Di ZhangLanguage Model Reasoning Evaluation LLM Reasoning
Reasoning Instructions Can Break Answer Decoding in Vision--Language Models
Sep 24, 2026Zeyan Li, Siyuan Qiu, Jianfeng XuVLM Evaluation Language Model Generation Evaluation
Shanghai Jiao Tong University
LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator
Sep 19, 2026Yiming Feng, Naihao Deng, Yulong Chen +1Reasoning Evaluation Language Model Reasoning Evaluation
University of Michigan · University of Aberdeen · University of Cambridge
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
Sep 8, 2026Jaewoo Lim, Sungbok Shin, Sanghyun HongLLM Evaluation LLM Interpretability
Oregon State University, Corvallis USA · Sogang University, Seoul, South Korea
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Aug 31, 2026Qiyao Yan, Chenpeng Wang, Liangming PanLanguage Modeling Language Model Calibration
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · YiXin-AILab, YIXIN, Beijing, China +1
Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators
Aug 30, 2026Armaan Singh, Ryan Trinh Le, Jasmine Kaur +5Implicit Reasoning in Language Models Language Model Probing
1Stanford University · 2York University · 3Facebook
ClosureBench: A Constructive Benchmark for Compositional Graph Reasoning
Aug 18, 2026Stefano GoriaLLM Reasoning with Graphs Language Model Reasoning Evaluation
AIM Research Lab
Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
Aug 18, 2026Ayoub Kirouane, Christos PetrocheilosFine-Tuning Language Models Structured Reasoning in Language Models
Sophea AI, KIEFER SA, Athens, Greece
Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation
Aug 13, 2026Junhao Luo, Ning Huang, Ziqi Sha +2LLM Evaluation LLM Auditing
School of Statistics and Data Science, Southwestern University of Finance and Economics
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLLM Evaluation Language Model Generation Evaluation
Federal University of Santa Catarina (UFSC) Brazil
Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
Jul 22, 2026Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2LLM-as-a-Judge Reasoning Evaluation
Queen Mary University of London · Temple University
Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains
Jul 20, 2026Liam SwayneLLM Evaluation LLM Reasoning
Mirror Horizon: Viable Path Entropy as a Measure of Bounded Reflection
Jul 11, 2026Tiantian ZhangLLM Evaluation Language Model Reasoning Evaluation
Crystal · Department of Computer Science, Columbia University
Cost of Reasoning in non-English Languages: A Case Study on Japanese
Jul 11, 2026Yuu JinnaiCross-Lingual Reasoning in Language Models Multilingual Language Model Evaluation
CyberAgent / Tokyo, Japan
SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance
Jun 8, 2026Hanna Abi Akl, Fabien Gandon, Catherine Faron +1Logical Reasoning Symbolic Reasoning
Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France · Data ScienceTech Institute, Paris, France
TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
May 28, 2026Yundong Kim, Heyoung YangStructured Reasoning Language Model Generation Evaluation
Applied Agent Research Center, Korea Institute of Science and Technology Information (KISTI), Republic of Korea · Department of Computer Science and Engineering, University of Seoul, Republic of Korea.
ReasonOps: Operator Segmentation for LLM Reasoning Traces
May 28, 2026Daniel Lee, Owen Queen, James ZouLLM Reasoning Reasoning Trace Analysis
Stanford University
Absurd World: A Simple Yet Powerful Method to Absurdify the Real-world for Probing LLM Reasoning Capabilities
May 10, 2026Ryan Albright, Golam Md Muktadir, Zarif Ikram +3LLM Evaluation Language Model Reasoning Evaluation
The Nueva School, San Mateo, USA · University of Southern California · Notre Dame College, Dhaka, Bangladesh +2
Evaluating Reasoning Models for Queries with Presuppositions
May 4, 2026Rose Sathyanathan, Kinshuk Vasisht, Danish PruthiLLM Evaluation Language Model Robustness
Indian Institute of Science Bengaluru, KA, India
Measuring AI Reasoning: A Guide for Researchers
May 4, 2026Munachiso Samuel Nwadike, Zangir Iklassov, Kareem Ali +2Reasoning Evaluation LLM Reasoning
MBZUAI, Department of Natural Language Processing · MBZUAI, Department of Machine Learning
Analyzing LLM Reasoning to Uncover Mental Health Stigma
Apr 27, 2026Sreehari Sankar, Aliakbar Nafar, Mona Barman +8LLM Evaluation Social Bias in Language Models
BetterHelp
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
Apr 14, 2026Yuangang Li, Justin Tian Jin Chen, Ethan Yu +2Language Model Reasoning Evaluation LLM Evaluation
University of California, Irvine
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
Apr 13, 2026Manas Pathak, Xingyao Chen, Shuozhe Li +2LLM Evaluation Language Model Generation Evaluation
University of Texas at Austin
Unified Deployment-Aware Evaluation of Open Reasoning Language Models
Apr 8, 2026Md Motaleb Hossen Manik, Ge WangLLM Evaluation Language Model Reasoning Evaluation
Department of Computer Science, Rensselaer Polytechnic Institute · Department of Biomedical Engineering, Rensselaer Polytechnic Institute
Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness
Mar 24, 2026Abhinaba Basu, Pavan ChakrabortyLanguage Model Reasoning Evaluation Structured Reasoning in Language Models
Indian Institute of Information Technology Allahabad (IIITA) · National Institute of Electronics and Information Technology (NIELIT)
RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
Mar 21, 2026Kaiyuan Li, Jing-Cheng Pang, Yang YuLanguage Model Reasoning Evaluation RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · Polixir.ai, China · Huawei Technologies Co., Ltd., China
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Mar 16, 2026Yuzhe TangLanguage Model Reasoning Evaluation LLM Evaluation
Georgia Institute of Technology
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Mar 5, 2026Siddharth Boppana, Annabel Ma, Max Loeffler +5Faithfulness of Language Model Reasoning Language Model Reasoning Evaluation
Goodfire AI · Harvard University, Cambridge, MA

Oct 16, 2026J/K move · Enter open · S save

Reasoning Models Can Control Their Chains of Thought When They Tell Themselves To

Uriel Dolev, Idan Schwartz, Yoav Goldberg

Bar-Ilan University · Allen Institute for AI

Language Model Reasoning Evaluation LLM Prompting CoT Monitoring LLM Safety Monitoring

Open paper AskPDF arXiv

Chain-of-thought (CoT) controllability, the ability of reasoning models to control properties of their CoT, is consequential for both their usefulness and reliable oversight. Recent work evaluates this capability by measuring whether models follow CoT constraints given in the user prompt, and reports very low controllability across models. We argue that this low controllability may not reflect an inherent inability of reasoning models to control their CoT, but instead result from naively specifying constraints in the user prompt. To test this, we introduce Self-Instructed CoT Controllability: a complementary evaluation that places the constraint as a self-instruction inside the model's CoT and measures its ability to follow it. We show that, under self-instruction, models achieve significantly higher controllability than with standard user instructions. Even a generic reminder to follow the user's constraint substantially improves controllability. Moreover, effective self-instructions can be bootstrapped into the reasoning through a simple instruction in the user prompt, enabling improved control even without access to the reasoning channel. Finally, using self-instruction, we demonstrate in a realistic monitoring setup that CoT controllability can be leveraged for monitor evasion, posing a risk to CoT monitorability.