Reasoning Evaluation

Follow this topic Ask about this topic

Reasoning evaluation asks not only whether an AI gives the right answer, but how it gets there, what evidence it uses, and whether its skills transfer to new settings. Researchers build benchmarks for mathematics, science, medicine, commonsense, and multimodal tasks, including interactive tests and controlled variations that expose specific skill gaps. They assess intermediate steps with process rewards, trace audits, structured rubrics, and interventions that test whether answers depend on intended evidence. A central challenge is that accuracy and fluent explanations can hide shortcuts or invalid reasoning, while scalable, trustworthy evaluation remains difficult across domains and models.

Latest papers

51 papers3 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark
Oct 14, 2026Hanyao Guo, Mo Cui, Zihan Tan +21Chemical Reasoning in LLMs Scientific Reasoning in Language Models
Constructing Pareto Frontiers of Optimal Reasoning Termination Methods for Overthinking in Reasoning Models
Oct 9, 2026Weixin Guan, Liang Li, Shiyu Xie +7Overthinking in Language Models LLM Reasoning
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyberspace Security, University of Chinese Academy of Sciences, Beijing, China · University of Florida, Gainesville, FL, USA +2
REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing
Sep 29, 2026Jiawen Tao, Xiaokun Yuan, Yaoming Li +4Multi-Hop QA Reasoning Evaluation
Hunyuan Team, Tencent · Peking University
Governed Deduction: Policy-Grounded Premise Authorization Beyond Relevance
Sep 25, 2026Wesley Shu, Hsi-Ching LinAccess Control Reasoning Evaluation
The Institute of Energetic Paradigm · National Center for High-Performance Computing, Taiwan
Route-Verify-Vote: Procedure-Conditioned Self-Consistency for Mixed-Domain Reasoning
Sep 23, 2026Xinchen XiaoCommonsense Reasoning Self-Consistency in Language Models
Xinjiang University
LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator
Sep 19, 2026Yiming Feng, Naihao Deng, Yulong Chen +1Reasoning Evaluation Language Model Reasoning Evaluation
University of Michigan · University of Aberdeen · University of Cambridge
Constructing and Evaluating Clinical Reasoning Trajectories for Medical Agent
Sep 4, 2026Yunqi Zhu, Wen Tang, Jinghao Niu +2Reasoning Evaluation Clinical Reasoning
Guangzhou University, Guangzhou, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
Aug 27, 2026Bohan Yu, Shi-Yang Li, Pengfei Cao +2LLM Evaluation Reasoning Evaluation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Recursive Agentic Reasoning
Aug 25, 2026Shengxin Zhang, Xiaomin Wu, Xiyang Wu +1Agentic Reasoning Reasoning Evaluation
Google · University of Maryland
Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces
Aug 12, 2026Congchao Wang, Diwakar Singh, Qiaozi Gao +3LLM-as-a-Judge Reasoning Evaluation
Amazon AGI
Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
Aug 5, 2026Yinghui He, Ling Yang, Jiarui Liu +6LLM Evaluation Long-Horizon LLM Reasoning
Princeton University · Carnegie Mellon University · University of Toronto +3
Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks
Aug 4, 2026Ine Gevers, Walter DaelemansLLM Evaluation Reasoning Evaluation
CLiPS, University of Antwerp
Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks
Aug 3, 2026Xuan Ren, Weiqi Zhai, Tianle Pu +4LLM Answer Verification Reasoning Evaluation
1Alibaba Group · 2Alibaba DAMO Academy
Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
Aug 3, 2026Mengting Ai, Jingrui He, Yue GuoCoT Faithfulness Reasoning Evaluation
University of Illinois Urbana-Champaign
A Reference-Free Score for Detecting Silent Reasoning Failures in Large Language Models
Jul 28, 2026Vivek Shukla, Varun Shukla, Atul +2CoT Faithfulness Mathematical Reasoning Benchmarks
Allenhouse Institute of Technology Kanpur, India
ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation
Jul 25, 2026Ahmed Haj Ahmed, Alvin GrissomMultilingual Language Model Evaluation Reasoning Evaluation
Haverford College
Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
Jul 22, 2026Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2LLM-as-a-Judge Reasoning Evaluation
Queen Mary University of London · Temple University
TRACE: An Operational Reasoning Schema for Auditable Agentic Commitments
Jul 14, 2026Edward Y. Chang, Emily J. ChangData Provenance Reasoning Evaluation
Stanford University · Quadrivium AI
Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
Jul 9, 2026Riccardo Revalor, Jalees Rehman, Debjit PalCoT Faithfulness LLM Reasoning with Graphs
¶Department of Electrical and Computer Engineering · ‡Department of Biochemistry and Molecular Genetics
Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts
Jun 22, 2026Steven Young EuligAI Agent Reliability Reasoning Evaluation
Department of Physics and Laboratory for Particle Physics and Cosmology (LPPC), Harvard University, Cambridge, MA 02138, USA.
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
Jun 19, 2026Zhangwei Cao, Shuhan Fan, Yuting Wei +5Benchmark Construction Reasoning Evaluation
University of International Relations · 2Kedge Business School · 3Peking University +3
Beyond Accuracy: Measuring Logical Compliance of Predictive Models
Jun 18, 2026Guillaume Olivier Delplanque, Pierre Genevès, Nabil LayaïdaReasoning Evaluation
TYREX
AudioProcessBench: Benchmark for Identifying Process Errors in Audio-Grounded Reasoning
Jun 7, 2026Xiangyu Zhao, Junyu Yan, Yaling Shen +7Audio-Language Model Evaluation Reasoning Evaluation
Monash University · Xi’an Jiaotong-Liverpool University · Orygen +1
Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models
Jun 1, 2026Simone Caldarella, Davide Talon, Rahaf Aljundi +2LLM Evaluation Overthinking in Language Models
University of Trento · 3Fondazione Bruno Kessler · 2Toyota Motor Europe
An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
May 31, 2026Mingzhong Sun, Teresa Yeo, Armando Solar-Lezama +1Reasoning Evaluation LLM Interpretability
1NUS Department of Computer Science · 4Singapore-MIT Alliance for Research and Technology (SMART) · 2MIT EECS +1
The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure
May 27, 2026Yubo Li, Ramayya Krishnan, Rema PadmanCoT Faithfulness Reasoning Evaluation
Carnegie Mellon University
ProvMind: Provenance-grounded reasoning for materials synthesis
May 27, 2026Yiming Zhang, Ryo Tamura, Koji TsudaReasoning Evaluation Materials Science
Graduate School of Frontier Sciences, The University of Tokyo, Chiba, Japan. · Center for Basic Research on Materials, National Institute for Materials Science, Tsukuba, Ibaraki, Japan. · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan.
The Shape of Overthinking: Backtracking Bursts in Long Reasoning Traces
May 27, 2026Navid Rezazadeh, Arash Gholami DavoodiReasoning Evaluation Efficient Language Model Inference
University of California, Irvine · Carnegie Mellon University
Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning
May 26, 2026Zhe Yu, Wenpeng Xing, Yunzhao Wei +4Compositional Reasoning Reasoning Evaluation
Binjiang Institute of Zhejiang University · Zhejiang University · Hong Kong Baptist University +2
Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning
May 20, 2026Lukas Twist, Helen Yannakoudakis, Jie M. ZhangSupervised Fine-Tuning Reasoning Evaluation
King’s College London London, UK

Oct 14, 2026J/K move · Enter open · S save

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark

Hanyao Guo, Mo Cui, Zihan Tan, Leixin Sun+20

Chemical Reasoning in LLMs Scientific Reasoning in Language Models Reasoning Evaluation Multimodal Model Evaluation

Open paper AskPDF arXiv

Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still score answer endpoints alone. Such scoring conflates memorized facts with stepwise derivation, treats visual and textual inputs as interchangeable, and confounds long-horizon planning with stepwise execution. These blind spots prevent any diagnosis of where the reasoning chain breaks in a model. We present ChemArgus, a competition-level chemistry benchmark drawn from chemistry competition material, textbooks, and research papers, whose scoring replaces the answer endpoint with three measurement properties. Fine-grained scoring grades every subquestion point by point with partial credit, so a score lands at the level of the step rather than the outcome. Multimodal capability is priced on every problem containing a real image, paired with a controlled text description, so visual understanding is measured against text on identical tasks. Diagnostic attribution reads paired metrics that share one scoring unit and differ only in context construction, so a gap between two scores isolates a single cause, namely integration, propagation, or local reasoning. GPT-5.6-Sol scores 23.2% on unassisted whole-problem solving against 28.8% and 30.4% under the two sequential metrics, a gap measuring the integration burden. Through rubric partial credit and these cause-isolating score gaps, ChemArgus shows not only whether a model answers chemistry correctly but where its reasoning chain fails, so that every score it reports doubles as a diagnosis, a readout of the chain rather than a verdict, aiming to guide language models step by step on their path toward chemistry expertise, one corrected step at a time, each step priced by the rubric, point by point.