Reasoning Evaluation

Follow this topic Ask about this topic

Momentum

1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 35

All topics

REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing
Sep 29, 2026Jiawen Tao, Xiaokun Yuan, Yaoming Li +4Multi-Hop QA Reasoning Evaluation
Hunyuan Team, Tencent · Peking University
Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces
Aug 12, 2026Congchao Wang, Diwakar Singh, Qiaozi Gao +3LLM-as-a-Judge Reasoning Evaluation
Amazon AGI
Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
Aug 5, 2026Yinghui He, Ling Yang, Jiarui Liu +6LLM Evaluation Reasoning Evaluation
Princeton University · Carnegie Mellon University · University of Toronto +3
Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks
Aug 4, 2026Ine Gevers, Walter DaelemansLLM Evaluation Reasoning Evaluation
CLiPS, University of Antwerp
Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks
Aug 3, 2026Xuan Ren, Weiqi Zhai, Tianle Pu +4LLM Answer Verification Reasoning Evaluation
1Alibaba Group · 2Alibaba DAMO Academy
Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
Aug 3, 2026Mengting Ai, Jingrui He, Yue GuoCoT Faithfulness Reasoning Evaluation
University of Illinois Urbana-Champaign
A Reference-Free Score for Detecting Silent Reasoning Failures in Large Language Models
Jul 28, 2026Vivek Shukla, Varun Shukla, Atul +2CoT Faithfulness Mathematical Reasoning Benchmarks
Allenhouse Institute of Technology Kanpur, India
ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation
Jul 25, 2026Ahmed Haj Ahmed, Alvin GrissomMultilingual Language Model Evaluation Reasoning Evaluation
Haverford College
Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
Jul 22, 2026Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2LLM-as-a-Judge Reasoning Evaluation
Queen Mary University of London · Temple University
TRACE: An Operational Reasoning Schema for Auditable Agentic Commitments
Jul 14, 2026Edward Y. Chang, Emily J. ChangData Provenance Reasoning Evaluation
Stanford University · Quadrivium AI
Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
Jul 9, 2026Riccardo Revalor, Jalees Rehman, Debjit PalCoT Faithfulness LLM Reasoning with Graphs
¶Department of Electrical and Computer Engineering · ‡Department of Biochemistry and Molecular Genetics
Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts
Jun 22, 2026Steven Young EuligAI Agent Reliability Reasoning Evaluation
Department of Physics and Laboratory for Particle Physics and Cosmology (LPPC), Harvard University, Cambridge, MA 02138, USA.
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
Jun 19, 2026Zhangwei Cao, Shuhan Fan, Yuting Wei +5Benchmark Construction Reasoning Evaluation
University of International Relations · 2Kedge Business School · 3Peking University +3
Beyond Accuracy: Measuring Logical Compliance of Predictive Models
Jun 18, 2026Guillaume Olivier Delplanque, Pierre Genevès, Nabil Layaïda +1Reasoning Evaluation
LIG · LIG,TYREX
AudioProcessBench: Benchmark for Identifying Process Errors in Audio-Grounded Reasoning
Jun 7, 2026Xiangyu Zhao, Junyu Yan, Yaling Shen +7Audio-Language Model Evaluation Reasoning Evaluation
Monash University · Xi’an Jiaotong-Liverpool University · Orygen +1
Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models
Jun 1, 2026Simone Caldarella, Davide Talon, Rahaf Aljundi +2LLM Evaluation Overthinking in Language Models
University of Trento · 3Fondazione Bruno Kessler · 2Toyota Motor Europe
An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
May 31, 2026Mingzhong Sun, Teresa Yeo, Armando Solar-Lezama +1Reasoning Evaluation LLM Interpretability
1NUS Department of Computer Science · 4Singapore-MIT Alliance for Research and Technology (SMART) · 2MIT EECS +1
The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure
May 27, 2026Yubo Li, Ramayya Krishnan, Rema PadmanCoT Faithfulness Reasoning Evaluation
Carnegie Mellon University
ProvMind: Provenance-grounded reasoning for materials synthesis
May 27, 2026Yiming Zhang, Ryo Tamura, Koji TsudaReasoning Evaluation Materials Science
Graduate School of Frontier Sciences, The University of Tokyo, Chiba, Japan. · Center for Basic Research on Materials, National Institute for Materials Science, Tsukuba, Ibaraki, Japan. · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan.
The Shape of Overthinking: Backtracking Bursts in Long Reasoning Traces
May 27, 2026Navid Rezazadeh, Arash Gholami DavoodiReasoning Evaluation Efficient Language Model Inference
University of California, Irvine · Carnegie Mellon University
Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning
May 26, 2026Zhe Yu, Wenpeng Xing, Yunzhao Wei +4Compositional Reasoning Reasoning Evaluation
Binjiang Institute of Zhejiang University · Zhejiang University · Hong Kong Baptist University +2
Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning
May 20, 2026Lukas Twist, Helen Yannakoudakis, Jie M. ZhangSupervised Fine-Tuning Reasoning Evaluation
King’s College London London, UK
Strikingness-Aware Evaluation for Temporal Knowledge Graph Reasoning
May 13, 2026Rikui Huang, Shengzhe Zhang, Wei WeiReasoning Evaluation Temporal Link Prediction
School of Computer Science & Technology, Huazhong University of Science and Technology · Institute of Artificial Intelligence, Huazhong University of Science and Technology · School of Artificial Intelligence & Automation, Huazhong University of Science and Technology
Measuring AI Reasoning: A Guide for Researchers
May 4, 2026Munachiso Samuel Nwadike, Zangir Iklassov, Kareem Ali +2Reasoning Evaluation LLM Reasoning
MBZUAI, Department of Natural Language Processing · MBZUAI, Department of Machine Learning
ISAAC: Auditing Causal Reasoning in Deep Models for Drug-Target Interaction
May 3, 2026Barbara Tarantino, Sun Kim, Yijingxiu Lu +1Reasoning Evaluation Causal Interpretability
Department of Economics and Management, University of Pavia, Pavia, Italy · Department of Computer Science and Engineering, Seoul National University, Gwanak-gu 08826, Korea.
The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoning
May 3, 2026Kwan Soo ShinReasoning Evaluation Evidence-Grounded Reasoning
PolymathMinds AI Lab
Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks
Apr 21, 2026Jing Jin, Hao Liu, Yan Bai +9Reasoning Evaluation Multimodal Reasoning
1Tsinghua University · 3Central South University · 2Meituan Inc +3
From Natural Language to Executable Narsese: A Neuro-Symbolic Benchmark and Pipeline for Reasoning with NARS
Apr 20, 2026Mina Gabriel, Pei WangLogical Reasoning Reasoning Evaluation
Temple University, Philadelphia, PA 19122, USA
STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs
Apr 20, 2026Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur +2LLM Evaluation Reasoning Evaluation
IBM Research
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
Apr 13, 2026Manas Pathak, Xingyao Chen, Shuozhe Li +2LLM Evaluation Language Model Generation Evaluation
University of Texas at Austin
A Very Big Video Reasoning Suite
Feb 23, 2026Maijunxian Wang, Ruisi Wang, Juyi Lin +53Reasoning Evaluation Video Reasoning
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
Feb 10, 2026Pei-Chi Pan, Yingbin Liang, Sen LinReward Modeling Reward Hacking
Department of Computer Science University of Houston · The Ohio State University · University of Houston
AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs
Jan 20, 2026Aahana Basappa, Pranay Goel, Anusri Karra +3Reasoning Evaluation Multimodal Large Language Models
Centennial High School, Frisco, Texas, USA · Algoverse AI Research, Palo Alto, Cali- fornia, USA · Lebanon Trail High School, Frisco, Texas, USA +1
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
Oct 10, 2025Yuchen Lu, Run Yang, Yichen Zhang +6LLM Evaluation Reasoning Evaluation
1Shanghai University of Finance and Economics
Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks
Apr 10, 2025Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4Benchmark Auditing Reasoning Evaluation
1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France

Sep 29, 2026, cs.AIJ/K move · Enter open · S save

REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing

Jiawen Tao, Xiaokun Yuan, Yaoming Li, Chenxu Liu+3

Hunyuan Team, Tencent · Peking University

Multi-Hop QA Reasoning Evaluation Multi-Hop Reasoning

Open paper AskPDF arXiv

Complex questions often require multi-hop reasoning that connects facts distributed across sources or distant regions of a long context through intermediate steps. Benchmarks commonly evaluate this ability with questions built around predefined reasoning chains, treating a correct answer as evidence that the intended composition was used. Yet answer correctness alone leaves open whether success depends on the evidence associated with each intended step: models may instead rely on memorized associations, shorter paths, or partial evidence. We examine this dependence using the Behavioral Necessity Rate (BNR), which measures how often targeted evidence removal prevents answer recovery on initially correct instances. Across five existing benchmarks, panel-mean BNR ranges from 16.6% to 48.9%, exposing a substantial gap between annotated structure and observed dependence. Guided by this diagnosis, we introduce REALHOP, a diagnose-construct-verify framework that rebinds entities, factorizes selected relations, adds complete competing paths, and places evidence at traceable locations. Structural and semantic checks precede freezing; behavioral interventions follow. On 790 paired MuSiQue questions, REALHOP raises panel-mean BNR from 27.4% to 94.4% while retaining high Full accuracy. It also yields high BNR on REALHOP-FRAMES and REALHOP-LONGBENCH. On 216 long-context questions, the matched multiple-choice spread across 16 models grows from 13.9 to 59.2 points and persists under repeated open-ended evaluation. Together, these results show that a conceptually coherent chain and a correct final answer do not by themselves establish multi-hop reasoning. Verifying that success depends on every intended hop is therefore as fundamental to multi-hop evaluation as measuring answer accuracy itself.

Momentum

Related topics

Latest papers 35