Reasoning Evaluation

Follow this topic Ask about this topic

Reasoning evaluation asks not only whether an AI gives the right answer, but how it gets there, what evidence it uses, and whether its skills transfer to new settings. Researchers build benchmarks for mathematics, science, medicine, commonsense, and multimodal tasks, including interactive tests and controlled variations that expose specific skill gaps. They assess intermediate steps with process rewards, trace audits, structured rubrics, and interventions that test whether answers depend on intended evidence. A central challenge is that accuracy and fluent explanations can hide shortcuts or invalid reasoning, while scalable, trustworthy evaluation remains difficult across domains and models.

Latest papers

51 papers3 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

Strikingness-Aware Evaluation for Temporal Knowledge Graph Reasoning
May 13, 2026Rikui Huang, Shengzhe Zhang, Wei WeiReasoning Evaluation Temporal Link Prediction
School of Computer Science & Technology, Huazhong University of Science and Technology · Institute of Artificial Intelligence, Huazhong University of Science and Technology · School of Artificial Intelligence & Automation, Huazhong University of Science and Technology
Measuring AI Reasoning: A Guide for Researchers
May 4, 2026Munachiso Samuel Nwadike, Zangir Iklassov, Kareem Ali +2Reasoning Evaluation LLM Reasoning
MBZUAI, Department of Natural Language Processing · MBZUAI, Department of Machine Learning
ISAAC: Auditing Causal Reasoning in Deep Models for Drug-Target Interaction
May 3, 2026Barbara Tarantino, Sun Kim, Yijingxiu Lu +1Reasoning Evaluation Causal Interpretability
Department of Economics and Management, University of Pavia, Pavia, Italy · Department of Computer Science and Engineering, Seoul National University, Gwanak-gu 08826, Korea.
The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoning
May 3, 2026Kwan Soo ShinReasoning Evaluation Evidence-Grounded Reasoning
PolymathMinds AI Lab
Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks
Apr 21, 2026Jing Jin, Hao Liu, Yan Bai +9Reasoning Evaluation Multimodal Reasoning
1Tsinghua University · 3Central South University · 2Meituan Inc +3
From Natural Language to Executable Narsese: A Neuro-Symbolic Benchmark and Pipeline for Reasoning with NARS
Apr 20, 2026Mina Gabriel, Pei WangLogical Reasoning Reasoning Evaluation
Temple University, Philadelphia, PA 19122, USA
STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs
Apr 20, 2026Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur +2LLM Evaluation Reasoning Evaluation
IBM Research
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
Apr 13, 2026Manas Pathak, Xingyao Chen, Shuozhe Li +2LLM Evaluation Language Model Generation Evaluation
University of Texas at Austin
Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation
Mar 13, 2026Wayner Barrios, SouYoung JinReasoning Evaluation MLLM Reasoning
Dartmouth College, Hanover, NH, USA
Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning
Mar 10, 2026Tiehua Mei, Minxuan Lv, Leiyu Pan +5RL for Language Model Reasoning LLM Reasoning
School of Data Science, Fudan University · University of Chinese Academy of Sciences · College of Intelligence and Computing, Tianjin University
Common Sense vs. Morality: The Curious Case of Narrative Focus Bias in LLMs
Mar 10, 2026Saugata Purkayastha, Pranav Kushare, Pragya Paramita Pal +1Moral Reasoning in Language Models Language Model Bias Evaluation
Department of Language Science and Technology, Universität des Saarlandes · TU Darmstadt, Germany.
X-RAY: Mapping LLM Reasoning Capability via Formalized and Calibrated Probes
Mar 5, 2026Tianxi Gao, Yufan Cai, Yusi Yuan +1Structured Reasoning in Language Models Reasoning Evaluation
National University of Singapore, Singapore
Interactive Benchmarks
Mar 5, 2026Baoqing Yue, Zihan Zhu, Yutong Han +6Reasoning Evaluation
InteractiveBench · Princeton University
Chart Deep Research in LVLMs via Parallel Relative Policy Optimization
Mar 3, 2026Jiajin Tang, Gaoyang, Wenjie Wang +2Chart Understanding Large Vision-Language Models
ByteDance · ShanghaiTech University · School of Computer Science and Engineering, Sun Yat-sen University
A Very Big Video Reasoning Suite
Feb 23, 2026Maijunxian Wang, Ruisi Wang, Juyi Lin +53Reasoning Evaluation Video Reasoning
Watson & Holmes: A Naturalistic Benchmark for Comparing Human and LLM Reasoning
Feb 23, 2026Thatchawin Leelawat, Lewis D GriffinInductive Reasoning in Language Models Reasoning Evaluation
Computer Science, University College London, UK
Classroom Final Exam: An Instructor-Tested Reasoning Benchmark
Feb 23, 2026Chongyang Gao, Diji Yang, Shuyan Zhou +4MLLM Reasoning LLM Evaluation
Northwestern University · UC Santa Cruz · Duke University +3
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
Feb 10, 2026Pei-Chi Pan, Yingbin Liang, Sen LinReward Modeling Reward Hacking
Department of Computer Science University of Houston · The Ohio State University · University of Houston
AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs
Jan 20, 2026Aahana Basappa, Pranay Goel, Anusri Karra +3Reasoning Evaluation Multimodal Large Language Models
Centennial High School, Frisco, Texas, USA · Algoverse AI Research, Palo Alto, Cali- fornia, USA · Lebanon Trail High School, Frisco, Texas, USA +1
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
Oct 10, 2025Yuchen Lu, Run Yang, Yichen Zhang +6LLM Evaluation Reasoning Evaluation
1Shanghai University of Finance and Economics
Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks
Apr 10, 2025Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar +4Commonsense Reasoning Benchmark Auditing
1CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt · 2PleIAs, Paris, France

May 13, 2026J/K move · Enter open · S save

Strikingness-Aware Evaluation for Temporal Knowledge Graph Reasoning

Rikui Huang, Shengzhe Zhang, Wei Wei

School of Computer Science & Technology, Huazhong University of Science and Technology · Institute of Artificial Intelligence, Huazhong University of Science and Technology · School of Artificial Intelligence & Automation, Huazhong University of Science and Technology

Reasoning Evaluation Temporal Link Prediction Temporal KGs

Open paper AskPDF arXiv

Temporal Knowledge Graph Reasoning (TKGR) aims at inferring missing (especially future) events from historical data. Current evaluation in TKGR uniformly weights all events, ignoring that most are trivial repetitions, which overestimate the true reasoning ability. Therefore, the rare outstanding events, whose prediction demands deeper reasoning, should be distinguished and emphasized. To this end, we propose a strikingness-aware evaluation framework, which introduces a rule-based strikingness measuring framework (RSMF) to quantify event strikingness by comparing its expected occurrence with peer events derived from temporal rules. Strikingness is then integrated as a weighting factor into metrics like weighted MRR and Hits@k. Experiments on four TKG benchmarks reveal: 1) All representative models perform worse as event strikingness increases, 2) Path-based methods excel on low-strikingness events and representation-based ones on high-strikingness events, 3) We design an ensemble method whose gains stem from fitting trivial events rather than reasoning improvement. Our framework provides a more rigorous evaluation, refocusing the field on predicting outstanding events.