Language Model Reasoning Evaluation

Follow this topic Ask about this topic

Latest papers

128 papers66 papers in the last four weeks, up 2100% on the four weeks before. 0.4% of all new papers.

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains
Jul 20, 2026Liam SwayneLLM Evaluation LLM Reasoning
Mirror Horizon: Viable Path Entropy as a Measure of Bounded Reflection
Jul 11, 2026Tiantian ZhangLLM Evaluation Language Model Reasoning Evaluation
Crystal · Department of Computer Science, Columbia University
Cost of Reasoning in non-English Languages: A Case Study on Japanese
Jul 11, 2026Yuu JinnaiCross-Lingual Reasoning in Language Models Multilingual Language Model Evaluation
CyberAgent / Tokyo, Japan
SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance
Jun 8, 2026Hanna Abi Akl, Fabien Gandon, Catherine Faron +1Language Model Bias Evaluation Logical Reasoning
Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France · Data ScienceTech Institute, Paris, France
TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
May 28, 2026Yundong Kim, Heyoung YangStructured Reasoning Language Model Generation Evaluation
Applied Agent Research Center, Korea Institute of Science and Technology Information (KISTI), Republic of Korea · Department of Computer Science and Engineering, University of Seoul, Republic of Korea.
ReasonOps: Operator Segmentation for LLM Reasoning Traces
May 28, 2026Daniel Lee, Owen Queen, James ZouLLM Reasoning Reasoning Trace Analysis
Stanford University
Absurd World: A Simple Yet Powerful Method to Absurdify the Real-world for Probing LLM Reasoning Capabilities
May 10, 2026Ryan Albright, Golam Md Muktadir, Zarif Ikram +3LLM Evaluation Language Model Reasoning Evaluation
The Nueva School, San Mateo, USA · University of Southern California · Notre Dame College, Dhaka, Bangladesh +2
Evaluating Reasoning Models for Queries with Presuppositions
May 4, 2026Rose Sathyanathan, Kinshuk Vasisht, Danish PruthiLLM Evaluation Language Model Robustness
Indian Institute of Science Bengaluru, KA, India
Measuring AI Reasoning: A Guide for Researchers
May 4, 2026Munachiso Samuel Nwadike, Zangir Iklassov, Kareem Ali +2Reasoning Evaluation LLM Reasoning
MBZUAI, Department of Natural Language Processing · MBZUAI, Department of Machine Learning
Analyzing LLM Reasoning to Uncover Mental Health Stigma
Apr 27, 2026Sreehari Sankar, Aliakbar Nafar, Mona Barman +8LLM Evaluation Social Bias in Language Models
BetterHelp
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
Apr 14, 2026Yuangang Li, Justin Tian Jin Chen, Ethan Yu +2Language Model Reasoning Evaluation LLM Evaluation
University of California, Irvine
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
Apr 13, 2026Manas Pathak, Xingyao Chen, Shuozhe Li +2LLM Evaluation Language Model Generation Evaluation
University of Texas at Austin
Unified Deployment-Aware Evaluation of Open Reasoning Language Models
Apr 8, 2026Md Motaleb Hossen Manik, Ge WangLLM Evaluation Language Model Reasoning Evaluation
Department of Computer Science, Rensselaer Polytechnic Institute · Department of Biomedical Engineering, Rensselaer Polytechnic Institute
Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness
Mar 24, 2026Abhinaba Basu, Pavan ChakrabortyLanguage Model Reasoning Evaluation Structured Reasoning in Language Models
Indian Institute of Information Technology Allahabad (IIITA) · National Institute of Electronics and Information Technology (NIELIT)
RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
Mar 21, 2026Kaiyuan Li, Jing-Cheng Pang, Yang YuLanguage Model Reasoning Evaluation RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · Polixir.ai, China · Huawei Technologies Co., Ltd., China
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Mar 16, 2026Yuzhe TangCommonsense Reasoning Language Model Reasoning Evaluation
Georgia Institute of Technology
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Mar 5, 2026Siddharth Boppana, Annabel Ma, Max Loeffler +5Faithfulness of Language Model Reasoning Language Model Reasoning Evaluation
Goodfire AI · Harvard University, Cambridge, MA
Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models
Mar 2, 2026Arghodeep Nandi, Ojasva Saxena, Tanmoy ChakrabortyOrdinary Differential Equations Language Model Reasoning Evaluation
Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, New Delhi, India · Indian Institute of Technology Delhi, Abu Dhabi, UAE
From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
Feb 27, 2026Seungdong Yoa, Sanghyu Yoon, Suhee Yoon +4Language Model Reasoning Evaluation LLM Evaluation
LG AI Research · Division of Computer Engineering, Hankuk University of Foreign Studies
Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem
Feb 25, 2026Heejin JoStructured Reasoning in Language Models Prompt Engineering
Independent Researcher
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
Feb 19, 2026Simon Henniger, Gabriel PoesiaLanguage Model Reasoning Evaluation LLM Evaluation
Harvard University
ABCD: All Biases Come Disguised
Feb 19, 2026Mateusz Nowak, Xavier Cadet, Peter ChinLLM Evaluation Language Model Bias Evaluation
Dartmouth College
Sci-CoE: Co-evolving Scientific Reasoning LLMs via Geometric Consensus with Sparse Supervision
Feb 12, 2026Xiaohan He, Shiyang Feng, Songtao Huang +3Scientific Reasoning in Language Models Language Model Reasoning Evaluation
Shanghai Artificial Intelligence Laboratory · Fudan University
Disentangling generalization and memorization in large language models using chess
Jan 23, 2026Leonard S. Pleiss, Maximilian Schiffer, Robert K. von WeizsaeckerLanguage Model Generalization Memorization in Language Models
Technical University of Munich
The Imperfective Paradox in Large Language Models
Jan 14, 2026Bolei Ma, Yusuke MiyaoFaithfulness of Language Model Reasoning Language Model Reasoning Evaluation
LMU Munich & MCML · The University of Tokyo
Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought
Dec 25, 2025Yuyi Zhang, Boyu Tang, Tianjie Ju +2Faithfulness of Language Model Reasoning Implicit Reasoning in Language Models
Shanghai Jiao Tong University
Semantic Deception: When Reasoning Models Can't Compute an Addition
Dec 23, 2025Nathaniël de Leeuw, Marceau Nahon, Mathis Reymond +2Language Model Reasoning Evaluation Symbolic Reasoning in Language Models
Institute of Intelligent Systems and Robotics, CNRS, Sorbonne University, Paris, F-75005, France. · Paris Cit´e University, Paris, F-75006, France.
Benchmarking Vietnamese Legal Knowledge of Large Language Models
Dec 16, 2025Nguyen Tien Dong, Minh-Anh Nguyen, Thanh Dat Hoang +6Language Model Reasoning Evaluation Legal Reasoning
CMC OpenAI · VinUniversity · HUST, Vietnam +1
Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM
Dec 14, 2025Furong Jia, Yuan Pu, Finn Guo +1Language Model Reasoning Evaluation Clinical QA
Duke University
Cognitive Foundations for Reasoning and Their Manifestation in LLMs
Nov 20, 2025Shuyue Stella Li, Priyanka Kargupta, Haocheng Wang +9Structured Reasoning in Language Models Language Model Reasoning Evaluation
University of Washington · University of Illinois Urbana-Champaign · Princeton University +1

Jul 20, 2026J/K move · Enter open · S save

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

Liam Swayne

LLM Evaluation LLM Reasoning Language Model Reasoning Evaluation

Open paper AskPDF arXiv

Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.