Language Model Reasoning Evaluation

Latest papers

36 papers2 papers in the last four weeks, down 60% on the four weeks before. 0.0% of all new papers.

Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models
Mar 2, 2026Arghodeep Nandi, Ojasva Saxena, Tanmoy ChakrabortyOrdinary Differential Equations Language Model Reasoning Evaluation
Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, New Delhi, India · Indian Institute of Technology Delhi, Abu Dhabi, UAE
From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
Feb 27, 2026Seungdong Yoa, Sanghyu Yoon, Suhee Yoon +4Language Model Reasoning Evaluation LLM Evaluation
LG AI Research · Division of Computer Engineering, Hankuk University of Foreign Studies
Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem
Feb 25, 2026Heejin JoStructured Reasoning in Language Models Prompt Engineering
Independent Researcher
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
Feb 19, 2026Simon Henniger, Gabriel PoesiaLanguage Model Reasoning Evaluation LLM Evaluation
Harvard University
ABCD: All Biases Come Disguised
Feb 19, 2026Mateusz Nowak, Xavier Cadet, Peter ChinLLM Evaluation Language Model Bias Evaluation
Dartmouth College
Scaling Evaluation-time Compute with Reasoning Models as Evaluators
Mar 25, 2025Seungone Kim, Ian Wu, Jinu Lee +8LLM Evaluation Language Model Generation Evaluation
CMU · UIUC2 · KAIST AI3 +3

Mar 2, 2026J/K move · Enter open · S save

Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models

Arghodeep Nandi, Ojasva Saxena, Tanmoy Chakraborty

Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, New Delhi, India · Indian Institute of Technology Delhi, Abu Dhabi, UAE

Ordinary Differential Equations Language Model Reasoning Evaluation Language Model Generation Evaluation Reasoning Trace Analysis

Open paper AskPDF arXiv

Reasoning traces produced by generative language models are increasingly used for tasks ranging from mathematical problem solving to automated fact checking. However, existing evaluation methods remain largely mechanical and fail to capture human-centric notions of reasoning quality in a way that generalizes across varied and progressively degraded reasoning. We introduce MarODE, an offline evaluation framework that assigns quality scores to reasoning traces. Its effectiveness is assessed using human-centric perturbations and human judgments, which jointly evaluate the fundamental dimensions of an evaluation metric - goodness and soundness. The approach is grounded in a Markovian formulation of reasoning progression and an ordinary differential equation based characterization of trace dynamics, enabling efficient evaluation of reasoning quality. In a large-scale evaluation, MarODE outperforms existing baselines by over 250% under Somers' D correlation. Our results emphasize the value of theory-driven evaluation frameworks as reasoning traces become central to language model-based systems.

Latest papers

Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models

From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning

Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem

The Token Games: Evaluating Language Model Reasoning with Puzzle Duels

ABCD: All Biases Come Disguised

Scaling Evaluation-time Compute with Reasoning Models as Evaluators