cs.CLMar 2, 2026
SaveMarkovian ODE-guided scoring can assess the quality of offline reasoning traces in language models
Organizations: Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, New Delhi, India · Indian Institute of Technology Delhi, Abu Dhabi, UAE
Abstract
Reasoning traces produced by generative language models are increasingly used for tasks ranging from mathematical problem solving to automated fact checking. However, existing evaluation methods remain largely mechanical and fail to capture human-centric notions of reasoning quality in a way that generalizes across varied and progressively degraded reasoning. We introduce MarODE, an offline evaluation framework that assigns quality scores to reasoning traces. Its effectiveness is assessed using human-centric perturbations and human judgments, which jointly evaluate the fundamental dimensions of an evaluation metric - goodness and soundness. The approach is grounded in a Markovian formulation of reasoning progression and an ordinary differential equation based characterization of trace dynamics, enabling efficient evaluation of reasoning quality. In a large-scale evaluation, MarODE outperforms existing baselines by over 250% under Somers' D correlation. Our results emphasize the value of theory-driven evaluation frameworks as reasoning traces become central to language model-based systems.
Figures & tables
| Metric | Qwen-3B-CoT | DeepSeek-Qwen-7B | Deepseek-Qwen-14B | Deepseek-LLaMA-8B | GPT-OSS-20B | LIAR (1-shot) | LIAR (2-shot) | LIAR (4-shot) | PolitiFact (1-shot) | PolitiFact (2-shot) | PolitiFact (4-shot) |
| ROSCOE-SA | 0.1187 | 0.1053 | 0.1111 | 0.1117 | 0.1048 | 0.1004 | 0.0983 | 0.1075 | 0.1145 | 0.1152 | 0.1281 |
| ROSCOE-SS | 0.1294 | 0.1242 | 0.1301 | 0.1256 | 0.1284 | 0.1167 | 0.1146 | 0.1386 | 0.1264 | 0.1350 | 0.1331 |
| ROSCOE-LI | 0.0318 | 0.0163 | 0.0496 | 0.0172 | 0.0146 | 0.0253 | 0.0279 | 0.0286 | 0.0283 | 0.0256 | 0.0200 |
| ROSCOE-LC | -0.0189 | -0.0009 | -0.0223 | 0.0228 | -0.0173 | -0.0073 | -0.0076 | -0.0184 | 0.0002 | -0.0047 | -0.0066 |
| ROSCOE_MEAN | 0.0819 | 0.0686 | 0.0958 | 0.0689 | 0.0622 | 0.0691 | 0.0728 | 0.0776 | 0.0793 | 0.0785 | 0.0742 |
| LLM_as_a_Judge | 0.0436 | 0.0271 | 0.0421 | 0.0375 | 0.0330 | 0.0367 | 0.0282 | 0.0140 | 0.0516 | 0.0455 | 0.0407 |
Table 1: Somers’ correlations measuring the association between human-centric perturbation scores and evaluation metrics across different backbone models and prompting settings. Columns correspond to reasoning traces generated by a range of LLMs as well as fact-checking benchmarks ( LIAR and PolitiFact ) under varying shot configurations. Higher values indicate stronger sensitivity to perturbation-induced changes in reasoning quality. For each column, the three highest correlations are highlighted in bold , while underlined values denote correlations that are not statistically significant ( ). Results include ROSCOE variants, LLM-based and coherence-oriented baselines, ReCEval, and MarODE with its component-wise and pairwise ablations, enabling a detailed analysis of goodness across models and prompting regimes.
| Metric | ENTAILMENT_BANK | PROOFWRITER | GSM | STRATQA |
|---|---|---|---|---|
| ROSCOE-SA | -0.10337 | 0.17031 | -0.04427 | 0.06290 |
| ROSCOE-SS | -0.04991 | 0.15439 | -0.03248 | 0.01250 |
| ROSCOE-LI | 0.03013 | 0.05386 | -0.02203 | 0.06036 |
| ROSCOE-LC | 0.01167 | 0.09714 | -0.05359 | 0.01563 |
| ROSCOE_MEAN | -0.00730 | 0.21145 | -0.03248 | 0.08869 |
| LLM_as_a_Judge | 0.07436 | 0.00000 | 0.00277 | 0.00449 |
Table 2: Somers’ correlations measuring alignment between human (manual) judgments and evaluation metrics across four reasoning benchmarks: EntailmentBank , ProofWriter , GSM , and StrategyQA . Higher values indicate stronger agreement with human assessments. For each dataset, the three highest correlations are highlighted in bold . Underlined values denote correlations that are not statistically significant ( ). Results include ROSCOE variants, aggregated and LLM-based baselines, ReCEval, and MarODE along with its component-wise and pairwise ablations, enabling comparison of overall performance as well as the contribution of individual reasoning dimensions.
Figure 1: Distribution of reasoning trace quality scores across different prompting regimes. Violin plots illustrate the score distributions for five evaluation metrics – ROSCOE, LLM-as-a-Judge, Local and Global Coherence, ReCEval, and MarODE, under three prompting settings (1, 2, and 4-shot) used to generate reasoning traces. For a given baseline, the overall shape of the distribution remains largely consistent across different shot counts, suggesting relative stability of the metric with respect to prompting variation. In contrast, distinct baselines exhibit markedly different distributional shapes, reflecting differences in scale, sensitivity, and scoring behavior across different evaluation approaches.
| Baselines | (4-shot – 1-shot) | (2-shot – 1-shot) | (4-shot – 2-shot) | |||
|---|---|---|---|---|---|---|
| Median Diff | Significant | Median Diff | Significant | Median Diff | Significant | |
| ROSCOE-SA | -0.000769 | TRUE | 0.000777 | TRUE | -0.001501 | TRUE |
| ROSCOE-SS | 0.001300 | TRUE | -0.001400 | TRUE | 0.002700 | TRUE |
| ROSCOE-LI | -0.000336 | FALSE | -0.002918 | TRUE | 0.004092 | TRUE |
| ROSCOE-LC | -0.001100 | TRUE | -0.001100 | TRUE | 0.000200 | TRUE |
| ROSCOE_MEAN | -0.000374 | FALSE | -0.001979 | TRUE | 0.001659 | TRUE |
Table 3: Median score differences and statistical significance ( ) from paired Wilcoxon signed-rank tests across three comparison settings: (4–1), (2–1), and (4–2). Each setting contrasts reasoning traces generated under different shot conditions, with positive or negative median differences indicating higher or lower metric scores respectively for the former condition in each pair. The Significant column denotes whether the observed median difference is statistically reliable. Results are reported for individual ROSCOE variants, aggregated baselines, LLM-based and coherence-focused evaluators, ReCEval, and MarODE along with its component wise and pairwise ablations, enabling a fine-grained analysis of sensitivity to changes in reasoning quality.
Figure 2: Comparative performance of post-hoc reasoning evaluation metrics across datasets and models, visualized as radar charts. Each subplot corresponds to a specific dataset (LIAR and PolitiFact under 1, 2, and 4-shot prompting), with axes representing the five backbone models. Colored lines denote metric-wise Somers’ correlations with human-centric perturbations for ROSCOE, Local-Global, LLM-as-a-Judge, ReCEval, and our proposed MarODE. MarODE consistently exhibits the highest correlations across datasets, shot counts, and model families, indicating stronger sensitivity to perturbations that affect reasoning quality.
Figure 3: An overview of the MarODE framework , which integrates three complementary components – Markovian Coherence , Directional Consistency , and Evidence Alignment , to capture a generalist notion of completeness and soundness in reasoning traces. (a) Markovian Coherence models the local flow of reasoning, where each step in a high-quality reasoning trace is meaningfully connected to its predecessor while also facilitating progression toward a successor, often with controlled informational overlap. (b) Directional Consistency enforces a crisp, one-directional progression of reasoning, ensuring that each step advances toward a conclusion rather than oscillating or regressing. (c) Redundancy , Redundant steps that fail to contribute significant new information not only increase the length of the reasoning chain but also compromise its overall quality and its ability to converge on a certain conclusion. (d) Evidence Alignment grounds reasoning steps in supporting evidence, mitigating hallucinations that can otherwise lead to confident yet incorrect conclusions.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Directional consistencies across seven reasoning scenarios. Each subplot shows the evolution of belief under two update methods: a logistic ODE solved via Runge–Kutta (blue circles) and a multiplicative Bayesian-style update (orange squares). Scenarios include consistently supportive, consistently contradictory, unverifiable, flipping, oscillating, gradual drift, and extreme certainty. The figure highlights differences in stability and volatility across reasoning dynamics.
Figure 5: Human evaluation questionnaire used to assess the quality of reasoning traces. Annotators rate each question on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree).
| Perturbation Name | Scope | Description | Failure Mode Targeted |
|---|---|---|---|
| Penultimate Ambiguity | Local | Introduces ambiguity in the second-to-last reasoning step | Verdict justification |
| Temporal Confusion | Local | Alters or misorders temporal references | Temporal coherence |
| Unsupported Conclusion | Global | Introduces conclusions not supported by prior steps | Evidence grounding |
| Irrelevant Elaboration | Local | Adds tangential but fluent content | Logical focus |
| Duplication | Local | Repeats reasoning steps verbatim | Redundancy |
| Underspecification | Local | Removes critical explanatory details | Step completeness |
Table 4: Summary of reasoning trace perturbation functions used in this study. Each perturbation targets a specific failure mode in multi-step reasoning, spanning local step level inconsistencies, global structural disruptions, and surface level noise. Perturbations are applied sequentially without replacement to each reasoning trace, with a maximum of six successfully applied perturbations per trace, ensuring controlled yet diverse degradations of reasoning quality.
Figure 6: Prompt template used in the LLM-as-a-Judge evaluation paradigm. The evaluator model is instructed to assess the quality of a given reasoning trace and output a single scalar score on a continuous scale, without producing explanatory text. This formulation enables scalable qualitative assessment but relies on the model’s implicit evaluation heuristics.
Figure 7: Standardized prompt template used for reasoning trace generation. The prompt provides the factual claim, associated evidence texts, and ground truth verdict label, and constrains model outputs to a structured multi step reasoning format delimited by explicit markers. Few-shot exemplars guide evidence grounding and logical progression toward the final verdict, enabling deterministic extraction and structural validation of generated reasoning traces.