Recent work uses entropy-based signals at multiple representation levels to study reasoning in large language models, but the field remains largely empirical. From a theoretical perspective, reductions in model uncertainty do not necessarily reflect progress toward correctness. In this work, we characterize when such alignment can be expected through the Stepwise Informativeness Assumption (SIA), which requires entropy-reducing reasoning-prefix extensions increase the probability assigned to valid answers in expectation. We show that next-token prediction can transfer answer-relevant information from training data to the model, but does not, by itself, guarantee alignment with correctness. Instead, we propose that this alignment depends on how the training regime couples reasoning traces to correct answers. Furthermore, we hypothesize observable signatures of this phenomenon and characterize its expected failure regimes. We empirically investigate how entropy dynamics relate to correctness across four reasoning benchmarks (GSM8K, ARC, SVAMP, MATH) and diverse open-weight language models spanning base, instruction-tuned, and RL-trained regimes. Our results show that entropy-correctness alignment strengthens with post-training but can weaken when task difficulty exceeds model capabilities.
Figures & tables
Figure 1: Overview of the proposed framework. We formalize the conditions for entropy-correctness alignment and investigate how different training stages may promote it.
Model
Training
GSM8K
SVAMP
ARC
MATH
Qwen2.5-3B
Base
0.682 [0.649, 0.714]
0.603 [0.512, 0.681]
0.344 [0.300, 0.386]
-0.0975 [-0.1939, -0.0042]
Qwen2.5-3B-it
SFT
0.744 [0.717, 0.770]
0.835 [0.771, 0.893]
0.666 [0.633, 0.701]
0.0082 [-0.0979, 0.1146]
Qwen2.5-Math-1.5B
SFT
0.499 [0.429, 0.567]
0.802 [0.643, 0.922]
0.701 [0.654, 0.747]
-0.5411 [-0.6248, -0.4579]
DeepSeek-Chat-7B
SFT
0.346 [0.305, 0.388]
0.295 [0.190, 0.414]
0.143 [0.090, 0.199]
-0.1277 [-0.1879, -0.0703]
DeepSeek-R1-Distilled
SFT+RL
0.795 [0.763, 0.829]
0.593 [0.436, 0.730]
0.783 [0.744, 0.817]
-0.7746 [-0.8462, -0.6865]
Gemma-2-2B
Base
-0.530 [-0.689, -0.346]
0.169 [-0.489, 0.844]
-0.208 [-0.266, -0.151]
-0.5595 [-0.7502, -0.3559]
Table 1: Training aligns entropy descent with the true answer. We report the correlation between conditional answer entropy and gold surprisal along each trace (SIA alignment coefficient ρSIA ), averaged by model and dataset. Negative or near-zero values indicate failure of alignment.
Figure 2: Observable signatures of entropy-correctness alignment on GSM8K. Correct traces accumulate answer-relevant information earlier and conditional answer entropy separates correct from incorrect traces before answer generation. Further results can be found in Appendix C .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Early information accumulation for aligned model-dataset pairs on GSM8K. Normalized cumulative information gain G(s) versus relative prefix length s , split by correctness. Each panel corresponds to one model. Shaded regions denote 95% bootstrap confidence intervals.
Figure 4: Early information accumulation for aligned model-dataset pairs on SVAMP. Normalized cumulative information gain G(s) versus relative prefix length s , split by correctness. Each panel corresponds to one model. Shaded regions denote 95% bootstrap confidence intervals.
Figure 5: Early information accumulation for aligned model-dataset pairs on ARC. Normalized cumulative information gain G(s) versus relative prefix length s , split by correctness. Each panel corresponds to one model. Shaded regions denote 95% bootstrap confidence intervals.
Figure 6: Early information accumulation for aligned model-dataset pairs on MATH. Normalized cumulative information gain G(s) versus relative prefix length s , split by correctness. Each panel corresponds to one model. Shaded regions denote 95% bootstrap confidence intervals.
Figure 7: Early information accumulation for non-aligned model-dataset pairs on GSM8K. Normalized cumulative information gain G(s) versus relative prefix length s , split by correctness. Each panel corresponds to one model. Shaded regions denote 95% bootstrap confidence intervals.
Figure 8: Early information accumulation for non-aligned model-dataset pairs on SVAMP. Normalized cumulative information gain G(s) versus relative prefix length s , split by correctness. Each panel corresponds to one model. Shaded regions denote 95% bootstrap confidence intervals.
Figure 9: Early information accumulation for non-aligned model-dataset pairs on ARC. Normalized cumulative information gain G(s) versus relative prefix length s , split by correctness. Each panel corresponds to one model. Shaded regions denote 95% bootstrap confidence intervals.
Figure 10: Early information accumulation for non-aligned model-dataset pairs on MATH. Normalized cumulative information gain G(s) versus relative prefix length s , split by correctness. Each panel corresponds to one model. Shaded regions denote 95% bootstrap confidence intervals.
Figure 11: Early separability for aligned model-dataset pairs on GSM8K. AUC for using conditional answer entropy to distinguish correct from incorrect traces as a function of relative prefix length s .
Figure 12: Early separability for aligned model-dataset pairs on SVAMP. AUC for using conditional answer entropy to distinguish correct from incorrect traces as a function of relative prefix length s .
Figure 13: Early separability for aligned model-dataset pairs on ARC. AUC for using conditional answer entropy to distinguish correct from incorrect traces as a function of relative prefix length s .
Figure 14: Early separability for aligned model-dataset pairs on MATH. AUC for using conditional answer entropy to distinguish correct from incorrect traces as a function of relative prefix length s .
Figure 15: Early separability for non-aligned model-dataset pairs on GSM8K. AUC for using conditional answer entropy to distinguish correct from incorrect traces as a function of relative prefix length s .
Figure 16: Early separability for non-aligned model-dataset pairs on SVAMP. AUC for using conditional answer entropy to distinguish correct from incorrect traces as a function of relative prefix length s .
Figure 17: Early separability for non-aligned model-dataset pairs on ARC. AUC for using conditional answer entropy to distinguish correct from incorrect traces as a function of relative prefix length s .
Figure 18: Early separability for non-aligned model-dataset pairs on MATH. AUC for using conditional answer entropy to distinguish correct from incorrect traces as a function of relative prefix length s .
Figure 19: LoRA SFT induces entropy-correctness alignment. We fine-tune Gemma-2-2B base on GSM8K chain-of-thought solutions using LoRA and evaluate the SIA alignment coefficient ρSIA at intermediate checkpoints. The coefficient increases from negative values early in training to consistently positive values after supervised fine-tuning, indicating that reasoning supervision causally strengthens the coupling between conditional answer entropy reduction and increasing probability of the gold answer. Shaded regions denote 95% bootstrap confidence intervals.
Figure 20: Scaling trends in entropy-correctness alignment. Mean ρSIA across datasets is plotted against model size for base and instruction-tuned variants. Larger models generally exhibit stronger entropy-correctness alignment, and instruction-tuned models consistently outperform base models within the same family. This suggests that model scale and reasoning-oriented supervision jointly determine whether entropy descent becomes aligned with the true answer.
Model
Original mean
Shuffled mean
Qwen2.5-3B
0.682
-0.132
Qwen2.5-3B-it
0.744
-0.005
DeepSeek-R1-Distilled
0.795
0.020
Gemma-2-2B-it
0.522
-0.063
Appendix
Table 2: Shuffle-prefix ablation . Entropy-correctness alignment ( ρSIA ) drops sharply when prefix tokens are permuted. Negative or near-zero values indicate coupling misalignment.
Model
Original mean
Ablated mean
Qwen2.5-3B
0.682
0.635
Qwen2.5-3B-it
0.744
0.831
DeepSeek-R1-Distilled
0.795
0.711
Gemma-2-2B-it
0.522
0.506
Appendix
Table 3: Monte-Carlo ablation. Entropy-correctness alignment ( ρSIA ) remains qualitatively stable across different fidelity Monte-Carlo approximations.
Model
T=0.2
T=0.4
T=0.6
T=0.7
T=0.8
T=1.0
Qwen2.5-3B
0.644
0.689
0.752
0.682
0.599
0.652
Qwen2.5-3B-it
0.762
0.733
0.905
0.744
0.756
0.821
DeepSeek-R1-Distilled
0.837
0.843
0.744
0.795
0.686
0.781
Gemma-2-2B-it
0.616
0.385
0.543
0.522
0.417
0.613
Appendix
Table 4: Temperature ablation . Entropy-correctness alignment ( ρSIA ) remains qualitatively stable across decoding temperatures.
Artifact
Source
License / terms
GSM8K
OpenAI
MIT License
ARC
AllenAI AI2 ARC
CC BY-SA 4.0
SVAMP
Patel et al. / HF mirror
MIT License
MATH
Hendrycks et al.
MIT License
Gemma-2-2B, Gemma-2-9B
Google
Gemma Terms of Use
LLaMA-3.2-3B
Meta
Llama 3.2 Community License
Appendix
Table 5: Licenses and usage terms for datasets and open-weight model checkpoints used in our experiments.
Chain-of-thought (CoT) reasoning can improve LLM performance, but its benefits vary across tasks, models, and individual queries. On some tasks, CoT provides marginal or negative accuracy gains while substantially increasing token consumption. We investigate whether early decoding dynamics can help identify when explicit reasoning is useful. Our analysis shows that tasks with larger CoT gains tend to exhibit decreasing early-stage entropy, while low-gain tasks more often show oscillatory or increasing trajectories. These patterns motivate an empirical regime interpretation of decoding behavior. We propose EDRM (Entropy Dynamics-based Reasoning Manifold), a lightweight routing framework whose default heuristic requires no router training. EDRM summarizes early entropy trajectories with a compact, interpretable representation and selects among Direct, Standard, and CoT decoding. Across 15 benchmarks and 7 LLMs, its global and instance-level variants offer 16--55% token savings at comparable accuracy. The Full instance-level configuration combines routing with a short Direct fallback branch; our heuristic-router ablation reports up to 4.9 percentage points of additional accuracy from oracle answer selection over routing without fallback. These results support selective reasoning based on early decoding signals and demonstrate the potential of entropy-based control for efficient LLM inference.
Wei Xia, Haoqing Wang, Zhi-Hong Deng +1
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China
Chain-of-thought reasoning provides a structured computation between a model's input and final answer. Yet it is often evaluated through endpoint accuracy, which ignores the path taken to reach that answer. An emerging line of work addresses this limitation using entropy profiles, which track how uncertainty evolves over the reasoning process but do not reveal which competing hypotheses account for that uncertainty. We introduce answer-distribution trajectories, a stochastic-dynamics-inspired representation that tracks the model's full predictive distribution over answers as reasoning unfolds. As a strictly finer representation than endpoint and entropy summaries, answer-distribution trajectories enable us to characterize a trace through a dynamical reasoning profile spanning exploration, revision, motion, and commitment, and to distinguish different dynamical mechanisms of reasoning success and failure. Across sixteen open-weight language models and four reasoning benchmarks, we show that traces with the same endpoint and similar entropy profiles can exhibit substantially different reasoning dynamics. We further find substantial variation in these dynamics both within and across models and tasks, with different objectives favoring different dynamical profiles. Additionally, we show that training and inference choices systematically reshape these profiles. Our results suggest that answer-distribution trajectories provide a rich framework for analysing and evaluating the dynamics of LLM reasoning.
Mar Gonzàlez I Català, Haitz Sáez de Ocáriz Borde, Davide Murari +3
Large Language Models (LLMs) with extended reasoning capabilities often generate verbose and redundant reasoning traces, incurring unnecessary computational cost. While existing reinforcement learning approaches address this by optimizing final response length, they neglect the quality of intermediate reasoning steps, leaving models vulnerable to reward hacking. We argue that verbosity is not merely a length problem, but a symptom of poor intermediate reasoning quality. To investigate this, we conduct an empirical study tracking the per-token predictive entropy of large reasoning models across reasoning trajectories. We find that high-quality reasoning traces exhibit two consistent properties: low uncertainty convergence and fast uncertainty descent. These findings suggest that high-quality reasoning traces are informationally dense, that is, reasoning steps contribute to reaching a low uncertainty level relative to the total reasoning length. Motivated by this, we propose InfoDensity, a reward framework for RL training that captures both properties through a single suffix-max envelope of the entropy trajectory, weighted by a length scaling term that favors achieving equivalent quality more concisely. Experiments on mathematical and general reasoning benchmarks demonstrate that InfoDensity outperforms state-of-the-art baselines on the accuracy-efficiency trade-off.
Chengwei Wei, Jung-jae Kim, Longyin Zhang +2
Institute for Infocomm Research (I2R), A*STAR, Singapore · Centre for Frontier AI Research (CFAR), A*STAR, Singapore