What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Content
Authors: Marios Papamichalis, Regina Ruane
Organizations: Human Nature Lab, Yale University, New Haven, CT 06511, USA. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA.
Entropy over chain-of-thought tokens decides which tokens receive the policy gradient, which get pruned, and whether a run has collapsed, yet each such statistic reads a next-token distribution mixing three choices: whether to emit connective scaffolding, which connective, and what the substantive continuation should be. Designating a scaffold vocabulary subset separates the three, exactly, for entropy, Kullback--Leibler divergence, and the first-order entropy velocity of a softmax policy. We prove the raw and content conventions disagree about which position is the larger fork on an explicit open region, and bound answer diversity by the content channel plus a leakage term a measured witness certifies. Across twenty-three configurations the scaffold side carries up to 41% of the raw high-entropy set; on a matched-tokenizer ladder, coupling changes only at the math-corpus step while the scaffold's entropy share keeps growing through distillation; a closed-form forecast from one channel correlation tracks selection retention over a 54-point range to five points, unfitted. On compression, the content convention beats raw surprisal in every cell; an answer-leakage audit then corrects our own headline control: re-fed chains earn a quarter to a half of their accuracy from restated answers, and once stripped, no token scorer beats a random contiguous block.
Figures & tables
Figure 1: The conflation, exactly and at model scale. (a) A two-position witness (closed form, Theorem 1 ): at a thought boundary t the gate is a coin flip and the connective choice is open while the continuation is decided ( HC=0.08 nats); mid-computation at u the scaffolding is quiet and the content forks ( HC=1.34 ). Raw entropy ranks t above u , selecting the position whose content uncertainty is 16× smaller. (b) Seventeen measured cells, fixed definition (Table 1 lists fourteen): as the scaffold side takes more of mean H , the raw top- 20% set becomes correspondingly scaffold-driven.
model (data, mode)
sˉ
Hˉ
gate%
scaf%
cont%
flip%
ret@20%
scaf-drv%
real- S %
base models, teacher-forced on GSM8K reference solutions (Pythia-70M, -160M, and -1B in App. G )
GPT-2
0.16
2.89
7.8±0.0
6.3±0.0
86.0±0.0
13.1±0.2
69.1±0.2
0.0±0.0
13.2±0.1
Pythia-410M
0.18
2.25
9.8±0.0
8.1±0.0
82.0±0.0
12.5±0.1
70.1±0.1
0.4±0.0
13.4±0.2
Pythia-1.4B
0.19
1.86
10.9±0.0
9.7±0.0
79.4±0.0
12.9±0.1
65.6±0.2
3.0±0.0
15.9±0.2
one tokenizer, one teacher-forced context set: corpus, scale, tuning, and distillation vary
Qwen2.5-1.5B
0.19
0.49
18.8±0.1
7.7±0.0
73.5±0.1
16.1±0.2
52.1±0.2
20.7±0.1
18.6±0.2
Table 1: Channel decomposition of CoT entropy on frozen models (mean ± sem over 3 seeds, fixed scaffold; pos / data sensitivity in App. G ). The middle block holds the tokenizer and the context set fixed, so each indented row differs from Qwen2.5-1.5B by the single intervention named and the last two share the Math-1.5B parent. Shares are of mean H ; flip is the fork-verdict flip rate on random position pairs; ret@ 20 is top- 20% retention between conventions; scaf-drv is the scaffold-driven share of the raw top- 20% set; real- S is its realized-scaffold share, the Wang et al. (2026b) connective observation quantified per channel.
Figure 2: Four checkpoint contrasts on one tokenizer, and only two move the channels. Teacher-forced GSM8K, three seeds, identical context set and token budget, mean ± sem; each bar is the released checkpoint differing from Qwen2.5-1.5B by the factor named, uncontrolled in whatever else that release changed. Code data, 2× parameters, and instruction tuning leave coupling (panels 2 and 3) inside noise and move retention (panel 4) by at most 2.1 points against 6.5 for the math corpus; distillation on the math corpus moves only allocation (panel 1).
R1-Distill-1.5B, GSM8K
R1-Distill, MATH
Qwen2.5-1.5B-It
arm
keep .25
keep .50
keep .75
keep .50
keep .50
scorers
k raw surprisal
12.2±0.3
20.2±0.4
46.2±0.6
15.9±0.7
25.8±1.4
e raw entropy
12.1±0.3
19.9±0.8
46.1±0.9
15.4±0.3
25.7±1.2
a attention received
14.5±1.0
25.3±0.3
44.5±1.3
19.1±1.0
11.5±0.5
k C raw score, content-only
12.9±0.7
25.5±0.7
74.4±0.7
21.9±1.4
16.0±0.2
k g gate-adj score, all
12.4±0.7
25.1±1.5
70.1±1.2
22.0±1.3
15.7±0.6
Table 2: Post-hoc CoT compression: answer accuracy (%) at matched token budgets , mean ± sem over three seeds ( 512 problems per seed on GSM8K, 256 on MATH-500); every arm keeps the same number of tokens in every cell. k is raw surprisal, the convention beneath LLMLingua-style importance scores ( Pan et al., 2024 ) (TokenSkip itself deploys a learned LLMLingua-2 classifier ( Xia et al., 2025 ) ); k C and k g are the factorial arms (raw score with content-only eligibility; gate-adjusted score with all tokens eligible); a is the receiver-attention scorer ( Bogdan et al., 2025 ) , d the content-channel scorer, last the final tokens of the chain. The lower panel reruns every arm after deleting each restatement of the gold answer and the chain’s final sentence; raw and content entropy track their surprisal twins within a point and are omitted there.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The three simulations. (a) SYN-1: sweeping the boundary-slot gate mass sˉ , the fork-verdict flip rate, the top- 20% retention between conventions, and the scaffold-driven share of the raw top- 20% set (mean ± sd, 3 seeds, 5×104 positions each). (b) SYN-2, one fixed position population transformed per arm: in the phantom arm the transformation moves only the nuisance group, so raw H falls 46% while the content channel is exactly invariant by Proposition 3 ; in the masked arm a genuine content collapse coexists with rising raw H . (c) SYN-3, end states of the in-silico GRPO 2×2 : fork-content entropy separates exploring from collapsed policies without overlap, and raw H does not, because the format-reward weight moves raw H by up to 2.5 nats at unchanged exploration.
position
s
H
Hb(s)
sHS
(1−s)HC
HC
G
t (boundary)
0.50
1.540
0.693
0.805
0.042
0.083
1.456
u (computation)
0.02
1.447
0.098
0.032
1.316
1.343
0.103
Appendix
Table 3: Exact witness for Theorem 1 (i) (the Figure 1 pair; all values in nats, computed in closed form by exact calculations). Raw entropy selects t ; the content channel says u forks 16× harder. Boldface marks each convention’s verdict.
Figure 4: Forecast against measurement, twenty-three cells, three held out. Each point is one (model, dataset, mode, scaffold) cell; the forecast uses only r=corr(H,HC) through the Gaussian specialization of Theorem 1 (iii); dashed line is y=x ; diamonds are the cells first measured after registration. (a) Retention carries no fitted parameter. (b) Flip carries the one frozen constant c=0.5988 .
Figure 5: What stripping the restated answers does to the compression arms (GSM8K, keep 0.50 , 512 problems × three seeds; solid bars as generated, hatched bars after every gold-answer restatement and the final sentence are deleted; dotted line is the stripped full-chain ceiling). (a) On the distilled model the ceiling halves and the recency arm collapses onto the random contiguous block: its tail premium was largely copyable answer text. (b) On the instruct model a real tail premium survives stripping. Numbers, budgets, and clustered intervals in Table 2 and § 5.4 .
Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss. We test the robustness of low- and high-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting. Moving from sentences to tokens, we then show that retaining low-entropy tokens seems effective only on mathematical benchmarks. We find this is due to the inherently low-entropy nature of numeric tokens, which also convey semantic content in such problems. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near-perfect full-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain.
Sara Candussio, Daniel Scalena, Luca Bortolussi +3
University of Trieste · University of Milano-Bicocca · CLCG, University of Groningen +2
This paper investigates the entropy dynamics of Chain-of-Thought (CoT) and uncovers a consistent two-phase structure: an Uncertainty Region of exploration transitioning sharply to a Confidence Region of convergence. We demonstrate that the Confidence Region possesses two critical properties: 1) High Reliability -- answers in the confidence region become highly accurate and stable, and 2) High Redundancy -- models generate unnecessary tokens long after reaching the correct answer. These properties unlock more efficient and reliable inference strategies: 1) Early Exit leverages reliability and redundancy to terminate computation safely when returns diminish, and 2)Test-Time Scaling uses the Confidence Region signal to prioritize converged trajectories. To operationalize these insights, we formulate Confidence Region detection as a sequential change-point detection problem, being the first to apply classical change-point methods to monitor CoT reasoning. Using the Cumulative Sum (CUSUM) algorithm, a statistically optimal change-point detector, we develop a training-free framework for real-time inference control. Experiments show our approach establishes a superior Pareto-frontier for early exit. CUSUM achieves 63.06% accuracy with 11.1% token reduction, outperforming DEER and Dynasor by 3.28% and 4.36% in accuracy respectively. For test-time scaling, CUSUM-weighted voting consistently outperforms self-consistency.
Ting Xu, Xu He, Yupu Lu +4
The Chinese University of Hong Kong · Huawei Technologies Ltd · School of Computing and Data Science, The University of Hong Kong
This empirical study is an independent reproduction of the dissociation Zhao reported in 2026. The shape of a large language model's chain-of-thought entropy trajectory predicts whether the final answer is correct, while the magnitude of its total entropy drop does not. The dissociation merits reproduction because the magnitude half rests on a single 300-problem run with one model at one seed, while the shape half was reported at full scale on both benchmarks and on a second model family. Registered at OSF before any confirmatory run, the reproduction crosses the complete GSM8K and MATH-500 benchmark test sets with four open-weight models including one reasoning-distilled model of a kind the original did not test. The shape signal replicates. The magnitude signal divides by setting. On the anchor model the accuracy gap between monotone and non-monotone chains is +9.6 percentage points on GSM8K and +27.5 on MATH-500, while the rank correlation of the total entropy drop with correctness is -0.018 on GSM8K and +0.414 on MATH-500. On the reasoning-distilled model the binary form of the shape signal fires on about one chain in a hundred, too few to estimate the registered contrast, while the graded violation count remains predictive there. In an exploratory comparison the final-step entropy alone outperforms the binary shape flag in all eight model-by-benchmark cells by ROC area, and in six or seven by the risk-coverage area the original reports, depending on an integration range the original does not state. The study contributes a reproduction of the shape signal at full test-set scale under seven documented protocol differences, a map of the settings where the magnitude signal holds and fails, and measurements of four protocol dependencies the original does not report.