What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Content

Marios Papamichalis; Regina Ruane

stat.MLAug 26, 2026

What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Content

Authors: Marios Papamichalis, Regina Ruane

Organizations: Human Nature Lab, Yale University, New Haven, CT 06511, USA. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA.

Abstract

Entropy over chain-of-thought tokens decides which tokens receive the policy gradient, which get pruned, and whether a run has collapsed, yet each such statistic reads a next-token distribution mixing three choices: whether to emit connective scaffolding, which connective, and what the substantive continuation should be. Designating a scaffold vocabulary subset separates the three, exactly, for entropy, Kullback--Leibler divergence, and the first-order entropy velocity of a softmax policy. We prove the raw and content conventions disagree about which position is the larger fork on an explicit open region, and bound answer diversity by the content channel plus a leakage term a measured witness certifies. Across twenty-three configurations the scaffold side carries up to 41% of the raw high-entropy set; on a matched-tokenizer ladder, coupling changes only at the math-corpus step while the scaffold's entropy share keeps growing through distillation; a closed-form forecast from one channel correlation tracks selection retention over a 54-point range to five points, unfitted. On compression, the content convention beats raw surprisal in every cell; an answer-leakage audit then corrects our own headline control: re-fed chains earn a quarter to a half of their accuracy from restated answers, and once stripped, no token scorer beats a random contiguous block.

Figures & tables

Figure 1: The conflation, exactly and at model scale. (a) A two-position witness (closed form, Theorem 1 ): at a thought boundary

t

the gate is a coin flip and the connective choice is open while the continuation is decided (

H^{\mathcal{C}}=0.08

nats); mid-computation at

u

the scaffolding is quiet and the content forks (

H^{\mathcal{C}}=1.34

). Raw entropy ranks

t

above

u

, selecting the position whose content uncertainty is

16\times

smaller. (b) Seventeen measured cells, fixed definition (Table 1 lists fourteen): as the scaffold side takes more of mean

H

, the raw top-

20\%

set becomes correspondingly scaffold-driven.

base models, teacher-forced on GSM8K reference solutions (Pythia-70M, -160M, and -1B in App. G )
model (data, mode)	$\bar{s}$	$\bar{H}$	gate%	scaf%	cont%	flip%	ret@20%	scaf-drv%	real- $\mathcal{S}$ %
GPT-2	$0.16$	$2.89$	$7.8{\scriptstyle\pm 0.0}$	$6.3{\scriptstyle\pm 0.0}$	$86.0{\scriptstyle\pm 0.0}$	$13.1{\scriptstyle\pm 0.2}$	$69.1{\scriptstyle\pm 0.2}$	$0.0{\scriptstyle\pm 0.0}$	$13.2{\scriptstyle\pm 0.1}$
Pythia-410M	$0.18$	$2.25$	$9.8{\scriptstyle\pm 0.0}$	$8.1{\scriptstyle\pm 0.0}$	$82.0{\scriptstyle\pm 0.0}$	$12.5{\scriptstyle\pm 0.1}$	$70.1{\scriptstyle\pm 0.1}$	$0.4{\scriptstyle\pm 0.0}$	$13.4{\scriptstyle\pm 0.2}$
Pythia-1.4B	$0.19$	$1.86$	$10.9{\scriptstyle\pm 0.0}$	$9.7{\scriptstyle\pm 0.0}$	$79.4{\scriptstyle\pm 0.0}$	$12.9{\scriptstyle\pm 0.1}$	$65.6{\scriptstyle\pm 0.2}$	$3.0{\scriptstyle\pm 0.0}$	$15.9{\scriptstyle\pm 0.2}$
one tokenizer, one teacher-forced context set: corpus, scale, tuning, and distillation vary
Qwen2.5-1.5B	$0.19$	$0.49$	$18.8{\scriptstyle\pm 0.1}$	$7.7{\scriptstyle\pm 0.0}$	$73.5{\scriptstyle\pm 0.1}$	$16.1{\scriptstyle\pm 0.2}$	$52.1{\scriptstyle\pm 0.2}$	$20.7{\scriptstyle\pm 0.1}$	$18.6{\scriptstyle\pm 0.2}$

Table 1: Channel decomposition of CoT entropy on frozen models (mean

\pm

sem over

3

seeds, fixed scaffold; pos / data sensitivity in App. G ). The middle block holds the tokenizer and the context set fixed, so each indented row differs from Qwen2.5-1.5B by the single intervention named and the last two share the Math-1.5B parent. Shares are of mean

H

; flip is the fork-verdict flip rate on random position pairs; ret@

20

is top-

20\%

retention between conventions; scaf-drv is the scaffold-driven share of the raw top-

20\%

set; real-

\mathcal{S}

is its realized-scaffold share, the Wang et al. (2026b) connective observation quantified per channel.

Figure 2: Four checkpoint contrasts on one tokenizer, and only two move the channels. Teacher-forced GSM8K, three seeds, identical context set and token budget, mean

\pm

sem; each bar is the released checkpoint differing from Qwen2.5-1.5B by the factor named, uncontrolled in whatever else that release changed. Code data,

2\times

parameters, and instruction tuning leave coupling (panels 2 and 3) inside noise and move retention (panel 4) by at most

2.1

points against

6.5

for the math corpus; distillation on the math corpus moves only allocation (panel 1).

		R1-Distill-1.5B, GSM8K			R1-Distill, MATH	Qwen2.5-1.5B-It
	arm	keep $.25$	keep $.50$	keep $.75$	keep $.50$	keep $.50$
scorers	k raw surprisal	$12.2{\scriptstyle\pm 0.3}$	$20.2{\scriptstyle\pm 0.4}$	$46.2{\scriptstyle\pm 0.6}$	$15.9{\scriptstyle\pm 0.7}$	$25.8{\scriptstyle\pm 1.4}$
	e raw entropy	$12.1{\scriptstyle\pm 0.3}$	$19.9{\scriptstyle\pm 0.8}$	$46.1{\scriptstyle\pm 0.9}$	$15.4{\scriptstyle\pm 0.3}$	$25.7{\scriptstyle\pm 1.2}$
	a attention received	$14.5{\scriptstyle\pm 1.0}$	$25.3{\scriptstyle\pm 0.3}$	$44.5{\scriptstyle\pm 1.3}$	$19.1{\scriptstyle\pm 1.0}$	$11.5{\scriptstyle\pm 0.5}$
	k C raw score, content-only	$12.9{\scriptstyle\pm 0.7}$	$25.5{\scriptstyle\pm 0.7}$	$74.4{\scriptstyle\pm 0.7}$	$21.9{\scriptstyle\pm 1.4}$	$16.0{\scriptstyle\pm 0.2}$
	k g gate-adj score, all	$12.4{\scriptstyle\pm 0.7}$	$25.1{\scriptstyle\pm 1.5}$	$70.1{\scriptstyle\pm 1.2}$	$22.0{\scriptstyle\pm 1.3}$	$15.7{\scriptstyle\pm 0.6}$

Table 2: Post-hoc CoT compression: answer accuracy (%) at matched token budgets , mean

\pm

sem over three seeds (

512

problems per seed on GSM8K,

256

on MATH-500); every arm keeps the same number of tokens in every cell. k is raw surprisal, the convention beneath LLMLingua-style importance scores ( Pan et al., 2024 ) (TokenSkip itself deploys a learned LLMLingua-2 classifier ( Xia et al., 2025 ) ); k C and k g are the factorial arms (raw score with content-only eligibility; gate-adjusted score with all tokens eligible); a is the receiver-attention scorer ( Bogdan et al., 2025 ) , d the content-channel scorer, last the final tokens of the chain. The lower panel reruns every arm after deleting each restatement of the gold answer and the chain’s final sentence; raw and content entropy track their surprisal twins within a point and are omitted there.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The three simulations. (a) SYN-1: sweeping the boundary-slot gate mass

\bar{s}

, the fork-verdict flip rate, the top-

20\%

retention between conventions, and the scaffold-driven share of the raw top-

20\%

set (mean

\pm

sd, 3 seeds,

5\times 10^{4}

positions each). (b) SYN-2, one fixed position population transformed per arm: in the phantom arm the transformation moves only the nuisance group, so raw

H

falls

46\%

while the content channel is exactly invariant by Proposition 3 ; in the masked arm a genuine content collapse coexists with rising raw

H

. (c) SYN-3, end states of the in-silico GRPO

2\times 2

: fork-content entropy separates exploring from collapsed policies without overlap, and raw

H

does not, because the format-reward weight moves raw

H

by up to

2.5

nats at unchanged exploration.

position	$s$	$H$	$H_{b}(s)$	$sH^{\mathcal{S}}$	$(1{-}s)H^{\mathcal{C}}$	$H^{\mathcal{C}}$	$G$
$t$ (boundary)	$0.50$	$\mathbf{1.540}$	$0.693$	$0.805$	$0.042$	$0.083$	$1.456$
$u$ (computation)	$0.02$	$1.447$	$0.098$	$0.032$	$1.316$	$\mathbf{1.343}$	$0.103$

Appendix

Table 3: Exact witness for Theorem 1 (i) (the Figure 1 pair; all values in nats, computed in closed form by exact calculations). Raw entropy selects

t

; the content channel says

u

forks

16\times

harder. Boldface marks each convention’s verdict.

Figure 4: Forecast against measurement, twenty-three cells, three held out. Each point is one (model, dataset, mode, scaffold) cell; the forecast uses only

r=\operatorname{corr}(H,H^{\mathcal{C}})

through the Gaussian specialization of Theorem 1 (iii); dashed line is

y=x

; diamonds are the cells first measured after registration. (a) Retention carries no fitted parameter. (b) Flip carries the one frozen constant

c=0.5988

Figure 5: What stripping the restated answers does to the compression arms (GSM8K, keep

0.50

512

problems

\times

three seeds; solid bars as generated, hatched bars after every gold-answer restatement and the final sentence are deleted; dotted line is the stripped full-chain ceiling). (a) On the distilled model the ceiling halves and the recency arm collapses onto the random contiguous block: its tail premium was largely copyable answer text. (b) On the instruct model a real tail premium survives stripping. Numbers, budgets, and clustered intervals in Table 2 and § 5.4 .

CoT Evaluation

Chain-of-Thought (CoT) evaluation measures the accuracy, validity, faithfulness, and usefulness of language models’ visible reasoning traces.

See all 21 papers

Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models
Jul 30, 2026Sara Candussio, Daniel Scalena, Luca Bortolussi +3CoT Compression Large Reasoning Models
University of Trieste · University of Milano-Bicocca · CLCG, University of Groningen +2
Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning
Jun 1, 2026Ting Xu, Xu He, Yupu Lu +4CoT Reasoning Test-Time Scaling
The Chinese University of Hong Kong · Huawei Technologies Ltd · School of Computing and Data Science, The University of Hong Kong
Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction
Sep 17, 2026Theodore O. CochranNumerical Reasoning in Language Models LLM Reliability
AI for Altruism

Jul 30, 2026J/K move · Enter open · S save

Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models

Sara Candussio, Daniel Scalena, Luca Bortolussi, Elisabetta Fersini+2

University of Trieste · University of Milano-Bicocca · CLCG, University of Groningen+2

CoT Compression Large Reasoning Models

Open paper AskPDF arXiv

Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss. We test the robustness of low- and high-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting. Moving from sentences to tokens, we then show that retaining low-entropy tokens seems effective only on mathematical benchmarks. We find this is due to the inherently low-entropy nature of numeric tokens, which also convey semantic content in such problems. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near-perfect full-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain.

Abstract

CoT Evaluation

Explore similar work

Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models

Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning

Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction