ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning

Xuan Xiong; Huan Liu; Li Gu; Zhixiang Chi; Yue Qiu; Yuanhao Yu; Yang Wang

cs.AIApr 7, 2026

ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning

Authors: Xuan Xiong, Huan Liu, Li Gu, Zhixiang Chi, Yue Qiu, Yuanhao Yu, Yang Wang

Organizations: University of Toronto · McMaster University · Concordia University · University of Ottawa

Abstract

Chain-of-thought (CoT) reasoning improves large language model performance on complex tasks, but often produces excessively long and inefficient reasoning traces. Existing methods shorten CoTs using length penalties or global entropy reduction, implicitly assuming that low uncertainty is desirable throughout reasoning. We show instead that reasoning efficiency is governed by the trajectory of uncertainty. CoTs with dominant downward entropy trends are substantially shorter. Motivated by this insight, we propose Entropy Trend Reward (ETR), a trajectory-aware objective that encourages progressive uncertainty reduction while allowing limited local exploration. We integrate ETR into Group Relative Policy Optimization (GRPO) and evaluate it across multiple reasoning models and challenging benchmarks. ETR consistently achieves a superior accuracy-efficiency tradeoff, improving DeepSeek-R1-Distill-7B by 9.9% in accuracy while reducing CoT length by 67% across four benchmarks. Code is available at https://github.com/Xuan1030/ETR

Figures & tables

Figure 1: Accuracy versus average chain-of-thought (CoT) length across model sizes. Accuracy is averaged over four representative benchmarks. Compared methods exhibit different accuracy–efficiency trade-offs, with ETR achieving strong accuracy at substantially shorter reasoning lengths.

Panel of Figure 2 — Figure 2: Step-wise entropy dynamics in generated CoTs on the MATH500 dataset Hendrycks et al. (2024) . (Left) A sample response illustrating high step-wise entropy during self-reflection, marked by uncertainty cues (e.g., but, wait, etc.). (Right) Across models, a higher Spearman rank correlation ( $\rho$ ) between reasoning step index and step-wise entropy indicates persistent entropy growth during reasoning. This is consistently associated with longer outputs and greater token redundancy.

Figure 3: Overview of Entropy Trend Reward (ETR) in RL training. ETR computes step-wise entropy from generated rollouts, aggregates entropy changes with momentum to capture global entropy descent, and provides a reward signal, combined with correctness supervision, to guide policy updates toward more convergent reasoning.

Figure 4: Evolution of the cumulative momentum weights

\alpha_{t}

under different momentum coefficients

\gamma

Method	AMC23			AIME24			MATH500			GPQA-D			Overall
Method	Acc $\uparrow$	Len $\downarrow$	CR $\downarrow$	Acc $\uparrow$	Len $\downarrow$	CR $\downarrow$	Acc $\uparrow$	Len $\downarrow$	CR $\downarrow$	Acc $\uparrow$	Len $\downarrow$	CR $\downarrow$	Acc $\uparrow$	Len $\downarrow$	AES $\uparrow$
DeepSeek-R1-Distill-7B
Original	80.0	6.6k	100%	43.3	11.8k	100%	85.0	4.2k	100%	24.2	11.3k	100%	58.1	8.5k	0.00
DEER	85.0	4.9k	74.2%	50.0	9.9k	83.9%	85.8	2.3k	54.8%	22.7	7.6k	67.3%	60.9	6.2k	0.51
NoThink	77.5	3.8k	57.6%	56.7	8.3k	70.3%	81.4	1.7k	40.5%	22.2	2.0k	17.7%	59.5	4.0k	0.65
LCPO	87.5	3.5k	53.0%	50.0	6.8k	57.8%	85.4	2.2k	52.6%	11.61	2.8k	24.6%	58.6	3.8k	0.60

Table 1: Evaluation results on four benchmarks using greedy decoding (pass@1). Acc denotes accuracy, Len denotes token length, and CR denotes compression rate.

\uparrow

and

\downarrow

indicate higher- and lower-is-better, respectively. AES is a unified score that jointly accounts for accuracy and efficiency.

Reward	AMC23		AIME24		MATH500		GPQA-D		AES $\uparrow$
Reward	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	AES $\uparrow$
Original	80.0	6.6k	43.3	11.8k	85.0	4.2k	24.2	11.3k	0.00
Min. $H$	80.0	2.1k	43.3	5.1k	88.2	1.3k	38.3	2.1k	1.06
Max. $H$	10.0	15.1k	0.0	16.4k	9.0	15.3k	1.5	16.0k	-5.4
No $\gamma$	87.5	4.9k	46.67	10.0k	87.8	3.6k	31.8	10.0k	0.61
No $R_{\mathrm{corr}}$	65.0	1.2k	23.3	1.4k	78.6	0.7k	29.8	0.7k	0.11

Table 2: Ablation Study on Entropy-based Reward Designs. We compare our momentum-based reward against extreme constraints and simplified variants across four benchmarks. Our method demonstrates a superior balance between accuracy and efficiency.

Figure 5: Spearman’s rank correlation coefficient between reasoning steps and step entropy. The shift toward negative values indicates a more convergent reasoning trajectory after ETR training.

Figure 6: ETR reduces CoT length primarily by maintaining low per-step verbosity, rather than by aggressively suppressing reasoning steps or reflective processes, in contrast to entropy minimization.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	LCPO	O1-Pruner	PEAR	ETR (Ours)
LoRA Rank	-	-	-	32
LoRA Alpha	-	-	-	64
Actor learning rate	$1e-6$	$1e-6$	$1e-6$	$1e-5$
train_batch_size	64	32	32	32
mini_batch_size	64	16	16	16
micro_batch_size	-	2	2	2

Appendix

Table 3: Training configs for various methods.

Hyperparameter	ETR (Ours)
LoRA Rank	32
LoRA Alpha	64
LoRA Target Modules	all-linear
Actor learning rate	$1e-5$
train_batch_size	32
mini_batch_size	16

Appendix

Table 4: Detaied Configurations for ETR

$\gamma$	AMC23		AIME24		MATH500		GPQA-D		AES $\uparrow$
$\gamma$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	AES $\uparrow$
0.5	85.0	2180.1	40.0	6381.6	90.4	1441.6	36.4	2684.5	1.04
0.8	87.5	2288.1	26.7	5622.7	89.0	1441.5	33.8	5622.7	0.66
0.9	87.5	2411.6	56.7	4639.3	90.6	1479.2	37.4	2452.4	1.53
0.95	82.5	3160.9	56.7	5289.3	89.2	1445.7	35.4	3052.2	1.29
0.99	80.0	2371.0	46.7	5070.0	88.4	1489.8	38.9	2621.4	1.12

Appendix

Table 5: Analysis on the impact of the momentum

\gamma

in Equation 7 .

	AMC23		AIME24		MATH500		GPQA-D		Overall (Avg)
	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$
Base	80.00%	6597.63	43.33%	11751.53	85.00%	4210.51	24.20%	11336.66	58.13%	8474.08
Rank 16	85.00%	3071.23	46.67%	6970.80	90.00%	1961.92	38.38%	4718.47	65.01%	4180.61
Rank 32	87.50%	2411.63	56.67%	4639.30	90.60%	1479.16	37.37%	2452.43	68.04%	2745.63
Rank 64	87.50%	2130.05	53.33%	4256.60	90.80%	1381.37	39.39%	2174.15	67.76%	2485.54
Full FT	87.50%	2863.15	53.33%	5208.20	91.20%	1739.81	39.90%	2267.26	67.98%	3019.61

Appendix

Table 6: Analysis on the impact of LoRA Rank

r

on model performance.

DeepSeek-R1-Distill-Qwen-1.5B
	amc23		aime24		MATH500		GPQA-D		Overall
	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$
Untrained	47.50%	9414.55	13.33%	13671.93	68.60%	6433.81	6.06%	14073.11	33.87%	10898.35
ETR (FT)	62.50%	4481.43	13.33%	10435.63	79.00%	2220.44	18.69%	7113.74	43.38%	6062.81
DeepSeek-R1-Distill-14B
Untrained	87.50%	5926.85	33.33%	12623.07	87.40%	4000.64	41.92%	8878.32	62.54%	7857.22

Appendix

Table 7: Model scale analysis on DeepSeek-R1-Distill-14B and DeepSeek-R1-Distill-Qwen-1.5B, showing consistent improvements in accuracy and reasoning efficiency using ETR, with substantial reductions in reasoning length across model sizes.

Figure 7: Entropy trajectories during reasoning. ETR-trained models demonstrate efficient convergence and necessary self-reflection, whereas baseline models exhibit frequent entropy spikes and redundant reasoning steps.

	AMC23		AIME24		MATH500		GPQA-D		Overall (Avg)
	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$	Acc $\uparrow$	Len $\downarrow$
Base	80.00%	6597.63	43.33%	11751.53	85.00%	4210.51	24.20%	11336.66	58.13%	8474.08
Token_Seg	72.50%	1210.25	36.67%	3098.17	82.20%	916.04	33.84%	919.84	56.31%	1536.08
ETR	87.50%	2411.63	56.67%	4639.30	90.60%	1479.16	37.37%	2452.43	68.04%	2745.63

Appendix

Table 8: Performance comparison focusing on token-level segmentation vs. ETR.

Model	Method	Accuracy $\uparrow$	Avg Length $\downarrow$
Qwen3-4B	Base	41.46%	3397.20
Qwen3-4B	ETR	53.66%	1838.60
DeepSeek-R1-Distill-7B	Base	75.00%	3975.97
DeepSeek-R1-Distill-7B	ETR	75.00%	1057.99

Appendix

Table 9: Model Performance on HumanEval Coding Benchmark

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 370 papers

ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning
Aug 28, 2026Xin Jiang, Minhao Wang, Wen Wu +4Reward Shaping RL for Language Model Reasoning
School of Computer Science and Technology, East China Normal University, Shanghai 200241, China · State Key Laboratory of Estuarine and Coastal Research, School of Computer Science and Technology, East China Normal University, Shanghai 200241, China
Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning
Jun 1, 2026Ting Xu, Xu He, Yupu Lu +4CoT Reasoning Test-Time Scaling
The Chinese University of Hong Kong · Huawei Technologies Ltd · School of Computing and Data Science, The University of Hong Kong
</think> Doesn't Stop Reasoning: Analysis of Spurious CoT Termination
Sep 3, 2026Seunghee Koh, Sungjae Choi, Minchan Kwon +2CoT Reasoning Efficient Language Model Reasoning
Korea Advanced Institute of Science and Technology, South Korea

Aug 28, 2026J/K move · Enter open · S save

ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning

Xin Jiang, Minhao Wang, Wen Wu, Zhentao Xie+3

School of Computer Science and Technology, East China Normal University, Shanghai 200241, China · State Key Laboratory of Estuarine and Coastal Research, School of Computer Science and Technology, East China Normal University, Shanghai 200241, China

Reward Shaping RL for Language Model Reasoning Efficient Language Model Reasoning Reinforcement Learning with Verifiable Rewards Large Reasoning Models

Open paper AskPDF arXiv

Large reasoning models achieve strong performance on complex tasks by generating extended chain-of-thought (CoT) traces via reinforcement learning with verifiable rewards (RLVR). While current RLVR methods have achieved strong results with correctness-based reward signals, they provide limited guidance on the quality of the reasoning process itself, leaving the internal reasoning structure largely unoptimized. Through empirical analysis across multiple model families, we identify a consistent pattern: correct reasoning trac es exhibit more frequent and larger token-level entropy drops within the thinking phase than incorrect ones. We propose ERR+, a two-phase RLVR framework grounded in this observation. The first phase trains with the Entropy Relief Reward (ERR), a bonus proportional to cumulative token-level entropy drops in the thinking phase, log-normalized by response length. Unlike prior methods that suppress entropy, ERR rewards the resolution of uncertainty while leaving exploratory high-entropy states unconstrained. The second phase introduces the Robust Relative Efficiency Reward, which scores each response's length against co-generated peers via a

\tanh

-transformed within-group

z

-score. We provide a formal analysis showing that joint optimization of the two objectives induces gradient conflict in early training, motivating the sequential design . Experiments on five datasets demonstrate consistent improvements in both accuracy and response conciseness across model backbones. Our code is available at https://github.com/XrkArul/err_response

Abstract

Efficient Language Model Reasoning

Explore similar work

ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning

Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning

</think> Doesn't Stop Reasoning: Analysis of Spurious CoT Termination