When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction

Feiyang Ren; Shengtao Wen; Yu Tian; Yuanning Cui; Xiaofeng Cao; Jiangchao Yao; Xu Chen; Xiang Chen

cs.AIAug 16, 2026

When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction

Authors: Feiyang Ren, Shengtao Wen, Yu Tian, Yuanning Cui, Xiaofeng Cao, Jiangchao Yao, Xu Chen, Xiang Chen

Organizations: MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · Tsinghua University · Nanjing University of Information Science and Technology

Abstract

Efficient LLM serving is often bottlenecked by the need to pad sequences to a fixed maximum length, and this wastes compute and degrades throughput. Output length prediction enables length-aware scheduling and reduces overhead. Existing methods, including entropy-guided token pooling, rely on token-level entropy but overlook semantic differences between tokens, causing them to underweight important tokens and overemphasize less informative tokens. We introduce ESTP(Entropy-and-Semantic Token Pooling), a lightweight framework that combines entropy representations with attention-based semantic representations. By deriving both representations directly during prefill, ESTP captures uncertainty and semantic importance while incurring negligible additional inference latency and memory overhead compared with methods using external auxiliary models. On the ForeLen benchmark, ESTP outperforms baseline methods, achieves better prediction accuracy and lower prediction errors across most scenarios. When integrated with a length-aware scheduler in end-to-end system tests, it further improves overall throughput and reduces the padding ratio.

Figures & tables

Panel of Figure 1 — Figure 1: Limitations of Entropy-Only Token Importance Assessment. (a) Comparison of Token-Level Weighting Strategies: Entropy-Only, Semantics-Only, and ESTP. (b) Quantitative Analysis of Token Misalignment Between Entropy and Semantic Importance.

Figure 2: The joint distribution of Attentions Scores and Gradient Importance, with the regression line confirming a positive correlation (r=0.62).

Scenario	P@30	R@30	P@50	R@50
RL	69%	43%	65%	73%
Reasoning	66%	49%	63%	74%
Average	68%	46%	64%	74%

Table 1: Attention-Semantic Alignment.

Figure 3: Overview of the ESTP framework. The framework consists of three main components: (1) Construction of Hidden States , which extracts hidden state representations from diverse scenarios including reasoning, long-sequence, and RL tasks; (2) Semantic and Entropy Dual-Channel Pooling , which computes attention-based semantic weights and aggregates entropy values to identify high-entropy frequent tokens (e.g., logical connectives, suppositional words, inference steps), yielding combined representations $z_{n}$ ; and (3) Lightweight Prediction , which applies softmax-normalized weights to hidden states and feeds the weighted representation into an MLP predictor to estimate the output length with a distributional CRPS loss and a continuous SmoothL1 regression loss.

Model	Scenario	Prediction Method
Model	Scenario	SSJF-Reg	SSJF-MC	LTR-C	TRAIL	EGTP	ESTP
Qwen2.5-3B	LongSeq	246.76	318.71	164.64	140.31	97.50	89.47
	Reasoning	322.77	242.26	287.77	138.01	143.11	124.9
	RL	120.37	206.42	96.65	153.72	99.78	73.38
	Avg	229.97	255.80	183.02	144.01	113.46	95.92
Qwen2.5-7B	LongSeq	206.11	346.96	169.28	124.05	88.29	85.13

Table 2: Comparison of MAE across different length prediction methods ( lower is better ). Best results are highlighted in bold , and second-best results are underlined . ESTP consistently achieves lower average prediction error across all backbone LLMs.

Output Length	Prediction Method
Output Length	SSJF-Reg	SSJF-MC	LTR-C	TRAIL	EGTP	ESTP
Short	50.37	56.70	27.56	55.82	22.03	57.30
Medium	80.79	70.87	62.15	66.42	81.31	85.44
Long	50.75	48.21	33.20	50.48	51.00	52.74
Overall	67.67	61.60	60.46	61.15	62.36	72.54

Table 3: Accuracy across different output length bins in RL scenario. Best results are highlighted in bold , and second-best results are underlined . ESTP achieves the highest overall accuracy on the Qwen2.5-3B while maintaining competitive performance across Short (<200 tokens), Medium (200–500 tokens), and Long (>500 tokens).

Figure 4: Performance comparison of RMSE across different length prediction methods.

Method	MAE	RMSE
ESTP	929.13 $\pm$ 119.35	1028.57 $\pm$ 131.12
EGTP	1012.08 $\pm$ 124.73	1183.30 $\pm$ 167.75

Table 4: Performance on extreme long-CoT regime.

Metrics	Prediction Method
Metrics	SSJF-Reg	SSJF-MC	LTR-C	TRAIL	EGTP	ESTP
Padding Ratio $\downarrow$	1.34	1.32	0.59	0.91	0.52	0.46
Throughput $\uparrow$	13.21	12.48	20.22	13.30	24.70	32.75
Avg. JCT $\downarrow$	20.41	21.50	17.62	20.84	14.49	10.61

Table 5: End-to-End System Performance Comparison on Llama3.2-3B. Best results are highlighted in bold , and second-best results are underlined . ESTP achieves the best performance among all methods.

Method	LongSeq	Reasoning	RL	Average
ESTP(Ours)	89.47	124.9	73.38	95.92
w/o Pos. Debiasing	95.32	138.68	78.77	104.26
w/o Smooth L1	102.91	138.09	80.67	107.22
w/o CRPS	106.05	133.07	76.45	105.19
Average Pooling	163.32	133.74	92.44	129.83
Entropy Pooling	97.50	143.11	99.78	113.46

Table 6: Ablation Study on the Qwen2.5-3B.

Figure 5: Average time(ms) and memory(MB) overhead per sample for predicting in LongSeq scenario.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Splitting strategy	QWQ-LongCoT-AIMO( $>$ 10k)
Train	269
Validation	30
Test	36

Appendix

Table 7: Dataset of QWQ-LongCoT-AIMO(

>

10k).

Value	Reasoning	RL	LongSeq	Avg
0.3	109.03	61.63	94.74	88.47
0.4	108.53	59.58	94.01	87.37
0.5	108.45	58.66	93.35	86.82
0.6	109.55	58.83	93.75	87.38
0.7	109.59	59.33	93.40	87.44

Appendix

Table 8: Hyperparameter Sensitivity Analysis.

Splitting strategy	Qwen2.5_3B			Qwen2.5_7B
Splitting strategy	Reasoning	RL	LongSeq	Reasoning	RL	LongSeq
Train	324	50253	660	324	52578	660
Validation	104	16751	220	106	17527	177
Test	113	16255	177	108	17213	220

Appendix

Table 9: Dataset splitting strategy on the Qwen2.5.

Splitting strategy	llama3.2_1B			llama3.2_3B
Splitting strategy	Reasoning	RL	LongSeq	Reasoning	RL	LongSeq
Train	299	54496	660	295	52998	660
Validation	90	18166	220	97	17515	220
Test	152	17825	42	149	17819	32

Appendix

Table 10: Dataset splitting strategy on the Llama3.2.

Reference and metric	Raw mean	Raw median	Debiased mean	Debiased median
Causal baseline Pearson	0.5401	0.5222	0.0851	0.1306
Causal baseline Spearman	0.5462	0.5692	0.0861	0.1103
Normalized position Pearson	-0.2633	-0.2449	-0.0892	-0.1152
Normalized position Spearman	-0.5462	-0.5692	-0.0861	-0.1103

Appendix

Table 11: Correlation of token scores with position based references.

Model	Input Length	Prediction Method
Model	Input Length	SSJF-Reg	SSJF-MC	LTR-C	TRAIL	EGTP	ESTP
Llama3.2-1B	Short	72.53	67.21	73.59	68.20	28.61	76.41
	Medium	60.10	63.12	46.81	52.63	79.50	66.11
	Long	55.08	58.41	14.80	60.54	55.67	62.16
	Overall	66.13	65.98	66.64	59.68	46.54	72.09
Llama3.2-3B	Short	60.31	61.16	77.54	51.22	24.10	62.91

Appendix

Table 12: Accuracy comparison across different output length bins in RL scenarios( higher is better ).

Model	Scenario	Prediction Method
Model	Scenario	SSJF-Reg	SSJF-MC	LTR-C	EGTP	ESTP
Qwen2.5 3B	LongSeq	302.82	351.14	210.61	109.42	103.07
	Reasoning	363.15	291	329.58	170.38	145.72
	RL	109.33	247.79	130.94	116.6	104.52
	Avg	258.43	296.64	223.71	132.13	117.77
Llama3.2 1B	LongSeq	448.9	215.84	410.37	104.41	103.02

Appendix

Table 13: Comparison of RMSE Metrics Across Different Length Prediction Methods.

Figure 6: Qualitative comparison demonstrates the token weighting distributions of EGTP and ESTP on specific samples. Red tokens indicate high-weight tokens, and the corresponding prediction results are provided for reference.

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
Sep 1, 2026Weihuang Wen, Yingying Liu, Yichuan Liu +5LLM Serving LLM Inference Scheduling
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · Huawei Technologies Co., Ltd.
Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
Apr 1, 2026Haoyu Zheng, Yongqiang Zhang, Fangcheng Fu +7LLM Inference Scheduling LLM Uncertainty Estimation
School of Computer Science, Wuhan University, Wuhan, China · Dameng Database Co., Ltd., Wuhan, China · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China +2
Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions
Apr 9, 2026Jing Wang, Yu-Yang Qian, Ke Xue +3LLM Serving Robust Estimation
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China

Sep 1, 2026J/K move · Enter open · S save

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

Weihuang Wen, Yingying Liu, Yichuan Liu, Wenqi Zeng+4

The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · Huawei Technologies Co., Ltd.

LLM Serving LLM Inference Scheduling

Open paper AskPDF arXiv

The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced frameworks (e.g., EAGLE-3) encode signals that are predictive of generation length. Building on this insight, we introduce OUTLETS (Output-Length Prediction from Speculative Decoding Backbones), which repurposes the speculative backbone as a trajectory-aware length predictor. When its draft representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head and achieves lower MAE than the evaluated methods. Under saturated disaggregated serving, OUTLETS predictions enable standard scheduling policies to prioritize shorter requests and distribute requests more evenly across decoding instances, reducing short-request P99 latency by 34.8%.

Abstract

LLM Serving

Explore similar work

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions

Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions