Mitigating the Length-Scaling Tax with Online Distillation

Xu Wan; Wenyue Xu; Shengjie Zhao; Mingyang Sun

cs.LGSep 30, 2026

Mitigating the Length-Scaling Tax with Online Distillation

Authors: Xu Wan, Wenyue Xu, Shengjie Zhao, Mingyang Sun

Organizations: ByteDance Seed · Tongji University · Peking University

Abstract

Length scaling during reinforcement-learning (RL) post-training is often viewed as a sign of improved reasoning ability, especially on difficult problems, but may also make responses to already-solved problems unnecessarily verbose. We quantify this side effect as the length-scaling tax (LST): excess response length on already-solved queries without a commensurate accuracy gain. To mitigate LST, we propose Length Self-Distillation (LSD), which routes solved prompts to on-policy distillation and retains the original RL objective for unsolved prompts. LSD uses an exponential moving average of the online policy as its teacher, requiring no external model. We find that LSD achieves comparable or better performance than RL across multiple variants, while substantially curbing response-length growth on easy queries. LSD reduces LST from 19.0% to -3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks, demonstrating that LSD effectively preserves concise response patterns on easy queries while supporting efficient exploration on difficult queries during RL post-training.

Figures & tables

Figure 1: Global validation accuracy and response length during RLVR. Purple circles show mean accuracy, and orange squares show mean response length on three benchmarks and their aggregate over training.

Figure 2: Easy-query accuracy saturates while response length keeps growing during RLVR. At each anchor $b\in\{0,25,50,75,100\}$ , we select queries with solve rate at least $\tau=0.875$ and freeze the resulting easy set.

Figure 3: Observable behavior on the fixed easyset. Reflection word frequency and repeated-bigram rate in responses generated by subsequent checkpoints for the same queries in $\mathcal{E}_{100}$ .

Setting	Budget	$t^{\star}$	$\operatorname{LST}_{0}(t^{\star})$	$\operatorname{LST}_{25}(t^{\star})$	$\operatorname{LST}_{50}(t^{\star})$	$\operatorname{LST}_{75}(t^{\star})$	$\operatorname{LST}_{100}(t^{\star})$	$\Delta$ Pass@1
All-4k	4096	240	$21.0$	$18.1$	$16.7$	$13.7$	$16.3$	$+9.1$ pp
All-8k	8192	$240^{\ast}$	$25.5\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 4.5}}$	$40.9\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 22.8}}$	$38.0\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 21.3}}$	$36.5\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 22.8}}$	$37.4\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 21.1}}$	$+9.7$ pp
All-4k	4096	400	$42.9$	$44.6$	$43.0$	$40.0$	$43.4$	$+11.8$ pp
All-4k-8k	$4096\!\rightarrow\!8192$	$100^{\dagger}$	$33.7\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 9.2}}$	$42.2\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 2.4}}$	$38.8\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 4.2}}$	$37.2\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 2.8}}$	$39.2\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 4.2}}$	${+12.6}$ pp
All-4k-8k	$4096\!\rightarrow\!8192$	$400$	$31.4\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 11.5}}$	$42.6\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 2.0}}$	$39.3\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 3.7}}$	$34.8\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 5.2}}$	$39.3\,{\scriptscriptstyle{\color[rgb]{0,0.5,0}\downarrow 4.1}}$	$\mathbf{+15.0}$ pp
Hard-4k	4096	400	$\mathbf{53.9}\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 11.0}}$	$\mathbf{58.0}\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 13.4}}$	$\mathbf{54.7}\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 11.7}}$	$\mathbf{49.7}\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 9.7}}$	$\mathbf{54.7}\,{\scriptscriptstyle{\color[rgb]{1,0,0}\uparrow 11.3}}$	$+11.8$ pp

Table 1: Controlled comparison of budget and training-data effects. LST is computed with the All-4k fixed-easy reference. Gray rows are the corresponding All-4k anchor baselines. Each non-anchor LST entry reports the value at

t^{\star}

, with the colored arrow showing its change from the corresponding anchor baseline. Red denotes LST increase and green denotes decrease.

\Delta

Pass@1 is full-test-set Pass@1 improvement over the base model.

Figure 4: Historical accuracy of prompts selected as easy at step 500. Each point is one prompt at one previous checkpoint. Yellow indicates $A(x)=0$ and purple indicates $A(x)=1$ .

Figure 5: Length growth on fixed easy queries. Across four anchor-defined sets, all three LSD variants have lower easy-query length and LST than RL through most of training. The upper panels report mean length and the lower panels report LST. Different anchors select different cohorts.

Panel of Figure 6 — Figure 6: Capability and length-scaling tax across tasks. (a) Single-turn average Pass@1, Pass@32, and $\mathrm{LST}_{1000}$ . (b) BrowseComp-Plus average Pass@1, turns, and $\mathrm{LST}_{50}$ . C and D denote CRISP and Fixed SG-FKL; F, R, and P denote LSD with SG-FKL, SG-RKL, and PG-RKL. RL+LP denotes RL + Length Penalty.

Figure 7: EMA half-life ablation. We compare $H=2,4,8$ at $\tau=1.0$ through step 1750. The upper row shows benchmark and aggregate Pass@1; the lower row shows LST for anchors $b=0,100,500,1000$ .

Figure 8: Routing-threshold ablation. We compare $\tau=1.0,0.85,0.75$ at $H=4$ through step 1750. Panels follow Figure 7 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Task performance (%)
	AMC		AIME25		AIME26		Avg. Acc.
Method	Pass@1	Pass@32	Pass@1	Pass@32	Pass@1	Pass@32	Pass@1	Pass@32
RL	60.09	84.34	22.19	43.33	16.04	36.67	42.90	65.73
CRISP	58.48	86.75	18.36	36.67	13.18	33.33	40.56	65.04
Fixed SG-FKL	45.07	87.95	11.46	40.00	8.96	36.67	30.44	67.13
LSD (SG-FKL)	61.30	86.75	22.92	43.33	18.15	36.67	44.20	67.13

Appendix

Table 2: Evaluation results for single-turn reasoning. Panel (a) reports accuracy in percent; panel (b) reports the mean token counts, and LST. Avg. Acc. weights benchmarks by their numbers of evaluation prompts. Avg. Len. covers all 143 prompts. Easy and Hard Len. use the fixed RL step-100 easy set (

A(x)\geq 0.9

) and its hard-query complement.

\mathrm{LST}_{1000}

uses the

\tau=1

RL step-1000 cohort in Table 8 . On this fixed easy set, RL step 885 has the lowest mean response length among RL checkpoints satisfying the accuracy constraint (100%). Its mean length, 853.716 tokens, is the shared reference for all methods.

Method	BrowseComp-Plus
Method	Pass@1	Avg. Turns	Easy Len.	Hard Len.	$\mathrm{LST}_{0}$	$\mathrm{LST}_{50}$	$\mathrm{LST}_{100}$
RL	29.50	5.41	3531.9	4065.3	42.3	31.4	11.8
RL + Length Penalty	22.58	4.66	2708	3426		1.2
LSD (SG-FKL)	30.72	5.24	2706.0	4252.0	24.9	13.7	1.2
LSD (SG-RKL)	29.60	4.93	2146.5	4034.8	13.6	9.2	$\mathbf{-6.2}$
LSD (PG-RKL)	31.65	5.66	2715.2	4139.5	31.4	16.1	2.4

Appendix

Table 3: Evaluation results on multi-turn agentic tasks. Values correspond to Figure 6b . Pass@1 and LST are in percent; lengths are in tokens.

$H$	$\tau$	Avg. Pass@1 (%)	$\mathrm{LST}_{0}$ (%)	$\mathrm{LST}_{100}$ (%)	$\mathrm{LST}_{500}$ (%)	$\mathrm{LST}_{1000}$ (%)
2	1.00	40.14	27.79	14.64	7.29	15.49
4	1.00	41.85	29.20	7.63	2.01	9.06
8	1.00	40.49	13.01	11.00	-6.21	2.91
4	0.85	40.91	27.55	21.24	10.84	18.42
4	0.75	39.31	19.15	10.97	-0.68	8.11

Appendix

Table 4: Ablation endpoint summary at step 1750. Avg. Pass@1 is weighted over all 143 evaluation prompts. LST is computed against the fixed aligned-RL accuracy-qualified reference. The

H=4,\tau=1.0

run is shared by both comparisons.

$H$	$\tau$	Pass@1 (%)	Easy seq. (%)	Easy tok. (%)	Effective OPD coef.	Param. gap ( $\times 10^{-5}$ )	OPD loss ( $\times 10^{-3}$ )
2	1.00	40.14	18.87	10.79	0.252	0.690	0.614
4	1.00	41.85	18.54	10.31	0.247	1.087	0.693
8	1.00	40.49	15.24	7.90	0.192	1.690	0.841
4	0.85	40.91	24.67	14.87	0.354	1.077	0.715
4	0.75	39.31	29.50	19.00	0.450	1.078	0.689

Appendix

Table 5: LSD training signals in the

H

and

\tau

ablations. Sequence and token shares refer to the easy route. Parameter gap is the logged maximum absolute teacher–student parameter difference before the EMA update. Training statistics average 1750 steps per run.

Figure 9: Training signals behind the EMA half-life ablation. All runs use SG-FKL and

\tau=1

. Faint lines show raw values; marked curves show centered 21-point moving means, using available points at the boundaries. Evaluation points are five training steps apart; the other metrics are logged every training step.

Figure 10: Training signals behind the routing-threshold ablation. All runs use SG-FKL and

H=4

, over steps 1–1750. Raw and smoothed curves follow the convention in Figure 9 . Lower thresholds increase the number of sequences and fraction of tokens routed to distillation, as well as the effective OPD coefficient.

Figure 11: Training dynamics of the three LSD objectives. Actor entropy (left) and the mean absolute teacher–rollout log-probability difference before updates (right) over the shared steps 501–1628. Faint traces show raw per-step values; marked curves show centered 21-step means, using available points at the boundaries. SG-RKL exhibits lower entropy and a smaller teacher–rollout discrepancy over this interval. Each trajectory uses its own run’s training batches.

Variant	Actor entropy	Teacher–rollout gap	Easy sequences (%)
SG-FKL	0.0743	0.00738	21.51
SG-RKL	0.0367	0.00481	20.23
PG-RKL	0.0542	0.00590	20.67

Appendix

Table 6: Optimization diagnostics over a shared training interval. Values are arithmetic means of the logged per-step scalars over all 1128 shared steps, without smoothing. These training steps are not independent experimental replicates.

Figure 12: Training token allocation between the easy and hard routes. Panels (a)–(c) show raw per-step fractions and centered 21-step means. Panel (d) summarizes the distribution of token shares across training steps: boxes show the interquartile range and median, and whiskers show the 5th and 95th percentiles. F, R, and P denote SG-FKL, SG-RKL, and PG-RKL. These are distributions of per-step shares, not individual response lengths or uncertainty intervals.

Variant	Easy seq. (%)	Easy tok. (%)	Hard tok. (%)	Easy tok. median [IQR]
SG-FKL	21.51	12.17	87.83	11.54 [8.09, 15.67]
SG-RKL	20.23	10.58	89.42	9.99 [7.11, 13.27]
PG-RKL	20.67	10.94	89.06	10.29 [7.31, 13.90]

Appendix

Table 7: Training-route allocation over steps 501–1628. The first three numeric columns are means of logged per-step fractions, expressed in percent. The last reports the median and interquartile range of the easy-token share.

Figure 13: Accuracy and length evaluated on the same fixed queries. Each column uses one RL-frozen cohort with

\tau=1

. The top row reports empirical accuracy; the bottom row reports mean response length. The dashed line marks the 100% anchor accuracy. F, R, and P denote SG-FKL, SG-RKL, and PG-RKL, respectively. Every method is evaluated on the identical query IDs within a column.

Anchor	Method	Correct / total	Acc. (%)	Mean tokens	$\Delta A$ (pp)
$b=100$	RL	318/320	99.38	783.1	+0.00
	LSD (SG-FKL)	320/320	100.00	592.0	+0.63
	LSD (SG-RKL)	320/320	100.00	585.3	+0.63
	LSD (PG-RKL)	320/320	100.00	585.1	+0.63
	Fixed SG-FKL	314/320	98.13	568.1	-1.25
$b=500$	RL	695/704	98.72	1013.5	+0.00

Appendix

Table 8: Paired evaluation on RL-frozen easy sets (

\tau=1

\Delta A

is the accuracy difference from RL (pp). For

b=1000

, the shared reference is RL step 885: 853.716 tokens at 100% accuracy, the minimum over 345 RL checkpoints.

Method	Avg. Pass@1 (%)	Easy queries		Hard queries		$\mathrm{LST}_{1000}$ (%)
Method	Avg. Pass@1 (%)	Acc. (%)	Mean tokens	Acc. (%)	Mean tokens	$\mathrm{LST}_{1000}$ (%)
RL	42.90	98.15	996.9	32.85	2329.3	19.03
LSD (SG-FKL)	44.20	99.10	859	34.22	2511	-3.72
LSD (SG-RKL)	42.80	99.43	757.2	32.50	2395.0	-10.92
LSD (PG-RKL)	43.56	99.15	801.6	33.45	2392.5	1.45
Random Routing	22.08	89.74	895.0	9.78	2006.6	-11.00

Appendix

Table 9: Paired accuracy and response length. Easy and hard columns use the RL step-100 cohort (

\tau=0.9

) and its complement; LSD and Random Routing Hard Acc. is derived from overall and easy-set accuracy.

\mathrm{LST}_{1000}

uses the strict step-1000 cohort and shared reference in Table 8 .

Figure 14: Query-level examples from the strict step-1000 cohort. Bar height is mean response length; labels give correct rollouts out of 32. All three queries were answered correctly in all 32 RL anchor rollouts. The examples include unchanged correctness, an LSD regression, and a Fixed regression. F, R, and P denote SG-FKL, SG-RKL, and PG-RKL, respectively.

Parameter	Single-turn reasoning	Multi-turn agentic tasks
Optimizer	AdamW; $\beta=(0.9,0.999)$
Learning rate / weight decay	$10^{-6}$ / $0.01$
Training batch	32 prompts $\times$ 8 rollouts = 256 trajectories
PPO epochs / advantage	1 / GRPO
PPO clip $(\epsilon_{\mathrm{low}},\epsilon_{\mathrm{high}})$	$(0.20,0.28)$
Entropy / RL KL coef.	0 / 0

Appendix

Table 10: Training and inference configurations for RL and LSD. Values spanning both columns are shared. Agent mini-batches count transformed training rows. A dash indicates an unspecified setting.

Parameter	RL	Fixed SG-FKL	SG-FKL	SG-RKL	PG-RKL
Teacher	–	Frozen $\pi_{0}$	EMA
Routing	–	Fixed map	Online rollout-group accuracy
Routing threshold $\tau$	–	1.0
EMA half-life $H$	–	–	4
Distillation objective	–	Top-32 FKL		Top-32 RKL	Sample-token RKL
Distillation gradient	–	Supervised			Policy gradient

Appendix

Table 11: RL and LSD configurations. LSD retains GRPO on the hard route; the gradient row describes only the distillation route. The three online LSD variants share these settings across tasks. Fixed SG-FKL results are reported for single-turn reasoning.

Explore similar work

Jul 23, 2026cs.AI

QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

Recent large reasoning models often develop long chain-of-thought responses during reinforcement learning (RL), resulting in high inference latency and deployment cost. Existing methods for response length control typically rely on explicit length penalties or additional control modules, which require careful tuning and may compromise reasoning quality. We propose Quadrant-weighted Sampling for Length-aware Policy Optimization (QLPO), a simple resampling-based variant of GRPO that introduces implicit length control without modifying the reward function. QLPO first over-generates candidate responses and then resamples the training group by preserving the empirical correct/incorrect ratio while favoring short correct responses and long incorrect responses. This reshapes the training distribution and implicitly encourages shorter model outputs. Across models ranging from 1.5B to 32B parameters, including both base models and strong reasoning models, QLPO consistently improves the accuracy-length trade-off. It reduces response length by 30% to 70% while preserving reasoning performance. These results suggest that structured resampling provides an effective and robust approach to efficient reasoning.

Siwei Chen, Siqi Chen, Xupeng Miao +1

School of Computer Science, Peking University · Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems · Department of Electronic Engineering, Tsinghua University +1

Frictive Policy Optimization Responses

Ask PDF

Jun 10, 2026cs.LG

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with that under privileged context, typically a verified solution. However, we show that the resulting distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce shorter, more direct outputs. We term this pathology \emph{privilege-induced style drift}, which can destabilize training and shorten responses. To address this, we propose \textbf{RLCSD} (Reinforcement Learning with Contrastive on-policy Self-Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing style shifts induced by hints regardless of correctness and yielding a signal more concentrated on task-bearing tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods, with additional results on agentic tasks supporting broader applicability. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to broader cross-model on-policy distillation.

Leyi Pan, Shuchang Tao, Yunpeng Zhai +5

1Tsinghua University · 2Tongyi Lab , Alibaba Group · 3Peking University

Unsupervised On-Policy Self-Distillation Token-Level Supervision

Ask PDF

May 8, 2026cs.AI

Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training

Reinforcement learning with verifiable rewards improves LLM reasoning but often induces overthinking, where models generate unnecessarily long reasoning traces. Existing methods mainly rely on length penalties or early-exit strategies; however, the former may degrade accuracy and induce underthinking, whereas the latter assumes that substantial portions of reasoning traces can be safely truncated. To obtain a compression signal without these limitations, we revisit the training dynamics of existing compression methods. We observe that the length--accuracy correlation is initially negative but continually increases during compression, indicating that shorter responses are initially more likely to be correct but gradually lose this property as the policy moves toward underthinking. Based on this observation, we formalize overthinking: a negative correlation indicates an overthinking regime, while a positive one indicates underthinking. When overthinking, the shortest correct responses are shorter than the group-average response length in expectation, making them natural compression targets already present in on-policy rollouts. We therefore propose \emph{Implicit Compression Regularization} (ICR), an on-policy regularization method whose compression signal comes from a virtual shorter distribution induced by the shortest correct responses in rollout groups, guiding the policy toward concise yet correct trajectories. Training dynamics show that ICR maintains a better length--accuracy correlation during compression, indicating that short responses remain better aligned with correctness instead of drifting toward underthinking. Experiments on three reasoning backbones and multiple mathematical and knowledge-intensive benchmarks show that ICR consistently shortens responses while preserving or improving accuracy, achieving a stronger accuracy--length Pareto frontier.

Chen Wang, Hexuan Deng, Yining Zhang +5

College of Software, Nankai University · 2Zhongguancun Academy · 3Harbin Institute of Technology +4

Short Reasoning Traces Regularization Post-Training

Ask PDF