Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

Zichao Li; Jie Lou; Fangchen Dong; Zhiyuan Fan; Mengjie Ren; Hongyu Lin; Xianpei Han; Debing Zhang

cs.LGMar 11, 2026

Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

Authors: Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang,

Organizations: Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.

Abstract

Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR $^3$ ), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR $^3$ ~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines.

Figures & tables

Figure 1: Comparison of GR 3 with open-source efficient reasoning models, all trained on DeepSeek-R1-Distill-7B. GR 3 pioneers a new paradigm that sustains stable performance gains under RL while simultaneously mitigating the length inflation issue.

Figure 2: Training dynamics of GR 3 , which retains GRPO’s reward gains without loss while significantly reducing average tokens. The base models used for the two settings are DeepSeek-R1-Distill-1.5B and Qwen3-8B (without thinking mode), respectively.

Method	$\boldsymbol{S}$ (Length shaping term)
Additive: $\hat{R}^{(+)}=R+\lambda\cdot S$
L1-Exact	$-\lvert\ell^{(i)}-\ell_{T}\rvert$
DAPO	$\begin{cases}0,&\ell^{(i)}\leq\ell_{T}-\ell_{C},\\ \frac{\ell_{T}-\ell_{C}-\ell^{(i)}}{\ell_{C}},&\ell_{T}-\ell_{C}<\ell^{(i)}\leq\ell_{T},\\ -1,&\ell^{(i)}>\ell_{T}\end{cases}$
Kimi-k1.5	$\begin{cases}0.5-\frac{\ell^{(i)}-\ell_{\min}}{\ell_{\max}-\ell_{\min}},&R=1,\\ \min\left(0.5-\frac{\ell^{(i)}-\ell_{\min}}{\ell_{\max}-\ell_{\min}},\,0\right),&R=0\end{cases}$
Truncation	$-\mathbb{I}(R=1)\cdot\mathbb{I}(\ell^{(i)}>\ell_{T})$
Efficiently	$-\mathbb{I}(R=1)\cdot s\left((\ell^{(i)}-\bar{\ell})/\sigma_{\ell}\right)$

Table 1: Comparison of length-aware reward shaping methods ( Liu et al., 2025c ; Arora and Zanette, 2025 ; Aggarwal and Welleck, 2025 ; Team et al., 2025 ; Yu et al., 2025 ; Cheng et al., 2025 ) .

\ell^{(i)}

is the response length of sample

i

\ell_{\min}

\ell_{\max}

\bar{\ell}

, and

\sigma_{\ell}

are computed within each group.

\alpha

\lambda

\ell_{T}

and

\ell_{C}

are fixed hyperparameters.

s(\cdot)

denotes the sigmoid function.

Figure 3: Additive length regularization consistently degrades task reward across different choices of

\lambda

, whereas the multiplicative scheme maintains stable reward improvement throughout training.

Model	AIME24		MATH500
	Score $\uparrow$	#Token $\downarrow$	Score $\uparrow$	#Token $\downarrow$
Standard
GRPO	41.0	12885	87.2	5788
Threshold-based
Truncation–4k	32.5	3419	86.5	1438
Truncation–8k	34.8	6684	87.6	2918

Table 2: Comparison of standard RLVR training, threshold-based truncation ( Hou et al., 2025 ) , and group-relative methods ( Arora and Zanette, 2025 ) . All methods are evaluated at the 800th step.

	AIME24		AIME25		AMC23		MATH500
Model	Avg@32 $\uparrow$	#Token $\downarrow$	Avg@32 $\uparrow$	#Token $\downarrow$	Avg@16 $\uparrow$	#Token $\downarrow$	Avg@4 $\uparrow$	#Token $\downarrow$
Initial model
DeepSeek–R1–Distill–7B	52.4	13213	39.4	14032	89.8	6385	92.1	3994
Length-oriented RL
LCR1–7B	47.9	7548	36.3	7960	85.8	2963	89.1	1546
Laser–DE–L4096–7B	51.7	4931	36.7	5048	88.1	2427	92.4	1580

Table 3: Mathematical reasoning performance for 7B models. Length-oriented methods reduce tokens but may sacrifice accuracy, while GR 3 consistently achieves comparable accuracy with significantly fewer tokens, establishing a markedly better Pareto frontier.

Figure 4: Sensitivity of $\alpha$ : reward gap relative to the standard GRPO baseline versus $\alpha$ (log scale). Marker color indicates the average CSR measured during actual training, while the triangle marker denotes the value of $\alpha$ selected during the calibration phase.

	LiveCodeBench v6		MultiPL-E
Model	Score $\uparrow$	#Token $\downarrow$	Score $\uparrow$	#Token $\downarrow$
DeepSeek–R1–Distill–1.5B
Initial	17.7	12665	45.1	6181
GRPO	23.4	11830	51.5	6589
GR 3 (ours)	24.9	8538	52.2	2414
DeepSeek–R1–Distill–7B

Table 4: Performance on code generation tasks. GR 3 achieves competitive scores with fewer tokens across model sizes.

	Arena–Hard–Auto		Alpaca–Eval
Model	Score $\uparrow$	#Token $\downarrow$	Score $\uparrow$	#Token $\downarrow$
Qwen3–4B
Initial	66.6	1139	40.1	737
GRPO	85.8	2374	33.9	1993
GR 3 (ours)	85.9	1377	44.1	859
Qwen3–8B

Table 5: Performance on chat tasks. GRPO improves performance but incurs substantial length bias, while GR 3 achieves stronger alignment gains while preserving the length of the initial policy .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Comparison of multiplicative and gated additive shaping during RLHF training. Multiplicative shaping matches standard GRPO in task reward while achieving controlled length reduction. In contrast, gated additive variants with different reward thresholds underperform and produce overly short responses, reflecting optimization instability introduced by hard gating.

Prompt

90^{\circ}

rotation around the origin in the counter-clockwise direction is applied to

7+2i.

What is the resulting complex number? Let’s think step by step and output the final answer within \boxed{} .

<think>

7cos90 -2i sin90=0-2i

// Extremely short reasoning performed

</think>

\boxed{-2i}

Appendix

Table 6: Example output from the Efficiently ( Arora and Zanette, 2025 ) 1.5B baseline at step 800, showing excessive reasoning truncation and omission of critical intermediate steps, resulting in an incoherent chain of thought.

Panel of Figure 6 — Figure 6: Effect of reward density on the strict advantage-preservation objective. As reward density increases, the strict constraint satisfaction rate decreases, even for a small penalty coefficient $\alpha=0.05$ , and more noticeably for a larger penalty $\alpha=5.0$ .

Hyperparameter	Math	Code	Chat
GR 3 Coefficient $\alpha$	0.33	0.33	0.00133
KL Coefficient $\beta$	0.0	0.0	0.001
Batch Size	128	128	128
Mini Batch Size	128	128	128
Prompt Length	1024	1024	1024
Rollout Length	16384	16384	4096

Appendix

Table 7: Hyper-parameters used in our experiments.

	AIME24		AIME25		AMC23		MATH500
Model	Avg@32 $\uparrow$	#Token $\downarrow$	Avg@32 $\uparrow$	#Token $\downarrow$	Avg@16 $\uparrow$	#Token $\downarrow$	Avg@4 $\uparrow$	#Token $\downarrow$
Initial model
DeepSeek–R1–Distill–1.5B	30.0	16531	23.6	15799	70.8	9351	83.9	5399
Length-oriented RL
LCR1–1.5B	23.5	9071	20.6	8275	67.8	4170	81.9	2520
Laser–DE–L4096–1.5B	30.1	5770	24.1	5008	73.4	3110	84.6	1931

Appendix

Table 8: Mathematical reasoning performance for 1.5B models.

	AIME24		AIME25		AMC23		MATH500
Model	Avg@32 $\uparrow$	#Token $\downarrow$	Avg@32 $\uparrow$	#Token $\downarrow$	Avg@16 $\uparrow$	#Token $\downarrow$	Avg@4 $\uparrow$	#Token $\downarrow$
Initial model
DeepSeek–R1–Distill–1.5B	30.0	16531	23.6	15799	70.8	9351	83.9	5399
Our method: varying $\alpha$
GR 3 ( $\alpha=1.00$ )	34.9	6316	27.4	5927	75.2	1754	83.0	848
GR 3 ( $\alpha=0.50$ )	40.5	7923	29.6	7399	82.3	3245	87.1	1705

Appendix

Table 9: Hyperparameter sweep of the length penalty coefficient

\alpha

in GR 3 on 1.5B models.

Prompt

A figure skater is facing north when she begins to spin to her right. She spins 2250 degrees. Which direction (north, south, east or west) is she facing when she finishes her spin? Let’s think step by step and output the final answer within \boxed{} .

Example Output from our GR 3 -trained 1.5B Model

<think>

Okay, so I need to figure out which direction a figure skater is facing after spinning 2250 degrees to her right. She starts facing north. Hmm, let’s break this down step by step.

// Restates the problem and sets up a step-by-step plan. First, I know that a full rotation is 360 degrees, so every time she spins 360 degrees, she’ll end up facing the same direction she started. So, to find out where she ends up after 2250 degrees, I should probably find the equivalent angle within a single rotation.

// Recalls the key concept: directions repeat every 360°. Let me calculate how many full rotations 2250 degrees is. I’ll divide 2250 by 360. Let’s see, 2250 ÷ 360. Let me compute that: 360 × 6 is 2160, and 2250 - 2160 is 90. So, 2250 degrees is equivalent to 6 full rotations plus an additional 90 degrees.

Appendix

Table 10: Example output from our GR 3 -trained 1.5B model, demonstrating robust, clearly structured step-by-step reasoning and a correct final answer in boxed format.

Prompt

Example Output from the GRPO-trained 1.5B Model

<think>

Okay, so I have this problem here about a figure skater spinning. Let me try to figure it out step by step. Hmm, she’s facing north and starts spinning to her right. She spins 2250 degrees, and we need to find out which direction she’s facing after that. The options are north, south, east, or west.

Alright, first, I know that when you spin, your direction changes based on how many degrees you turn. But since she’s starting facing north and spinning to her right, I need to figure out how that affects her direction after a certain number of degrees.

Wait, hold on. When you spin to your right, does that mean you’re turning clockwise or counterclockwise? I think in the context of facing north, spinning to your right would be clockwise because if you’re facing north and turn right, you’re going towards east, then south, then west, and back to north. So, right is clockwise.

Appendix

Table 11: Example output from the GRPO-trained 1.5B baseline, exhibiting redundant reasoning loops, repeated self-doubt, and failure to produce a conclusive final answer.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM
Feb 16, 2026Chang Liu, Yiran Zhao, Lawrence Liu +3RL for Language Models RL for Language Model Reasoning
University of California, Los Angeles · Salesforce AI Research · National University of Singapore +1
Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
May 19, 2025Kangwen Zhao, Jianfeng Cai, Jinhua Zhu +5Reward Modeling LLM Alignment
University of Science and Technology of China
Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learning
Sep 28, 2026Zijun Weng, Zhongan Bi, Xuanang Gao +5Open-Ended Generation RL for Language Models
Fudan University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Zhejiang University +1

Feb 16, 2026J/K move · Enter open · S save

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

Chang Liu, Yiran Zhao, Lawrence Liu, Yaoqi Ye+2

University of California, Los Angeles · Salesforce AI Research · National University of Singapore+1

RL for Language Models RL for Language Model Reasoning Efficient Language Model Inference

Open paper AskPDF arXiv

Reinforcement learning (RL) has enhanced the capabilities of large language models (LLMs) through reward-driven training. Nevertheless, this process can introduce excessively long responses, inflating inference latency and computational overhead. Prior length-control approaches typically rely on fixed heuristic reward shaping, which can misalign with the task objective and require brittle tuning. In this work, we propose LACONIC, a reinforcement learning method that enforces a target token budget during training. Specifically, we update policy models using an augmented objective that combines the task reward with a length-based cost. To balance brevity and task performance, the cost scale is adaptively adjusted throughout training. This yields robust length control while preserving task reward. We provide a theoretical guarantee that support the method. Across mathematical reasoning models and datasets, LACONIC preserves or improves pass@1 while reducing output length by over 50%. It maintains out-of-domain performance on general knowledge and multilingual benchmarks with 44% fewer tokens. Moreover, LACONIC integrates into standard RL-tuning with no inference changes and minimal deployment overhead.

Abstract

RL for Language Models

Explore similar work

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

Bias Fitting to Mitigate Length Bias of Reward Model in RLHF

Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learning