LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

Chang Liu; Yiran Zhao; Lawrence Liu; Yaoqi Ye; Csaba Szepesvári; Lin F. Yang

cs.LGFeb 16, 2026

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

Authors: Chang Liu, Yiran Zhao, Lawrence Liu, Yaoqi Ye, Csaba Szepesvári, Lin F. Yang

Organizations: University of California, Los Angeles · Salesforce AI Research · National University of Singapore · University of Alberta

Abstract

Reinforcement learning (RL) has enhanced the capabilities of large language models (LLMs) through reward-driven training. Nevertheless, this process can introduce excessively long responses, inflating inference latency and computational overhead. Prior length-control approaches typically rely on fixed heuristic reward shaping, which can misalign with the task objective and require brittle tuning. In this work, we propose LACONIC, a reinforcement learning method that enforces a target token budget during training. Specifically, we update policy models using an augmented objective that combines the task reward with a length-based cost. To balance brevity and task performance, the cost scale is adaptively adjusted throughout training. This yields robust length control while preserving task reward. We provide a theoretical guarantee that support the method. Across mathematical reasoning models and datasets, LACONIC preserves or improves pass@1 while reducing output length by over 50%. It maintains out-of-domain performance on general knowledge and multilingual benchmarks with 44% fewer tokens. Moreover, LACONIC integrates into standard RL-tuning with no inference changes and minimal deployment overhead.

Figures & tables

Figure 1: The top panel skethces RL-tuning with a fixed length-aware shaping objective (blue). As the heuristically shaped objective generally differs from the true task reward

R

(red), optimizing it may converge to a policy

\pi_{N}

that is suboptimal in

R

. The bottom panel sketches training with LACONIC . LACONIC adaptively updates the length-aware objective (green) so that it better aligns with the true task reward while achieving shorter outputs, yielding near-optimal policies.

Figure 2: Illustration of LACONIC. LACONIC alternates two steps: (1) in a primal update, the policy model is updated on an augmented objective that trades off task reward $r$ with a length-aware cost $c$ scaled by the dual variable $\lambda$ ; (2) in a dual update, $\lambda$ is adaptively updated to enforce a token budget constraint $B$ by increasing when the average length $\bar{L}$ exceeds the budget $B$ and decreasing otherwise. Together, these updates maximize task reward while meeting the budget on average.

Panel of Table 1 — Table 1: Evaluation results across four math benchmarks.

Panel of Table 2 — Table 2: Evaluation results across out-of-domain (OOD) benchmarks.

Panel of Figure 3 — Figure 3: Ablation of the cost functions on DeepScaleR-1.5B with token budget $B=1500$ . We plot (a) accuracy reward and (b) average response length over training steps. For the linear cost, the Langrangian reward used in primal updates is computed by $r(q,o)-\lambda\,\widetilde{c}(q,o)$ , where $\widetilde{c}(q,o)=(L(o)-B)/B$ . All other experiment setups and hyperparameters are identical.

Panel of Figure 4 — Figure 4: Ablation of the token budget $B$ on DeepScaleR-1.5B. We plot (a) accuracy reward; (b) average response length; and (c) dual variable $\lambda$ over training steps with budgets $B\in\{1000,1500,1750,2000\}$ . All other setups and hyperparameters are identical.

Token Budget	AIME2024		MATH		Minerva		Olympiad		Macro Average
Token Budget	Pass@1	# Tokens	Pass@1	# Tokens	Pass@1	# Tokens	Pass@1	# Tokens	Pass@1	# Tokens
+ LACONIC (2000)	34.79	3976	85.61	1556	31.76	1869	48.96	2448	50.28	2462
+ LACONIC (1750)	31.21	3803	84.82	1516	31.52	2059	48.88	2261	49.11	2410
+ LACONIC (1500)	29.58	3458	84.35	1353	31.48	1582	47.93	2084	48.33	2119
+ LACONIC (1000)	28.12	2665	83.75	1049	29.24	1189	47.07	1669	47.05	1643

Table 3: Evaluation results of the token-budget

B

ablation on DeepScaleR-1.5B across four math benchmarks.

Figure 5: Average computational resource usage of LACONIC (green) and GRPO (blue).

Panel of Figure 6 — Figure 6: Ablation of the dual step size $\eta$ on DeepScaleR-1.5B with $B=1500$ . We plot (a) accuracy reward; (b) average response length; and (c) dual variable $\lambda$ over training steps with step size $\eta\in\{0.001,0.002,0.01\}$ . We set $\Lambda=0.3$ for all experiments except for the run with $\eta=0.01$ and $\Lambda=0.15$ . All other setups and hyperparameters are identical.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 4 — Table 4: Evaluation results of the cost function ablation on DeepScaleR-1.5B across four math benchmarks.

Panel of Table 5 — Table 5: Evaluation results of the cost functions on Qwen-Math-1.5B across four math benchmarks.

Panel of Figure 7 — Figure 7: Ablation of the cost functions on Qwen-Math-1.5B with a token budget $B=550$ : (a) reward accuracy, (b) average response length, (c) dual variable $\lambda$ ,

Step size $\eta$	AIME2024		MATH		Minerva		Olympiad		Macro Average
Step size $\eta$	Pass@1	# Tokens	Pass@1	# Tokens	Pass@1	# Tokens	Pass@1	# Tokens	Pass@1	# Tokens
$\eta=0.001,\Lambda=0.3$	27.50	3231	84.25	1356	30.92	1543	47.87	2090	47.64	2055
$\eta=0.002,\Lambda=0.3$	29.58	3458	84.35	1353	31.48	1582	47.93	2084	48.33	2119
$\eta=0.01,\Lambda=0.3$	29.79	3429	84.30	1365	31.42	1602	47.19	2097	48.18	2123
$\eta=0.01,\Lambda=0.15$	29.79	3477	84.20	1319	31.15	1562	47.83	2067	48.24	2106

Appendix

Table 6: Evaluation results of the token-budget

B

ablation across four math benchmarks.

Panel of Table 7 — Table 7: Evaluation results of length-control on Qwen3-8B across four math benchmarks.

Panel of Table 8 — Table 8: Evaluation results of fixed $\lambda$ ablation across four math benchmarks.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models
Feb 25, 2026Wei Zhang, Lintong Du, Yuanhe Zhang +4Instruction Following LLM Fine-Tuning
Beijing University of Posts and Telecommunications, Beijing, China · Nanyang Technological University, Singapore, Singapore
Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
Mar 11, 2026Zichao Li, Jie Lou, Fangchen Dong +8RL for Language Models RL Post-Training
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.
QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization
Jul 23, 2026Siwei Chen, Siqi Chen, Xupeng Miao +1Overthinking in Language Models RL for Language Model Reasoning
School of Computer Science, Peking University · Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems · Department of Electronic Engineering, Tsinghua University +1

Feb 25, 2026J/K move · Enter open · S save

LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models

Wei Zhang, Lintong Du, Yuanhe Zhang, Zhenhong Zhou+3

Beijing University of Posts and Telecommunications, Beijing, China · Nanyang Technological University, Singapore, Singapore

Instruction Following LLM Fine-Tuning RL for Language Models

Open paper AskPDF arXiv

Despite the strong performance of Large Language Models (LLMs) on complex instruction-following tasks, precise control of output length remains a persistent challenge. Existing methods primarily attempt to enforce length constraints by externally imposing length signals or optimization objectives, while largely overlooking the underlying limitation: the model's intrinsic deficit in length cognition. To address this, we propose LARFT (Length-Aware Reinforcement Fine-Tuning), a training framework that aligns the model's length cognition with its action. Specifically, LARFT integrates length-oriented reinforcement learning with a hindsight length awareness. By transforming on-policy data into hindsight self-awareness tasks where the model learns to identify the actual length of its own generation, LARFT jointly optimizes the model's internal representation of length information and refines its policy to satisfy length constraints, thereby achieving precise and reliable length instruction following. Extensive experiments across four base models demonstrate that LARFT outperforms existing baselines, achieving an average improvement of +20.92 points across three length instruction following benchmarks with only a marginal decline of -1.45 points on four general capability benchmarks.

Abstract

RL for Language Models

Explore similar work

LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models

Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization