LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models

Wei Zhang; Lintong Du; Yuanhe Zhang; Zhenhong Zhou; Kun Wang; Li Sun; Sen Su

cs.CLFeb 25, 2026

LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models

Authors: Wei Zhang, Lintong Du, Yuanhe Zhang, Zhenhong Zhou, Kun Wang, Li Sun, Sen Su

Organizations: Beijing University of Posts and Telecommunications, Beijing, China · Nanyang Technological University, Singapore, Singapore

Abstract

Despite the strong performance of Large Language Models (LLMs) on complex instruction-following tasks, precise control of output length remains a persistent challenge. Existing methods primarily attempt to enforce length constraints by externally imposing length signals or optimization objectives, while largely overlooking the underlying limitation: the model's intrinsic deficit in length cognition. To address this, we propose LARFT (Length-Aware Reinforcement Fine-Tuning), a training framework that aligns the model's length cognition with its action. Specifically, LARFT integrates length-oriented reinforcement learning with a hindsight length awareness. By transforming on-policy data into hindsight self-awareness tasks where the model learns to identify the actual length of its own generation, LARFT jointly optimizes the model's internal representation of length information and refines its policy to satisfy length constraints, thereby achieving precise and reliable length instruction following. Extensive experiments across four base models demonstrate that LARFT outperforms existing baselines, achieving an average improvement of +20.92 points across three length instruction following benchmarks with only a marginal decline of -1.45 points on four general capability benchmarks.

Figures & tables

Figure 1: Overview of the Length-Aware Reinforcement Fine-Tuning (LARFT) framework. (I) Illustrates the Cognition-Action Gap, where standard LLMs fail to align their output length with user instructions. (II) Presents our LARFT method, which unifies Length-Oriented RL and Hindsight Length Awareness to align generation with constraints. (III) Demonstrates that LARFT achieves precise length control while maintaining robust general capabilities.

Method	In-Distribution Tasks (Length Following)						Out-of-Distribution Tasks (General)
	LIFEBench		LongBench		Lenctrl-Bench		MMLU $\uparrow$	GSM8K $\uparrow$	IFEval $\uparrow$	GPQA $\uparrow$
	LD $\downarrow$	LS $\uparrow$	S l $\uparrow$	S q $\uparrow$	MAE $\downarrow$	ROUGE-L $\uparrow$	MMLU $\uparrow$	GSM8K $\uparrow$	IFEval $\uparrow$	GPQA $\uparrow$
Qwen2.5-3B-Instruct
Origin	38.79	46.04	76.38	68.49	28.76	18.08	65.07	76.19	73.02	31.03
SFT	65.60	26.93	79.88	66.04	45.44	16.87	65.00	76.04	66.79	29.24
RL	30.71	54.10	87.23	69.49	13.92	20.86	65.00	74.83	74.10	30.13

Table 1: Main results on length-following tasks and general benchmarks. Bold values indicate the best performance.

Method	$\lambda_{t}$			w/o Sc
Method	0.005	0.01 (Default)	0.05	w/o Sc
LIFEBench LS ( $\uparrow$ )	75.67	80.57	79.29	73.44
LongBench $S_{l}$ ( $\uparrow$ )	94.58	96.75	96.12	94.22
Lenctrl-Bench MAE ( $\downarrow$ )	8.45	7.00	7.36	8.80

Table 2: Ablation study on optimization hyperparameters. We report the performance across three length-following benchmarks. “Sc” indicates Cosine Annealing Scheduling.

Figure 4: Length probing correlations across layers. The plots compare the ability to decode response length from the hidden states of the first token (Left) and the last token (Right) .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Cosine annealing schedule for $\lambda_{t}$ .

Panel of Figure 6 — Figure 6: Data construction pipeline with sample counts at each stage.

Length Category	Word Range	Count	Percentage
Short	10–100	2,156	24.7%
Medium	101–500	3,421	39.2%
Long	501–1,000	1,892	21.7%
Very Long	1,001–4,000	1,263	14.5%
Total	10–4,000	8,732	100.0%

Appendix

Table 3: Distribution of dataset samples by length category. The dataset covers a broad range of lengths, with the majority of samples falling into the Medium (101–500 words) category.

Benchmark	Task Type	Few-shot	Metric
MMLU ( Hendrycks et al., 2021 )	Knowledge	5-shot	Accuracy
GSM8K ( Cobbe et al., 2021 )	Math Reasoning	5-shot	Exact Match
IFEval ( Zhou et al., 2023 )	Instruction Following	0-shot	Prompt Strict Acc
GPQA ( Rein et al., 2024 )	Graduate-level QA	5-shot	Accuracy

Appendix

Table 4: General capability benchmark configurations.

Parameter	Length Benchmarks	General Benchmarks
Backend	vLLM	vLLM
Temperature	0.6	0.0 (greedy)
Top-p	0.9	1.0
Max New Tokens	8192	Task-dependent
Tensor Parallel Size	4	4

Appendix

Table 5: Inference configuration for all evaluations.

Model Family	Model	Params	Context	Maximum Output Length
Qwen2.5	Qwen2.5-3B-Instruct	3B	32K	8192
Qwen2.5	Qwen2.5-7B-Instruct	7B	32K	8192
Llama3	Llama-3.2-3B-Instruct	3B	128K	128K
	Llama-3.1-8B-Instruct	8B	128K	128K

Appendix

Table 6: Model specifications used in experiments.

Table 12

Hyperparameter	Value	Hyperparameter	Value
Data Configuration		Rollout Configuration
Rollout Batch Size	128	Rollouts per Prompt ( $G$ )	4
Max Prompt Length	2048	Temperature	0.7
Max Response Length	8000	Top-p	0.8
Truncation Side	Left	GPU Mem Utilization	0.8
Policy Optimization		Regularization & Schedule

Appendix

Table 9: Hyperparameters for GRPO-based reinforcement learning.

Instruction Following

Instruction following studies how AI systems interpret and reliably carry out user goals, constraints, formats, and priorities.

See all 135 papers

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM
Feb 16, 2026Chang Liu, Yiran Zhao, Lawrence Liu +3RL for Language Models RL for Language Model Reasoning
University of California, Los Angeles · Salesforce AI Research · National University of Singapore +1
Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
Apr 29, 2026Zhen Zhang, Changyi Yang, Zijie Xia +11LLM Inference Controllable Text Generation
University of California, Santa Barbara · Carnegie Mellon University · LMSYS Org +2
ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modeling
Sep 26, 2026Bosi Wen, Yilin Niu, Xiaoying Ning +3Instruction Following Instruction Tuning
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI

Feb 16, 2026J/K move · Enter open · S save

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

Chang Liu, Yiran Zhao, Lawrence Liu, Yaoqi Ye+2

University of California, Los Angeles · Salesforce AI Research · National University of Singapore+1

RL for Language Models RL for Language Model Reasoning Efficient Language Model Inference

Open paper AskPDF arXiv

Reinforcement learning (RL) has enhanced the capabilities of large language models (LLMs) through reward-driven training. Nevertheless, this process can introduce excessively long responses, inflating inference latency and computational overhead. Prior length-control approaches typically rely on fixed heuristic reward shaping, which can misalign with the task objective and require brittle tuning. In this work, we propose LACONIC, a reinforcement learning method that enforces a target token budget during training. Specifically, we update policy models using an augmented objective that combines the task reward with a length-based cost. To balance brevity and task performance, the cost scale is adaptively adjusted throughout training. This yields robust length control while preserving task reward. We provide a theoretical guarantee that support the method. Across mathematical reasoning models and datasets, LACONIC preserves or improves pass@1 while reducing output length by over 50%. It maintains out-of-domain performance on general knowledge and multilingual benchmarks with 44% fewer tokens. Moreover, LACONIC integrates into standard RL-tuning with no inference changes and minimal deployment overhead.

Abstract

Instruction Following

Explore similar work

LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling

ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modeling