LIFT: Scoped Constraints for Evaluating and Training Instruction Following

Eran Goldstein; Yuval Belfer; Yonatan Belinkov; Daniel Gissin

cs.CLOct 20, 2026

LIFT: Scoped Constraints for Evaluating and Training Instruction Following

Authors: Eran Goldstein, Yuval Belfer, Yonatan Belinkov, Daniel Gissin

Organizations: AI21 Labs

Abstract

Common instruction following (IF) benchmarks, such as IFEval, evaluate the ability of language models (LMs) to follow simple constraints that can easily be verified programmatically. These same constraints, in turn, are used as reward signals for reinforcement learning. This design may lead to reward hacking of constraints, pushing models away from the main task. Recent attempts to address this resulted in benchmarks with richer constraints, but required expert labor, remained closed-access, and cannot be easily extended. We present Lift-Bench, an IF benchmark that asks a model to draft a document given a topic and a structured outline. Constraints are scoped to certain document sections, providing finer control over constraint density. Lift-Bench includes both verifiable and judged constraints, as well as a training counterpart, Lift-Train. Importantly, Lift-Bench provides both main task and instruction-following constraints, mitigating shallow reward hacking. On experiments with multiple models, we find that frontier models struggle with the harder subsets of our benchmark and analyze model performance along multiple axes. We also find that training on Lift-Train is much less susceptible to reward hacking than training on the common IFTrain. Lift-Bench and Lift-Train are publicly available, as well as the data generating pipeline.

Figures & tables

Benchmark	Size	Authoring	Style	Content	Max	Scope	Train set	Pipeline
	Data		Constraints				Released
IFEval	541	manual	✓	✗	3	global	✗	✗
IFBench	300	manual	✓	✗	2	global	✓	✗
ComplexBench	1,150	manual	✓	✓	5	global	✗	✗
AdvancedIF	1,645	manual	✓	✓	10	global	(✓)	(✓)
MOSAIC	4,000	synthetic	✓	✗	20	global	✗	✓

Table 1: Comparison with related IF benchmarks. (✓): exists but not publicly released.

	GPT			Claude		Open Models
split	5.5	5.6 Sol	5 nano	Fable 5	Opus 4.8	DS V4 Pro	GLM-5.3	Kimi K3	Qwen3.8-A95B
short/few	0.97	0.95	0.71	0.92	0.94	0.95	0.87	0.97	0.89
short/some	0.96	0.94	0.62	0.91	0.92	0.95	0.88	0.96	0.88
short/many	0.98	0.94	0.58	0.82	0.89	0.93	0.83	0.97	0.82
short/numerous	0.93	0.92	0.47	0.81	0.82	0.92	0.80	0.92	0.81
long/few	0.98	0.96	0.67	0.92	0.96	0.95	0.93	0.96	0.90

Table 2: Mean score per split; best per row in bold; averages are

\pm

95% CI; ‘avg (all)’ weighs the structured and flat averages equally.

Figure 3: Benchmark scores broken by number of judged and verifiable constraints. In the lightest cell (bottom left) GPT-5.5 and GPT-5 nano both score near 1, but at the densest loads the weaker model falls to 0.2–0.5 while the stronger stays above 0.7.

Model	short	long	mean
	% score change by document size
Kimi K3	$-0.9$ %	$+0.2$ %	$-0.4$ %
Claude Opus 4.8	$-0.4$ %	$-0.5$ %	$-0.4$ %
GPT-5.6 Sol	$-0.4$ %	$-0.9$ %	$-0.7$ %
GPT-5.5	$-1.3$ %	$-1.7$ %	$-1.5$ %
GPT-5 nano	$-7.0$ %	$-4.1$ %	$-5.5$ %

Table 3: Change in section score, estimated via linear regression and averaged over density types. At fixed per-section density, frontier models are only mildly affected by constraints elsewhere in the document while smaller models continue to degrade. The full results are in Table 11 and the fitting procedure in Appendix I .

Figure 4: Benchmark scores per constraint type. Models display distinct constraint-type fingerprints, even where their averages are close.

Benchmark	base	IFTrain	LIFT-Train
IFEval	0.712	0.764 ( $\mathbf{+5.2}$ )	0.723 ( $+1.1$ )
IFBench	0.270	0.430 ( $\mathbf{+16.0}$ )	0.278 ( $+0.8$ )
LIFT-Bench (10-split avg)	0.423	0.451 ( $+2.8$ )	0.822 ( $\mathbf{+39.9}$ )
AdvancedIF (3-split avg)	0.392	0.374 ( $-1.8$ )	0.401 ( $+0.9$ )
ComplexBench (EN)	0.324	0.316 ( $-0.8$ )	0.327 ( $+0.3$ )

Table 4: Benchmark scores after GRPO training on Qwen2.5-7B (step 180), with the change in points against the base model in parentheses. Shaded cells are in-distribution; other cells are out-of-distribution. Each recipe gains most on the benchmark its reward can see, but only LIFT-Train also improves the external judged benchmarks.

Figure 5: Comparison of training mixtures on Qwen3-32B: LIFT-Bench score at GRPO step 180, averaged over the flat splits, the structured splits, and their macro-average (mean of the flat and structured averages).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Type	Candidate source
Judged	content	LLM-based
	tables / list / code_snippets	LLM-based; often empty if not relevant to the topic
Verifiable	language	sampled over ten fixed languages
	case_type	sampled over {upper, lower, title}
	emoji_count	sampled over {in-range, more-than, less-than}
	lexical_count	keyword count / must-appear / must-not-appear

Appendix

Table 5: Constraint types and the source of each block’s candidates. Keywords for lexical_count constraints are LLM-generated to make them relevant to the discussed topic and not arbitrary. We used ifbench_constraint for ablations, but not in the final splits. Shared hyper-parameters are in Table 7 .

		Document		Scope	Constraints / section		Examples
Split	Verbalization	branching	depth	sections	verifiable	judged	test	train
short / few	structured	3	2	3	1–2	1–2	85	3,681
short / some	structured	3	2	5	1–3	1–2	85	3,681
short / many	structured	3	2	7	1–3	1–3	85	3,681
short / numerous	structured	3	2	7	1–4	1–4	85	3,681
long / few	structured	5	2	5	1–2	1–2	66	2,978

Appendix

Table 6: The ten released splits and the pipeline parameters that distinguish them. sections is

M

, the number of leaf sections one sampling pass may constrain; the two constraints / section columns give

N

per family, so a section receives between

1

and

N

constraints of that family. Every other pipeline parameter is identical across the ten splits, and the corresponding configuration keys are listed in Table 7 . Example counts are after failed items are dropped (see Yield ).

Stage	Parameter	Value
Seeding	seed corpus	Wikipedia Vital Articles, level 5
	max_examples	4000
	random_seed	1337
All LLM stages	model	Qwen2.5-32B-Instruct
	temperature	0.6
	max_tokens	2048

Appendix

Table 7: Pipeline hyper-parameters shared by all ten released splits; the four parameters that vary across splits are listed at the bottom, with their values in Table 6 .

Type	Mechanism	Example
doc_structure	Verifiable	Section headers must match the outline.
language	Verifiable	Write the text in Spanish.
lexical_count	Verifiable	Use the word “cards” 3–6 times.
emoji_count	Verifiable	Include at least 1 emoji.
case_type	Verifiable	Write the text in uppercase.
content	LM judge	Discuss the historical significance of Dogs Playing Poker in American pop culture.

Appendix

Table 8: Constraint types used in the drafting pipeline and their scoring mechanisms, with examples.

Table 9: Judge meta-evaluation against human labels (

n=305

;

227

valid,

78

invalid). The original baseline judge is marked

\dagger

. Every scoped judge beats every unscoped one, regardless of model.

Panel of Figure 6 — Figure 6: Descriptive statistics for the released splits.

Table 10: Mean score per split for all 17 evaluated models, shown as two panels: the ten closed-weight models in ( 10(a) ) and the seven open-weight models in ( 10(b) ). Bold marks the best value across all 17 models, not the best within a panel. Averages carry a

\pm

95% CI across examples; ‘avg (all)’ weighs the structured and flat averages equally.

Figure 7: Fewer constraints, worse scores: the globally-scoped (flat) regime is harder than the scoped one and ranks models differently. Mean score (

\pm

95% CI) at a matched constraint budget (

\leq

10 total constraints), structured vs. flat

Figure 8: Constraint-type fingerprint, structured splits pooled.

Model	short	long	mean
	% score change by document size
Kimi K3	$-0.9$ %	$+0.2$ %	$-0.4$ %
Claude Opus 4.8	$-0.4$ %	$-0.5$ %	$-0.4$ %
DeepSeek V4 Pro	$-0.8$ %	$-0.5$ %	$-0.7$ %
GPT-5.6 Sol	$-0.4$ %	$-0.9$ %	$-0.7$ %
Qwen3.8-A95B	$-1.1$ %	$-1.2$ %	$-1.1$ %

Appendix

Table 11: Section-load slopes. Columns as in Table 3 .

Panel of Figure 9 — Figure 9: Rollout health during GRPO training on Qwen2.5-7B: the lexical-only reward is the one that gets hacked. Held-out validation pass every $10$ steps ( $200$ completions per point).

Instruction Following

Instruction following studies how AI systems interpret and reliably carry out user goals, constraints, formats, and priorities.

See all 134 papers

Jul 30, 2026

IFHierBench: Hierarchical Instruction Following for Large Language Models

Instruction-following ability is critical for deploying large language models in real-world applications, where downstream components depend on the output satisfying specific constraints. Modern deployments increasingly handle the full task in a single LLM call, with one prompt specifying a layered output whose overall artifact, structural sections, and nested fields must each satisfy concrete constraints. Existing instruction-following benchmarks treat the constraint set as a flat list applied uniformly to the response, so they cannot scope a check to a particular section of the output. We introduce IFHierBench, a hierarchical instruction-following benchmark of 600 prompts stratified across four constraint-tree depths and 35 distinct constraints, each prompt paired with a deterministic checker that verifies satisfaction at every scope. Evaluating seven leading proprietary and open-weight models, we find that even the strongest model only marginally exceeds 50% prompt-level accuracy and that accuracy degrades sharply as constraint depth grows. Reliably following nested constraints remains a substantial gap for current LLMs, motivating future training methods that consider constraint adherence at finer granularity to achieve better instruction-following ability.

Yuetian Mao, Chunyang Chen

Technical University of Munich

LLM Evaluation Language Model Generation Evaluation Instruction Following

Ask PDF

Mar 5, 2026

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

Instruction-following is a foundational capability of large language models (LLMs), with its improvement hinging on scalable and accurate feedback from judge models. However, the reliability of current judge models in instruction-following remains underexplored due to several deficiencies of existing meta-evaluation benchmarks, such as their insufficient data coverage and oversimplified pairwise evaluation paradigms that misalign with model optimization scenarios. To this end, we propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following that covers diverse instruction and constraint types. For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses based on instruction-following quality. This design enables a listwise evaluation paradigm that assesses the capabilities of judge models to rank multiple responses, which is essential in guiding model alignment. Extensive experiments on IF-RewardBench reveal significant deficiencies in current judge models and demonstrate that our benchmark achieves a stronger positive correlation with downstream task performance compared to existing benchmarks. Our codes and data are available at https://github.com/thu-coai/IF-RewardBench.

Bosi Wen, Yilin Niu, Cunxiang Wang +5

The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI · University of Electronic Science and Technology of China

LLM-as-a-Judge Instruction Following Reward Model Evaluation Learning to Rank

Ask PDF

Apr 19, 2026

TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles

Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints, ranging from verifiable ones (e.g., output length) to unverifiable ones (e.g., tone). Reinforcement learning with verifiable rewards has emerged as a paradigm for IF tasks, leveraging LLM-as-a-judge to assess unverifiable constraints. However, we empirically find that this approach remains a significant bottleneck, suffering from severe reward hacking and higher computational overhead. In this work, we first analyze the generalization capabilities of unverifiable constraints and discover that specific constraints exhibit distinct, high-generalization patterns. Motivated by this, we propose TinyJudge, a framework that employs an ensemble of specialized tiny language models (

\sim0.6B

) to provide rewards for soft constraints. By distilling expertise from frontier models into these tiny models, it achieves high-precision, lightweight evaluation. Extensive evaluations across five benchmarks demonstrate that TinyJudge outperforms the baselines by

\sim10\%

in average performance and

12\%

in reward precision. Crucially, it also achieves a

3\times

speedup in total training time. Our work provides a scalable and robust path for aligning LLMs with unverifiable human instructions.

Yirong Zeng, Yufei Liu, Xiao Ding +9

Harbin Institute of Technology SCIR Lab · Peking University · Huawei Technologies Co., Ltd

Reward Modeling LLM Alignment LLM-as-a-Judge Instruction Following Reinforcement Learning with Verifiable Rewards

Ask PDF

Abstract

Instruction Following

Explore similar work

IFHierBench: Hierarchical Instruction Following for Large Language Models

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles