RL has emerged as a powerful paradigm for enhancing the instruction following capabilities of LLMs. While existing training recipes achieve substantial gains, we find that they suffer from exploration bias towards easy instructions when the training data has multiple instructions in a prompt. This bias is caused by two main reasons: 1) the policy model's initial ability to satisfy hard instructions is too low to trigger successful exploration during RL training, so the optimization is biased towards easy instructions; and 2) canonical RL training recipes typically employ a cumulative reward (the number of instructions fulfilled), treating all instructions equally, which biases the policy model towards fulfilling easy instructions to obtain the same amount of reward. To address these issues, we first propose two metrics to measure the exploration bias in instruction following and then introduce a two-stage framework to alleviate it: 1) Behavioral Bootstrapping, a lightweight rejection sampling fine-tuning stage before RL to activate hard instructions; and 2) Scarcity-Aware Rewards, a new RL reward function that assigns rewards to instructions based on their empirical scarcity. Experiments show that the proposed metrics are highly correlated with model performance, and our methods unleash the potential of RL training: our best models outperform the baselines by a significant margin across three verifiable instruction following benchmarks. We release codes at https://github.com/mianzhang/MulIF.
Figures & tables
Figure 1: Instruction Accuracy: Before vs After RL
Figure 2: VIA and VSA vs Prompt Accuracy
Figure 3: Overview of the framework for mitigating exploration bias. Top (Behavioral Bootstrapping): The policy model is first evaluated on the training set to identify hard instruction types. Prompts containing at least one hard instruction are then sampled, and a subsequent rejection sampling phase is conducted to collect bootstrapping data. Finally, the policy model is supervised fine-tuned for one epoch on this collected data. Bottom (Scarcity-Aware Rewards): The bootstrapped model is fine-tuned via RL using Scarcity-Aware Rewards (Unary + Binary), which are calculated across a response group to incentivize the exploration of rare instructions and combinations.
IFBench
IFEval
Multi-IF
Model
Loose
Strict
Loose
Strict
Turn 1
Turn 2
Turn 3
Average
Off-the-shelf
GPT-OSS-120B
65.2
59.6
86.6
83.7
76.4
65.1
54.8
70.2
Llama-3.1-70B
35.0
29.8
83.7
81.8
80.5
71.9
63.1
63.7
Qwen3-32B
31.9
28.0
89.0
86.1
84.1
74.3
60.6
64.8
Gemma-3-27B
33.3
25.6
82.9
80.2
82.5
72.9
63.7
63.0
Table 1: Performance on Instruction Following Benchmarks. All off-the-shelf model results are reproduced by us. For the Qwen3 series, we use their non-thinking mode, while for GPT-OSS models, we use the default (medium) thinking effort. The best performance within each column for the same base model is boldfaced and the performance comparison of our methods with the baselines (Qwen1.7B-CR and Qwen7B-CR) is annotated.
Figure 4: The training curves of VIA, VSA, ICR, and Prompt Accuracy.
Figure 5: Accuracy Shift by Instruction Type
Figure 6: Policy Entropy
Figure 7: Ablation results on BeBoot data (left), reward balancing factor (middle), and response group size (right).
Instruction following is a fundamental capability of large language models (LLMs), yet continuously improving this capability remains challenging. Existing methods typically rely either on costly external supervision from humans or strong teacher models, or on self-play training with static-difficulty instructions that cannot evolve as the model's capabilities improve. To address these limitations, we propose SEIF (Self-Evolving Reinforcement Learning for Instruction Following), a self-evolving framework for enhancing the instruction-following ability of LLMs. SEIF forms a closed self-evolution loop that improves the model's instruction-following ability, where instruction difficulty evolution and model capability evolution reinforce each other. SEIF consists of four roles: an Instructor that generates increasingly challenging instructions, a Filter that removes conflicting or invalid instructions to ensure data quality, a Follower that learns to follow evolved instructions, and a Judger that provides reward signals for reinforcement learning. The Instructor and Follower are alternately trained and co-evolve throughout the process. Experiments across multiple model scales and architectures show that SEIF consistently improves instruction-following performance, suggesting strong generality. Further analyses reveal the sources of improvement and identify an effective training strategy for self-evolution on open-ended tasks: sufficient early-stage training to build a solid foundation, followed by moderate late-stage training to mitigate overfitting and achieve better final performance. The code and data are publicly available at https://github.com/Rainier-rq1/SEIF.
Qingyu Ren, Qianyu He, Jiajie Zhu +7
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · School of Data Science, Fudan University · Ant Group
Evaluation protocols can lag behind LLM capabilities. Programmatically verified benchmarks cover narrow surface constraints, whereas real-world instruction following and agentic workflows require judging semantic, contextual, and policy-dependent behavior. We study expert-curated rubric-based evaluation as a unified mechanism for measurement and reinforcement-learning rewards across two settings: complex instruction following and enterprise agentic tasks. We identify rubric-design choices that affect reward quality, including maximum viable atomicity, intent-aware criterion design, and LLM-judge calibration. We introduce ComplexConstraints, an expert-curated instruction-following suite comprising a public 75-prompt benchmark with 1,559 rubric criteria and a disjoint 1,000-prompt training set, with 10-40 atomic criteria per prompt. Empirically, rubric rewards improve training in both fixed task datasets, such as ComplexConstraints, and stateful RL environments, such as CoreCraft. Training a 4B model on ComplexConstraints improves mean criterion pass rate by +15.5 pp on a held-out split, bringing it within 0.5 pp of the untrained baseline of a roughly 60x larger Qwen3 model, and the gains transfer to external benchmarks the model never saw during training: +8.4 pp on AdvancedIF and +10.1 pp on MultiChallenge. In CoreCraft, rubric-reward RL likewise transfers to out-of-distribution benchmarks (+4.5 pp BFCL, +7.4 pp tau^2-Bench, +6.8 pp Toolathlon). These results show that expert-authored rubrics provide effective evaluation targets and scalable reward signals for improving LLM instruction following and agentic behavior.
Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which appends one of a small fixed set of instructions to task prompts during training, using the same set for every problem, increasing the coverage of behaviours attempted. To facilitate ICE, we combine RL on the instruction-conditioned policy with self-distillation of its correct rollouts into the unconditioned test-time policy. ICE with this objective improves Qwen3-1.7B held-out pass@1 performance at 4K response length on mathematical reasoning tasks by 5.0% relative to training with DAPO, with improvement persisting at a longer 8K context. The improvement does not appear for Qwen3-4B at 4K, where the instructions do not expand base-model coverage.
Jim Dilkes, Vahid Yazdanpanah, Sebastian Stein
University of Southampton Southampton United Kingdom