RL has emerged as a powerful paradigm for enhancing the instruction following capabilities of LLMs. While existing training recipes achieve substantial gains, we find that they suffer from exploration bias towards easy instructions when the training data has multiple instructions in a prompt. This bias is caused by two main reasons: 1) the policy model's initial ability to satisfy hard instructions is too low to trigger successful exploration during RL training, so the optimization is biased towards easy instructions; and 2) canonical RL training recipes typically employ a cumulative reward (the number of instructions fulfilled), treating all instructions equally, which biases the policy model towards fulfilling easy instructions to obtain the same amount of reward. To address these issues, we first propose two metrics to measure the exploration bias in instruction following and then introduce a two-stage framework to alleviate it: 1) Behavioral Bootstrapping, a lightweight rejection sampling fine-tuning stage before RL to activate hard instructions; and 2) Scarcity-Aware Rewards, a new RL reward function that assigns rewards to instructions based on their empirical scarcity. Experiments show that the proposed metrics are highly correlated with model performance, and our methods unleash the potential of RL training: our best models outperform the baselines by a significant margin across three verifiable instruction following benchmarks. We release codes at https://github.com/mianzhang/MulIF.
Figures & tables
Figure 1: Instruction Accuracy: Before vs After RL
Figure 2: VIA and VSA vs Prompt Accuracy
Figure 3: Overview of the framework for mitigating exploration bias. Top (Behavioral Bootstrapping): The policy model is first evaluated on the training set to identify hard instruction types. Prompts containing at least one hard instruction are then sampled, and a subsequent rejection sampling phase is conducted to collect bootstrapping data. Finally, the policy model is supervised fine-tuned for one epoch on this collected data. Bottom (Scarcity-Aware Rewards): The bootstrapped model is fine-tuned via RL using Scarcity-Aware Rewards (Unary + Binary), which are calculated across a response group to incentivize the exploration of rare instructions and combinations.
IFBench
IFEval
Multi-IF
Model
Loose
Strict
Loose
Strict
Turn 1
Turn 2
Turn 3
Average
Off-the-shelf
GPT-OSS-120B
65.2
59.6
86.6
83.7
76.4
65.1
54.8
70.2
Llama-3.1-70B
35.0
29.8
83.7
81.8
80.5
71.9
63.1
63.7
Qwen3-32B
31.9
28.0
89.0
86.1
84.1
74.3
60.6
64.8
Gemma-3-27B
33.3
25.6
82.9
80.2
82.5
72.9
63.7
63.0
Table 1: Performance on Instruction Following Benchmarks. All off-the-shelf model results are reproduced by us. For the Qwen3 series, we use their non-thinking mode, while for GPT-OSS models, we use the default (medium) thinking effort. The best performance within each column for the same base model is boldfaced and the performance comparison of our methods with the baselines (Qwen1.7B-CR and Qwen7B-CR) is annotated.
Figure 4: The training curves of VIA, VSA, ICR, and Prompt Accuracy.
Figure 5: Accuracy Shift by Instruction Type
Figure 6: Policy Entropy
Figure 7: Ablation results on BeBoot data (left), reward balancing factor (middle), and response group size (right).
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · School of Data Science, Fudan University · Ant Group