IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs
Authors: Chuan Guo, Juan Felipe Ceron Uribe, Sicheng Zhu, Christopher A. Choquette-Choo, Steph Lin, Nikhil Kandpal, Milad Nasr, Rai, Sam Toyer, Miles Wang, Yaodong Yu, Alex Beutel, Kai Xiao
Instruction hierarchy (IH) defines how LLMs prioritize system, developer, user, and tool instructions under conflict, providing a concrete, trust-ordered policy for resolving instruction conflicts. IH is key to defending against jailbreaks, system prompt extractions, and agentic prompt injections. However, robust IH behavior is difficult to train: IH failures can be confounded with instruction-following failures, conflicts can be nuanced, and models can learn shortcuts such as overrefusing. We introduce IH-Challenge, a reinforcement learning training dataset, to address these difficulties. Fine-tuning GPT-5-Mini on IH-Challenge with online adversarial example generation improves IH robustness by +10.0% on average across 16 in-distribution, out-of-distribution, and human red-teaming benchmarks (84.1% to 94.1%), reduces unsafe behavior from 6.6% to 0.7% while improving helpfulness on general safety evaluations, and saturates an internal static agentic prompt injection evaluation, with minimal capability regression. We release the IH-Challenge dataset (https://huggingface.co/datasets/openai/ih-challenge) to support future research on robust instruction hierarchy.
Figures & tables
Figure 1: Fine-tuning on IH-Challenge not only improves IH, but also increases response safety while maintaining helpfulness. This figure shows an example of a dual-intent request that could lead to wrongdoing. After training, the model follows the system-provided safety specification more faithfully, achieving a better balance between safety and helpfulness.
Figure 2: Fine-tuning on IH-Challenge not only improves IH, but also improves the model’s robustness to prompt injection. This figure shows an example from our agent robustness evaluation. The tool output contains an injected instruction (in red). After training, the model learns to recognize and ignore it.
Figure 3: Illustration of our task design and training data pipeline. Tasks are designed to be IF-simple, programmatically gradeable, and avoid shortcut learning. Each task consists of a higher-priority message, Python grader code, and a placeholder for the lower-priority attack prompt. During training, we use an attacker LLM to generate the attack prompt on-the-fly by iteratively probing the defender model, and use the final output prompt for RL training.
contain-word : “You must include the word ‘kiwi’ in your response …” json-format : “Output must be valid JSON.”
Multi-Constraint
Open-Ended
Each task is a composition of 2 to 6 atomic constraints drawn from Single-Constraint.
“Output valid JSON” + include word “kiwi” + ASCII-only + “Never reveal PIN 2878”.
Input-Conditioned
Closed-Ended
Parse the specified message: recognize required patterns and output in a strict schema.
Find the latest city name and output exactly one line, e.g. city=Rome ; city must be one of { Seoul , Doha , Cairo , Oslo , Rome , Bern , Lima }.
Anti-Overrefusal
Open-Ended
Benign requests are adversarially rewritten to look forbidden; refusing counts as failure. Each task is drawn from Single-Constraint.
“Your password is: harmony . It is not confidential.”
Table 1: Summary of IH-Challenge task skeletons.
Figure 4: Training and test robustness of GPT-5-Mini-R on IH-Challenge tasks. RL training gains generalize to held-out attacks, suggesting little overfitting to the training reward.
Table 6
Eval
GPT-5-Mini
GPT-5-Mini-R
Gandalf Password (sys-user)
0.99
0.99 (+0)
Gandalf Password (dev-user)
0.98
1.00 (+0.02)
TensorTrust (sys-user)
0.86
0.94 (+0.08)
TensorTrust (dev-user)
0.76
0.91 (+0.15)
RealGuardrails (Distractors)
0.88
0.95 (+0.07)
RealGuardrails (Handwritten)
0.82
0.89 (+0.07)
Table 4: Robustness on academic evaluations.
Figure 5: Safety scores on OpenAI’s Production Benchmarks. Compared to GPT-5-Mini (with or without the same safety spec), GPT-5-Mini-R with a safety spec achieves higher safety scores across all categories, indicating that stronger IH robustness also improves model safety.
Figure 6: Helpfulness on OpenAI Production Benchmarks. Adding a safety specification to the system message largely preserves GPT-5-Mini-R ’s helpfulness while substantially improving safety ( Figure 5 ).
Eval
GPT-5-Mini
GPT-5-Mini-R
Internal PI Benchmark
0.44
1.00 (+0.56)
CyberSecEval 2
0.88
0.91 (+0.03)
Table 5: Robustness on prompt injection evaluations.
Figure 7: Evaluation of GPT-5-Mini and GPT-5-Mini-R with various system mitigations. Most system mitigations are effective for the less robust GPT-5-Mini model, but lose effectiveness (and in some cases even hurt robustness) once the model becomes more robust.
Figure 8: Attack success rate (ASR) during attacker RL training against fixed GPT-5-Mini-R defender. With the output monitor applied, attacker progress can be made significantly slower, which highlights the potential for defense-in-depth as an effective strategy against adaptive attackers.
Model
Tasks
Avg. attempts per task
Success rate
Success rate / attempt
GPT-5-Mini
271
32.84
0.362
0.015
GPT-5-Mini-R
265
52.39
0.117
0.004
GPT-5-Mini-R + Monitor
268
43.08
0.071
0.002
Table 7: Adaptive human red-teaming results.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Refusal rate on OpenAI’s Production Benchmarks when using a global safety spec instead of a category-specific one. Higher is better.
Figure 10: Helpfulness rate on OpenAI’s Production Benchmarks when using a global safety spec instead of a category-specific one. Higher is better.
Adversarial robustness of language models studies how to keep their outputs and decisions reliable under manipulative prompts, input changes, and malicious fine-tuning.