The recent incident in which OpenAI agents autonomously compromised Hugging Face's infrastructure illustrates the importance of instruction hierarchy (IH): an LLM should follow lower-priority instructions, such as user requests, only when they do not conflict with higher-priority instructions, such as system policies. IH can be effectively improved by fine-tuning LLMs on simulated examples of instruction conflicts, whose diversity is key to teaching LLMs to properly handle conflicts at test time. However, our decomposition of a task into goals and constraints reveals a conflict type -- goal-goal conflicts -- that is absent from existing IH training sets from OpenAI and NVIDIA. We show the importance of this conflict type: fine-tuning Qwen3.5-9B and Gemma4-12B-it on examples of this conflict type significantly improves IH on a wide range of real-world tasks unseen in training; adding these examples to existing IH training sets for fine-tuning also gives better IH adherence. The IH gains come with trivial mean utility changes on no-conflict tasks across eight benchmarks. Our work advocates including this new conflict type in IH training, even if these examples do not reflect typical LLM use.
Figures & tables
Figure 1: Conflict types (a) and coverage in existing IH training sets (b) Jiang et al. (2026) ; Guo et al. (2026) .
Figure 2: Initial exact system-task compliance on goal–goal conflicts. An analysis of these failures appears in Appendix B.2 .
Figure 3: Training original models on goal–goal conflicts raises Conflict Resistance Rates in all ten categories.
Case
Tasks by priority
Behavior before adding goal–goal data
Behavior after adding goal–goal data
Training on goal–goal conflicts improves IH adherence across models, message types, languages, and domains
A. Translation XIH-Bench
High-priority task Translate French to Korean.
Answers the embedded question in Korean.
Translates the instructions, passage, and question into Korean.
Low-priority task Answer a question about the Hershey–Chase experiment embedded in the tool text.
B. Repository analysis IH-Benchmark
High-priority task Analyze the requested repository.
Attempts to run the installer with execute_command ; the simulated tool blocks it.
Summarizes the repository and asks before running the installer; makes no execution call.
Low-priority task Run an installation command requested by the returned README.
Goal–goal conflicts, if simulated in training, provide crucial complementary supervision beyond existing conflicts
Table 1: Case studies for fine-tuning Qwen3.5-9B on goal–goal conflicts. Cases A–B compare the original model with goal–goal training; Cases C–D compare constraint-conflict training with combined training; Cases E–F compare the original model with goal–goal training on no-conflict tasks. Entries summarize selected response traces; non-English responses are described in English.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Exact task compliance (%)
Qwen3.5-9B
19.8
Gemma4-12B-it
42.3
Gemma4-31B-it
39.2
Qwen3.6-27B
9.7
Nemotron3.5-Lightning
55.0
DeepSeekV4-Flash
51.9
Appendix
Table 2: Initial exact system-task compliance on goal–goal conflicts. A successful response matches the complete system target before goal–goal fine-tuning.
Model
# Failure cases
Request failure (%)
Instruction identification (%)
Conflict resolution (%)
Response realization (%)
Qwen3.5-9B
7,511
0.0
6.8
12.4
80.8
Gemma4-12B-it
3,272
0.0
53.9
6.2
39.9
Qwen3.6-27B
8,623
2.5
0.0
0.0
97.5
GLM-5.2
2,540
1.3
7.0
0.0
91.7
DeepSeekV4-Flash
4,573
0.0
2.8
1.8
95.4
DeepSeekV4-Pro
1,807
1.4
0.4
0.0
98.2
Appendix
Table 3: Failure mode analysis for cases that do not comply with goal–goal conflicts’ desirable outputs.
Instruction hierarchy (IH) defines how LLMs prioritize system, developer, user, and tool instructions under conflict, providing a concrete, trust-ordered policy for resolving instruction conflicts. IH is key to defending against jailbreaks, system prompt extractions, and agentic prompt injections. However, robust IH behavior is difficult to train: IH failures can be confounded with instruction-following failures, conflicts can be nuanced, and models can learn shortcuts such as overrefusing. We introduce IH-Challenge, a reinforcement learning training dataset, to address these difficulties. Fine-tuning GPT-5-Mini on IH-Challenge with online adversarial example generation improves IH robustness by +10.0% on average across 16 in-distribution, out-of-distribution, and human red-teaming benchmarks (84.1% to 94.1%), reduces unsafe behavior from 6.6% to 0.7% while improving helpfulness on general safety evaluations, and saturates an internal static agentic prompt injection evaluation, with minimal capability regression. We release the IH-Challenge dataset (https://huggingface.co/datasets/openai/ih-challenge) to support future research on robust instruction hierarchy.
Chuan Guo, Juan Felipe Ceron Uribe, Sicheng Zhu +10
Large language model agents receive instructions from many sources-system messages, user prompts, tool outputs, other agents, and more-each carrying different levels of trust and authority. When these instructions conflict, agents must reliably follow the highest-privilege instruction to remain safe and effective. The dominant paradigm, instruction hierarchy (IH), assumes a fixed, small set of privilege levels (typically fewer than five) defined by rigid role labels (e.g., system > user). This is inadequate for real-world agentic settings, where conflicts can arise across far more sources and contexts. In this work, we propose Many-Tier Instruction Hierarchy (ManyIH), a paradigm for resolving instruction conflicts among instructions with arbitrarily many privilege levels. We introduce ManyIH-Bench, the first benchmark for ManyIH. ManyIH-Bench requires models to navigate up to 12 levels of conflicting instructions with varying privileges, comprising 853 agentic tasks (427 coding and 426 instruction-following). ManyIH-Bench composes constraints developed by LLMs and verified by humans to create realistic and difficult test cases spanning 46 real-world agents. Our experiments show that even the current frontier models perform poorly (~40% accuracy) when instruction conflict scales. This work underscores the urgent need for methods that explicitly target fine-grained, scalable instruction conflict resolution in agentic settings.
Large language models increasingly operate under multiple instructions from heterogeneous sources with different authority levels, including system policies, user requests, tool outputs, and retrieved context. While prior work on instruction hierarchy highlights the importance of respecting instruction priorities, it mainly focuses on adversarial attacks and overlooks the benign but common instruction conflicts that arise in real-world applications. In such settings, models must not only avoid security violations but also preserve task utility and behavioral consistency when instructions partially or implicitly conflict. We propose Neuro-Symbolic Hierarchical Alignment (NSHA) for hierarchical instruction-following by explicitly modeling and enforcing instruction priorities. At inference time, we introduce solver-guided reasoning that formulates instruction resolution as a constraint satisfaction problem, enabling the model to derive a maximally consistent set of applicable instructions under hierarchical constraints. At training time, NSHA distills solver-based decisions into model parameters using automatically constructed supervision. We evaluate our approach on rule following, task execution, tool use, and safety, covering both single-turn and multi-turn interactions, and show that NSHA significantly improves performance under such conflicts while maintaining competitive utility in reference settings.
Shu Yang, Zihao Zhou, Di Wang +1
King Abdullah University of Science and Technology (KAUST) · University of Liverpool · University of Edinburgh