HIPO: Instruction Hierarchy via Constrained Reinforcement Learning

Keru Chen; Jun Luo; Sen Lin; Yingbin Liang; Alvaro Velasquez; Nathaniel Bastian; Shaofeng Zou

cs.LGMar 17, 2026

HIPO: Instruction Hierarchy via Constrained Reinforcement Learning

Authors: Keru Chen, Jun Luo, Sen Lin, Yingbin Liang, Alvaro Velasquez, Nathaniel Bastian, Shaofeng Zou

Organizations: School of ECEE Arizona State University · Department of ECE The Ohio State University · Computer Science Department University of Houston · College of Engineering & Applied Science CU Boulder ·

Abstract

Hierarchical Instruction Following (HIF) refers to the problem of prompting large language models with a priority-ordered stack of instructions. Standard methods like RLHF and DPO typically fail in this problem since they mainly optimize for a single objective, failing to explicitly enforce system prompt compliance. Meanwhile, supervised fine-tuning relies on mimicking filtered, compliant data, which fails to establish the priority asymmetry at the algorithmic level. In this paper, we introduce HIPO, a novel alignment framework that formulates HIF as a Constrained Markov Decision Process. HIPO elevates system prompts from mere input context to strict algorithmic boundaries. Using a primal-dual safe reinforcement learning approach, the algorithm dynamically enforces system prompt compliance as an explicit constraint, maximizing user utility strictly within this feasible region. Extensive evaluations across diverse model architectures (e.g., Qwen, Phi, Llama) demonstrate that HIPO significantly improves both system compliance and user utility. Furthermore, mechanistic analysis reveals that this constrained optimization autonomously drives the model to shift its attention toward long-range system tokens, providing a principled foundation for reliable LLM deployment in complex workflows.

Figures & tables

Figure 1: The HIPO framework for hierarchical instruction following. (A) Problem formulation: maximizing user utility ( $\mathbb{E}[r_{\text{user}}]$ ) subject to a system adherence constraint ( $\mathbb{E}[r_{\text{sys}}]\geq\tau$ ). (B) Performance illustration: HIPO improves the trade-off between system compliance and user utility relative to baselines. (C) The optimization pipeline: distinct system ( $r_{\text{sys}}$ ) and user ( $r_{\text{user}}$ ) rewards are derived for constrained optimization.

Method	Conflicting Split					Aligned Split
Method	QW-1.7B	QW-4B	Phi-3.8B	LL-3B	QW-8B	QW-1.7B	QW-4B	Phi-3.8B	LL-3B	QW-8B
Base	0.56 / 0.26	0.64 / 0.32	0.58 / 0.28	0.61 / 0.35	0.60 / 0.27	0.59 / 0.35	0.68 / 0.39	0.64 / 0.43	0.67 / 0.39	0.67 / 0.37
SFT	0.60 / 0.36	0.61 / 0.41	0.59 / 0.45	0.63 / 0.38	0.66 / 0.38	0.65 / 0.55	0.70 / 0.61	0.69 / 0.61	0.64 / 0.56	0.67 / 0.56
DPO	0.57 / 0.32	0.65 / 0.37	0.63 / 0.36	0.64 / 0.31	0.65 / 0.33	0.63 / 0.37	0.65 / 0.42	0.67 / 0.47	0.64 / 0.39	0.68 / 0.38
S-SM	0.57 / 0.21	0.65 / 0.31	0.52 / 0.33	0.62 / 0.35	0.61 / 0.30	0.60 / 0.27	0.66 / 0.39	0.63 / 0.49	0.64 / 0.46	0.64 / 0.36
FL	0.55 / 0.22	0.60 / 0.25	0.51 / 0.14	0.52 / 0.18	0.58 / 0.29	0.61 / 0.27	0.61 / 0.37	0.54 / 0.18	0.51 / 0.21	0.63 / 0.37

Table 1: Comprehensive evaluation of alignment methods on system prompt compliance (

r_{\text{sys}}

) and user prompt utility (

r_{\text{user}}

) across the conflicting and aligned test splits. Cell format:

r_{\text{sys}}

r_{\text{user}}

. The threshold for system prompt compliance is

\tau=0.7

for our HIPO algorithm. Abbreviations: QW (Qwen3), Phi (Phi-3), LL (Llama-3.2), S-SM (Split-Softmax), FL (FocalLoRA), S/U-only (Sys/User-only). Single-objective ablations for QW-8B are omitted due to computational constraints.

Method	MMLU-Redux ( $\uparrow$ )	WildJailbreak		DirectRequest	HumanJailbreaks
Method	MMLU-Redux ( $\uparrow$ )	ASR ( $\downarrow$ )	Over-refusal ( $\downarrow$ )	ASR ( $\downarrow$ )	ASR ( $\downarrow$ )
Base	0.5946	0.4845 $\rightarrow$ 0.3060	0.0238 $\rightarrow$ 0.0762	0.1625 $\rightarrow$ 0.1000	0.3581 $\rightarrow$ 0.2725
SFT	0.5784	0.5685 $\rightarrow$ 0.3250	0.0190 $\rightarrow$ 0.2809	0.2219 $\rightarrow$ 0.0531	0.2806 $\rightarrow$ 0.1938
DPO	0.5882	0.4485 $\rightarrow$ 0.2765	0.0142 $\rightarrow$ 0.0762	0.1750 $\rightarrow$ 0.1000	0.3544 $\rightarrow$ 0.2831
FL	0.5923	0.4630 $\rightarrow$ 0.3005	0.0333 $\rightarrow$ 0.0667	0.1563 $\rightarrow$ 0.1094	0.3394 $\rightarrow$ 0.2719
HIPO	0.5916	0.4230 $\rightarrow$ 0.2255	0.0286 $\rightarrow$ 0.0857	0.1375 $\rightarrow$ 0.0656	0.2538 $\rightarrow$ 0.1944

Table 2: Safety benchmarks and general capability (MMLU-Redux) results on Qwen3-1.7b. For safety metrics, values are reported in the format of “ Without

\rightarrow

With Safety System Prompt”. Lower values are better for both ASR and Over-refusal. HIPO effectively leverages system prompts to minimize ASR while avoiding the degenerate over-refusal behavior observed in SFT. Abbreviations: FL (FocalLoRA).

Metric	Base	HIPO
Attention Decay Metrics
Centroid	0.7159	0.7210 $\uparrow$ +0.0051
FarMass	0.6362	0.6418 $\uparrow$ +0.0056
NearMass	0.2083	0.2034 $\downarrow$ -0.0049
FarNearRatio	3.1703	3.2746 $\uparrow$ +0.1043
Span Attention Quality

Table 3: Mechanistic metrics at response onset (

n=200

). Paired differences ( HIPO - Base) are shown alongside HIPO metrics.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Representative examples of conflicting (left) and aligned (right) system-user prompt pairs.

Method	DeepSeek	Claude	GPT-4o	Qwen-Plus
Base	0.575 / 0.302	0.301 / 0.233	0.531 / 0.163	0.399 / 0.207
SFT	0.624 / 0.466	0.527 / 0.420	0.659 / 0.427	0.570 / 0.397
DPO	0.597 / 0.344	0.293 / 0.243	0.549 / 0.138	0.384 / 0.233
Split-Softmax	0.583 / 0.236	0.299 / 0.159	0.583 / 0.236	0.368 / 0.130
FocalLoRA	0.582 / 0.240	0.297 / 0.184	0.474 / 0.096	0.365 / 0.166
Sys-only	0.745 / 0.232	0.504 / 0.219	0.572 / 0.040	0.549 / 0.132

Appendix

Table 4: Cross-model validation of LLM-as-a-Judge reliability. We evaluate the generated responses of Qwen3-1.7B across different alignment methods using four distinct frontier LLMs. Values are reported as mean scores in the format of System / User . The relative performance ranking and the favorable trade-off achieved by HIPO remain generally consistent across the specific LLM evaluators tested in our experiments.

	DS	G4o	G4o-M	Grk	Qw-P
DS	1.000	0.793	0.686	0.753	0.840
G4o	0.793	1.000	0.856	0.836	0.920
G4o-M	0.686	0.856	1.000	0.760	0.770
Grk	0.753	0.836	0.760	1.000	0.829
Qw-P	0.840	0.920	0.770	0.829	1.000

Appendix

Table 5: Pairwise concordance rate (Eq. 6 ) on the 200-instance test set. Higher is better. To fit the single-column width, models are abbreviated as follows: DS : DeepSeek (our primary judge), G4o : GPT-4o, G4o-M : GPT-4o-mini, Grk : Grok-4.2, Qw-P : Qwen-Plus.

Figure 3: Training dynamics of the Lagrange multiplier $\lambda$ (top) and the system reward EMA (bottom). The multiplier increases to penalize the model when the reward is below the 0.7 target threshold, and automatically drops to zero once the constraint is consistently satisfied.

Hyperparameter	Value
Training & Hardware Setup
Base Models	Open-weight LLMs
Reference Model	Frozen
Quantization	4-bit NF4
Hardware	NVIDIA A100 / H100 GPUs
Precision	bfloat16

Appendix

Table 6: Key hyperparameters and implementation settings.

Panel of Figure 4 — Figure 4: The detailed evaluator prompts for System Prompt Compliance ( $r_{\text{sys}}$ ).

Panel of Figure 5 — Figure 5: The detailed evaluator prompts for User Utility ( $r_{\text{user}}$ ).

Panel of Figure 6 — Figure 6: (Example prompt from dataset; not an instruction to reviewers.) Qualitative comparison of model outputs in a conflicting scenario . The Base Model violates the system prompt by hallucinating and attempting to execute internal tool calls (e.g., search_web , run_python ) instead of relying on the provided FAQ. In contrast, our proposed HIPO successfully enforces the system constraint, directly guiding the user using only the provided facts.

Panel of Figure 7 — Figure 7: (Example prompt from dataset; not an instruction to reviewers.) Qualitative comparison in an aligned scenario where the user’s request does not conflict with the system constraints. The Base Model is distracted by the lengthy system prompt, leaking internal tags ( <exact instructions> ) and outputting a rambling, incomplete response with poor user utility. In contrast, HIPO gracefully handles the complex context, adhering to the negative constraint while delivering a direct, high-quality poem that maximizes user utility.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Reasoning Up the Instruction Ladder for Controllable Language Models
Oct 30, 2025Zishuo Zheng, Vidhisha Balachandran, Chan Young Park +2LLM Safety Alignment Instruction Following
Department of Computer Science and Engineering, The Ohio State University · Microsoft Research · Allen Institute for AI
Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency
Apr 10, 2026Shu Yang, Zihao Zhou, Di Wang +1LLM Alignment Inference-Time Language Model Alignment
King Abdullah University of Science and Technology (KAUST) · University of Liverpool · University of Edinburgh
IFHierBench: Hierarchical Instruction Following for Large Language Models
Jul 30, 2026Yuetian Mao, Chunyang ChenLLM Evaluation Language Model Generation Evaluation
Technical University of Munich

Oct 30, 2025J/K move · Enter open · S save

Reasoning Up the Instruction Ladder for Controllable Language Models

Zishuo Zheng, Vidhisha Balachandran, Chan Young Park, Faeze Brahman+1

Department of Computer Science and Engineering, The Ohio State University · Microsoft Research · Allen Institute for AI

LLM Safety Alignment Instruction Following LLM Reliability

Open paper AskPDF arXiv

As large language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources within a single prompt context. Enforcing an instruction hierarchy, where higher-level directives override lower-priority requests, is critical to the reliability and control of LLMs. In this work, we reframe instruction hierarchy resolution as a reasoning task. The model must first "think" about the relationship between a given user prompt and higher-priority instructions before generating a response. To enable this capability, we construct VerIH, a training dataset of constraint-following tasks with verifiable answers, comprising aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general reasoning capabilities of models to instruction prioritization. Our method leads to consistent improvements across multiple model families on both instruction following and instruction hierarchy benchmarks, achieving ~20% absolute improvement in conflict setups. Our method also leads to improved alignment to safety-critical scenarios beyond the training distribution, exhibiting increased robustness against jailbreak and prompt injection, reducing absolute attack success rates by up to 20%. Our results establish reasoning over instruction hierarchies as a practical mechanism for improving AI reliability, where targeted updates to system prompts produce predictable, controllable, and robust changes in model behavior.

Abstract

LLM Alignment

Explore similar work

Reasoning Up the Instruction Ladder for Controllable Language Models

Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency

IFHierBench: Hierarchical Instruction Following for Large Language Models