Cross-Relational Preference Learning for Better LLM Instruction Following

Runsheng Li; Kai Sun; Bin Shi; Bo Dong

cs.AIAug 29, 2026

Cross-Relational Preference Learning for Better LLM Instruction Following

Authors: Runsheng Li, Kai Sun, Bin Shi, Bo Dong

Organizations: School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Distance Education, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China

Abstract

Large Language Models (LLMs) still exhibit limited capability in following complex instructions. While existing approaches often rely on preference learning to enhance this ability, they typically overlook the relationships between the permissible response spaces of different instructions, which restricts a model to align with subtle and diverse constraint variations. To address this, we propose Cross-Relational Preference Learning (CRPL), a novel framework for constructing preference data that explicitly models inter-instruction relationships through two key techniques: Cross-Relationship Perturbation and Cross-Region Pair Sampling. This enables the generation of more diverse preference data that captures a wide spectrum of constraint variations. Additionally, we introduce an atomic constraint-based verification mechanism to rigorously assess response satisfaction, ensuring high-quality preference pair construction. Extensive experiments across multiple preference learning methods (e.g., DPO, KTO), LLM backbones and four instruction-following benchmarks demonstrate that our approach achieves substantial improvements over prior baselines and exhibits strong generalization.

Figures & tables

Figure 1: Relationships between the permissible response spaces of an original instruction and its perturbed variant. Colors denote: red — original instruction’s response space; green — perturbed instruction’s response space; blue — intersection of the two spaces.

Figure 2: An overview of our Cross-Relational Preference Learning (CRPL) framework. (1) Cross-Relationship Perturbation: We decompose the original instruction into atomic constraints and generate perturbed instructions that exhibit specific response-space relationships (containment, partial overlap, or disjoint) with the original one. (2) Cross-Region Pair Sampling: Guarded by an atomic constraint-based verification mechanism, we sample positive and negative responses from distinct sub-regions of the combined response space to construct diverse, high-quality preference pairs.

Method	Training size	IFEval		IFBench		Perturbed-IF		FollowBench		Avg.
Method	Training size	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	Avg.
Backbone: Qwen2.5-7B-Instruct
Qwen2.5-7B-Instruct	-	71.2	78.7	26.5	29.0	42.5	65.0	50.9	58.2	52.8
Training strategy: DPO
Rejection Sampling	12k	75.7	81.0	24.8	26.3	51.0	66.6	46.2	55.6	53.4
From Complex to Simple	12k	74.1	81.6	27.6	29.9	45.5	64.8	51.6	58.5	54.2

Table 1: Performance comparison of our method against prior baselines across the four test sets. For each backbone and training strategy, the best and second-best accuracy scores are highlighted in bold and underlined , respectively.

\text{Ours}_{\text{IFEval}}

and

\text{Ours}_{\text{IFBench}}

indicate models trained on the IFEval and IFBench training sets, respectively.

Model	Original		Containment-1		Containment-2		Partial Overlap		Disjoint
Model	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$
Qwen2.5-7B-Instruct	71.2	78.7	53.7	71.0	58.8	73.2	50.0	67.9	51.2	70.1
Rejection Sampling	75.7	81.0	57.5	75.0	60.0	75.6	55.0	72.8	62.5	73.7
From Complex to Simple	84.1	81.6	62.5	76.7	60.0	72.6	50.0	69.1	53.7	70.1
AutoIF	84.1	81.0	51.2	72.7	57.5	73.2	50.0	70.4	51.2	71.9
Ours	90.7	94.3	68.8	80.7	67.5	80.4	62.5	79.6	68.8	80.2

Table 2: Performance on original instructions and four types of perturbed instructions. Containment-1 and Containment-2 represent two subtypes of containment relationships, namely containing and contained.

Panel of Figure 3 — Figure 3: Performance scaling with training data size for DPO (left) and KTO (right) across different methods.

Method	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$
Ours	90.7	94.3
w/o Containment-1	85.9	90.7
w/o Containment-2	84.7	88.8
w/o Partial Overlap	84.3	88.8
w/o Disjoint	82.9	87.2

Table 3: Ablated results on different relationship types.

Method	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$
Ours	81.3	85.6
w/o Pair1	77.1	82.5
w/o Pair2	77.3	83.1
w/o Pair3	78.5	83.5
w/o Pair4	79.2	84.2

Table 4: Ablated results on cross-region pair sampling.

Method	TP	FP	TF	NF	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$
ACV (ours)	66	0	27	15	85.4	89.2
ILV	47	0	27	34	84.0	88.7
LAJ	52	4	23	29	80.8	85.8

Table 5: Performance comparison for different verification methods.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	DPO	KTO
Learning Rate	1e-5	1e-5
Train Batch Size	32	32
Micro Train Batch Size	8	8
Max Sequence Length	5102	5102
Beta	0.1	0.1
Max Epochs	2	3

Appendix

Table 6: Ablated results on different relationship types.

Model	Data	IFEval		IFBench
Model	Data	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$	$\text{Acc}_{\text{ins}}$	$\text{Acc}_{\text{con}}$
Qwen2.5-14B-Instruct	-	85.0	89.1	33.2	35.4
	$\text{Ours}_{\text{IFEval}}$	88.2	91.5	36.1	38.1
	$\text{Ours}_{\text{IFBench}}$	88.2	92.1	41.6	44.3
Qwen3-8B	-	89.4	93.4	25.6	28.0
	$\text{Ours}_{\text{IFEval}}$	89.1	93.1	29.6	31.3

Appendix

Table 7: Performance of Our Method across Different Backbone Models.

Figure 4: Prompt for instruction decomposition

Figure 5: Prompt for generating evaluation functions for atomic constraints.

Figure 6: Prompt for the first type of containment relationship perturbation. The valid response space of the original instruction contains that of the perturbed instruction.

Figure 7: Prompt for the second type of containment relationship perturbation. The valid response space of the perturbed instruction contains that of the original instruction.

Figure 8: Prompt for the partial overlap relationship perturbation.

Figure 9: Prompt for the disjoint relationship perturbation.

Instruction Following

Instruction following studies how AI systems interpret and reliably carry out user goals, constraints, formats, and priorities.

See all 135 papers

Instruction Quality Matters: Refining Instructions for Effective Preference Learning
Aug 27, 2026Seohyeong Lee, Hwaran Lee, Buru ChangPreference Learning LLM Alignment
Sogang University · Korea University
Mitigating Exploration Bias in RL for Multi-Instruction Following
Aug 24, 2026Mian Zhang, Yueqin Yin, Kaiyu He +4Instruction Following RL for Language Models
UT Dallas · UT Austin · UC Santa Barbara
ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modeling
Sep 26, 2026Bosi Wen, Yilin Niu, Xiaoying Ning +3Instruction Following Instruction Tuning
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI

Aug 27, 2026J/K move · Enter open · S save

Instruction Quality Matters: Refining Instructions for Effective Preference Learning

Seohyeong Lee, Hwaran Lee, Buru Chang

Sogang University · Korea University

Preference Learning LLM Alignment Preference Alignment

Open paper AskPDF arXiv

Preference learning optimizes models using response pairs, yet the informativeness of these pairs is fundamentally shaped by the instructions from which they are generated. We identify instruction quality as a hidden bottleneck in preference learning: low-quality or ambiguous instructions restrict the response-quality distribution, limiting strong chosen responses and weakening preference signals. Through Best- and Worst-of-N analyses, we show that instruction quality constrains both the ceiling and floor of sampled response quality. Motivated by this observation, we introduce an instruction-refinement pipeline that selects weak instructions using reward signals and revises them with rubric-guided LLM feedback, improving preference data without discarding examples. Across offline and online preference learning settings, experiments on multiple models and benchmarks show broad alignment improvements over original data and alternative data-improvement strategies. Further analyses indicate that instruction refinement raises achievable response quality and complements response-centric preference data curation. Overall, instruction quality emerges as a key factor governing how informative preference signals are formed for LLM alignment. Code is available at: https://github.com/01choco/instruction-refinement/

Abstract

Instruction Following

Explore similar work

Instruction Quality Matters: Refining Instructions for Effective Preference Learning

Mitigating Exploration Bias in RL for Multi-Instruction Following

ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modeling