Instruction Quality Matters: Refining Instructions for Effective Preference Learning

Seohyeong Lee; Hwaran Lee; Buru Chang

cs.CLAug 27, 2026

Instruction Quality Matters: Refining Instructions for Effective Preference Learning

Authors: Seohyeong Lee, Hwaran Lee, Buru Chang

Organizations: Sogang University · Korea University

Abstract

Preference learning optimizes models using response pairs, yet the informativeness of these pairs is fundamentally shaped by the instructions from which they are generated. We identify instruction quality as a hidden bottleneck in preference learning: low-quality or ambiguous instructions restrict the response-quality distribution, limiting strong chosen responses and weakening preference signals. Through Best- and Worst-of-N analyses, we show that instruction quality constrains both the ceiling and floor of sampled response quality. Motivated by this observation, we introduce an instruction-refinement pipeline that selects weak instructions using reward signals and revises them with rubric-guided LLM feedback, improving preference data without discarding examples. Across offline and online preference learning settings, experiments on multiple models and benchmarks show broad alignment improvements over original data and alternative data-improvement strategies. Further analyses indicate that instruction refinement raises achievable response quality and complements response-centric preference data curation. Overall, instruction quality emerges as a key factor governing how informative preference signals are formed for LLM alignment. Code is available at: https://github.com/01choco/instruction-refinement/

Figures & tables

Figure 1: Data example from the UltraFeedback Cui et al. (2024) . Ambiguous instructions can produce invalid responses, leading to weak preference learning signals. Refining such instructions yields high-quality outputs that provide stronger signals.

Figure 2: Instruction quality shapes response distributions and preference-pair informativeness. Left : (a) Best-of- $N$ reward shows that high-quality instructions raise the response-quality ceiling. (b) Worst-of- $N$ reward shows that they also raise the response-quality floor. Right : (c): Reward score distributions of chosen $y^{+}$ and rejected $y^{-}$ responses paired with low-quality instructions $x_{low}$ . (d): Those of high-quality instructions $x_{high}$ .

Method		DPO						SimPO
Backbone	Data Type	MT-Bench		Evol-Instruct		AlpacaEval 2.0		MT-Bench		Evol-Instruct		AlpacaEval 2.0
Backbone	Data Type	WR	Score	WR	Score	WR	LC WR	WR	Score	WR	Score	WR	LC WR
Llama3-8b	SFT	27.8	4.61	23.4	5.80	3.95	7.49	27.8	4.61	23.4	5.80	3.95	7.49
Llama3-8b	Original	39.7	4.91	33.3	6.15	5.91	11.90	47.5	5.26	40.8	6.53	8.65	15.90
	Refined	47.8	5.38	35.8	6.39	8.86	18.78	48.1	5.56	44.5	6.62	9.19	20.30
Mistral-7b	SFT	21.6	4.04	16.3	4.87	4.30	8.25	21.6	4.04	16.3	4.87	4.30	8.25

Table 1: Evaluation results on MT-Bench, Evol-Instruct, and AlpacaEval. For MT-Bench and Evol-Instruct, each cell reports the pairwise win rate and single-answer score; for AlpacaEval, each cell reports the win rate and length-controlled win rate. The highest values for each backbone are highlighted in bold .

Method	MT-Bench		Evol-Instruct		AlpacaEval
	WR	Score	WR	Score	WR	LC WR
SFT	27.8	4.61	23.4	5.80	3.55	7.15
DPO (Original Dataset)	39.7	4.91	33.3	6.15	5.91	11.90
Prompt Engineering
CoT Prompting ( Wei et al., 2022 )	30.6	4.68	28.2	5.98	4.82	9.42
Paraphrasing ( Zhou et al., 2022 )	33.8	4.80	30.3	5.95	4.82	9.75

Table 2: Results on MT-Bench, Evol-Instruct, and AlpacaEval across dataset improvement strategies. WR: win rate; Score: average score.

Method	MT-Bench		Evol-Instruct		AlpacaEval 2.0
Method	Win Rate	Score	Win Rate	Score	Win Rate	LC Win Rate
SFT	27.8	4.61	23.4	5.8	3.95	7.49
DPO	39.4	5.12	37.4	6.05	7.73	12.33
SPA (iter 3)	50.6	5.48	55.0	6.91	22.89	21.52
+ w/ instruction refinement	55.9 ( ↑ 5.3%p)	5.49 ( ↑ 0.01)	55.0 (–)	7.24 ( ↑ 0.33)	23.60 ( ↑ 0.71%p)	21.81 ( ↑ 0.29%p)
Reward Model (iter 3)	47.8	5.40	47.0	6.39	11.78	19.92

Table 3: Evaluation results on MT-Bench, Evol-Instruct, and AlpacaEval 2.0 for LLaMA-3-8B models trained with SPA (iter 3), reward model annotation, and implicit reward annotation. Bold indicates the better result within each method, and arrows indicate the change compared to the non-refined counterpart.

Model	DPO						SimPO
Model	MT-Bench		Evol-Instruct		AlpacaEval		MT-Bench		Evol-Instruct		AlpacaEval
	Pairwise	Single	Pairwise	Single	WR	LC WR	Pairwise	Single	Pairwise	Single	WR	LC WR
SFT	27.8%	4.61	23.4%	5.80	3.55	7.15	27.8%	4.61	23.4%	5.80	3.55	7.15
Original-OSSAT	41.9%	5.11	29.8%	6.32	7.90	11.71	37.5%	5.26	26.4%	6.41	8.75	12.32
Refined-OSSAT	40.9%	5.24	36.5%	6.26	7.99	13.86	44.7%	5.07	32.3%	6.48	8.37	13.31

Table 4: Cross-reward-model evaluation results on MT-Bench, Evol-Instruct, and AlpacaEval. For MT-Bench and Evol-Instruct, each cell reports the pairwise win rate and single-answer score, while for AlpacaEval, it reports the win rate and length-controlled win rate. The highest values in each column are highlighted in bold .

Figure 3: Best-of- $N$ and Worst-of- $N$ analysis showing that instruction refinement raises the response quality ceiling across high and low instruction quality groups, with more pronounced improvements for lower-quality instructions.

Figure 4: Sensitivity analysis of the minimum-reward threshold

\tau

for instruction refinement. Refinement consistently improves over the non-refined baseline, with optimal performance at moderate thresholds.

Method	MT-Bench		Evol-Instruct		AlpacaEval
	WR	Score	WR	Score	WR	LC WR
SFT	27.8	4.61	23.4	5.80	3.55	7.15
DPO (Original Dataset)	39.7	4.91	33.3	6.15	5.91	11.90
Refined Dataset
w/o Rubric-Based Prompt	47.8	5.52	36.2	6.54	8.60	16.23
w/ Rubric-Based Prompt	47.5	5.60	41.7	6.70	9.92	19.54

Table 5: Rubric ablation results across evaluation benchmarks. WR: win rate; Score: average score.

$\Delta$ Rubric Score	w/o Rubric		w/ Rubric
	$\Delta$ Avg.	$\Delta$ Min.	$\Delta$ Avg.	$\Delta$ Min.
Clarity	0.048	0.047	0.064	0.077
Specificity	-0.041	-0.029	-0.008	0.015
Completeness	-0.028	-0.017	-0.032	-0.005
Safety	0.062	0.057	0.091	0.091
Answerability	-0.002	0.009	0.056	0.078

Table 6: Pearson correlation between rubric score gains and average/minimum reward score gains, with and without rubric-based feedback.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method		DPO						SimPO
Backbone	Data Type	MT-Bench		Evol-Instruct		AlpacaEval		MT-Bench		Evol-Instruct		AlpacaEval
Backbone	Data Type	WR	Score	WR	Score	WR	LC WR	WR	Score	WR	Score	WR	LC WR
Llama3-8b	SFT	27.8	4.61	23.4	5.80	3.95	7.49	27.8	4.61	23.4	5.80	3.95	7.49
Llama3-8b	Original	45.0	5.57	40.6	6.60	8.74	17.59	45.6	5.45	38.1	6.63	7.08	14.92
	Refined	51.6	5.53	36.0	6.63	9.26	19.00	48.8	5.39	41.1	6.64	8.87	17.07

Appendix

Table 7: Evaluation results of prompt refinement experiment with the Feedback & Refine module, conducted under the same experimental setting as Table 1 . The highest values for each backbone are highlighted in bold .

Method		DPO						SimPO
Backbone	Data Type	MT-Bench		Evol-Instruct		AlpacaEval		MT-Bench		Evol-Instruct		AlpacaEval
Backbone	Data Type	WR	Score	WR	Score	WR	LC WR	WR	Score	WR	Score	WR	LC WR
Llama3-8b	SFT	0.278	4.61	0.234	5.80	3.55	7.15	0.278	4.61	0.234	5.80	3.55	7.15
	Original	0.466	5.47	0.374	6.39	9.43	16.56	0.466	5.23	0.401	6.56	3.61	8.07
	Refined	0.453	5.22	0.383	6.47	9.38	17.20	0.466	5.30	0.436	6.49	6.75	13.84

Appendix

Table 8: Results with an alternative reward model on MT-Bench, Evol-Instruct, and AlpacaEval. For MT-Bench and Evol-Instruct, conducted under the same experimental setting as Table 1 . The highest values for each backbone are highlighted in bold .

Method	MMLU-med						MedQA	MedMCQA	PubMedQA
	Clinical Knowledge	Medical Genetics	Anatomy	Professional Medicine	College Biology	College Medicine	Acc.	Acc.	Acc.
SFT	0.592	0.710	0.556	0.592	0.618	0.578	0.515	0.486	0.022
DPO (Original)	0.660	0.620	0.593	0.607	0.681	0.566	0.534	0.496	0.196
Refined	0.694	0.680	0.593	0.588	0.681	0.578	0.537	0.502	0.254

Appendix

Table 9: Medical-domain evaluation results on MMLU-med, MedQA, and PubMedQA. The highest values in each column are highlighted in bold .

Figure 5: Distribution of feedback scores and minimum response rewards before and after instruction refinement, showing consistent improvements in both instruction quality and lower-bound response quality.

Method	MT-Bench		Evol-Instruct		AlpacaEval 2.0
Method	Win Rate	Score	Win Rate	Score	Win Rate	LC Win Rate
SFT	27.8	4.61	23.4	5.8	3.95	7.49
DPO	39.4	5.12	37.4	6.05	7.73	12.33
SPA (iter 2)	53.1	5.56	56.0	6.46	19.14	20.27
+ w/ instruction refinement	54.7 ( ↑ 1.6%p)	5.59 ( ↑ 0.03)	55.7 (↓ 0.3%p)	6.89 ( ↑ 0.43)	20.80 ( ↑ 1.66%p)	20.82 ( ↑ 0.55%p)
Reward Model (iter 2)	43.1	5.19	40.8	6.41	9.19	15.49

Appendix

Table 10: Evaluation results on MT-Bench, Evol-Instruct, and AlpacaEval 2.0 for LLaMA-3-8B models trained with SPA (iter 2). Bold indicates the better result within each method, and arrows indicate the change compared to the non-refined counterpart.

Figure 6: AlpacaEval LC win rate result per iteration of online preference learning. Model capability increases as train iteration repeat, showing major increase in SPA + prompt refinement.

Figure 7:

\tau

Sensitivity Analysis Result in MT-Bench Zheng et al. (2023) , Evol-Instruct Xu et al. (2023) , and AlpacaEval 2.0 Li et al. (2023) .

Dataset	MT-Bench				Evol-Instruct
Dataset	Overall	Low-IFD	Mid-IFD	High-IFD	Overall	Low-IFD	Mid-IFD	High-IFD
Original	4.913	4.302	5.056	4.925	6.147	6.342	6.208	6.411
Refined	5.381	5.037	5.717	5.396	6.390	6.205	6.417	6.548

Appendix

Table 11: Evaluation results across different IFD difficulty levels on MT-Bench ( Zheng et al., 2023 ) and Evol-Instruct ( Xu et al., 2023 ) . Each cell reports the average score. The highest value in each column is highlighted in bold .

Category	Original instruction	Refined instruction
Major Shift	How to use corkscrew to chop the vegetables?	Ensure safety by securely holding a knife to chop carrots, onions, and bell peppers on a stable cutting board.
Minor Shift	Create a sentence using five synonyms	Use five synonyms for “happy” in a single sentence.

Appendix

Table 12: Examples of major and minor instruction shifts between original and refined instructions.

Figure 8: Performance across refinement iterations. Win rate generally improves from SFT to later refinement iterations across MT-Bench, Evol-Instruct, and AlpacaEval.

Original instruction: How long is the needle inside the monkey’s ear?

Original Response 1: The needle inside the monkey’s ear is 2cm long.

Original Response 2: The needle is 2.5cm long and it is inserted into the monkey’s ear. […]

Feedback (Evaluator): Clarity: 1/5, Specificity: 1/5, Completeness: 1/5, Safety: 3/5, Answerability: 1/5, Conciseness: 3/5, FormatConsistency: 4/5. "Unclear context, lacks details, unanswerable without additional information."

Refined instruction: Provide the typical length of needles used in veterinary procedures for monkeys’ ears, specifying the context and purpose of the procedure.

Refined Response 1: The typical length ranges from 1 to 1.5 inches . These needles are usually used for ear canal biopsies or ear canal washes to diagnose […]

Refined Response 2: The typical length is 1 inch (2.5 cm) . This needle length is used for blood collection, injection, or fluid removal […]

Appendix

Table 13: Case study of instruction refinement.

Question: Identify the named entities (people, organizations, locations) mentioned in the given news article. Please generate a JSON dictionary that lists the named entities in three separate groups based on their entity types. The key is the type of entity and the value is a list of strings.

Article: Yesterday, Adamson Emerson, the CEO of Faraday, and Dieter Zetsche, the CEO of Daimler AG, announced plans to build a new Gigafactory in Berlin. The facility will be a joint venture between Faraday and Daimler, producing electric vehicles and battery packs for both companies, creating thousands of job opportunities in the region. Emerson and Zetsche stated that the strategic location of Berlin, coupled with its skilled workforce and strong infrastructure, makes it an ideal choice for expansion. The new Gigafactory aims to meet the growing demand for electric vehicles in Europe and contribute to a sustainable future. Volkswagen CEO Herbert Diess welcomed the news, saying greater collaboration will benefit the auto industry’s transition to e-mobility.

Appendix

Table 14: Representative degradation case in a strict-format named entity extraction task. The refined model adds non-requested content and misclassifies entities, leading to a lower evaluation score.

Preference Learning

Preference learning infers people’s values and trade-offs from choices, comparisons, rankings, and feedback to guide predictions and decisions.

See all 150 papers

Aug 29, 2026

Cross-Relational Preference Learning for Better LLM Instruction Following

Large Language Models (LLMs) still exhibit limited capability in following complex instructions. While existing approaches often rely on preference learning to enhance this ability, they typically overlook the relationships between the permissible response spaces of different instructions, which restricts a model to align with subtle and diverse constraint variations. To address this, we propose Cross-Relational Preference Learning (CRPL), a novel framework for constructing preference data that explicitly models inter-instruction relationships through two key techniques: Cross-Relationship Perturbation and Cross-Region Pair Sampling. This enables the generation of more diverse preference data that captures a wide spectrum of constraint variations. Additionally, we introduce an atomic constraint-based verification mechanism to rigorously assess response satisfaction, ensuring high-quality preference pair construction. Extensive experiments across multiple preference learning methods (e.g., DPO, KTO), LLM backbones and four instruction-following benchmarks demonstrate that our approach achieves substantial improvements over prior baselines and exhibits strong generalization.

Runsheng Li, Kai Sun, Bin Shi +1

School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Distance Education, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China

Instruction Following LLM Post-Training Preference Optimization Preference-Based RL

Ask PDF

Sep 5, 2026

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection

Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective alignment. Existing datasets are frequently plagued by inherent noise and distribution shifts, which inherently limit model performance. To bridge this gap, we propose AlignDiff, a preference data filtering framework driven by intrinsic model signals. AlignDiff first identifies samples with clear preferences using both positive and inverse signals, then prioritizes the more challenging samples based on the average negative log-likelihood gap, encouraging the model to learn richer information from them. AlignDiff is evaluated on two widely used model families (LLaMA and Qwen) and three benchmarks widely adopted in the alignment community (AlpacaEval 2.0, Arena-Hard, and MT-Bench). Across all settings, it consistently outperforms seven strong baselines. We conduct comprehensive ablation studies to validate the effectiveness of AlignDiff, and further show that difficulty-based curriculum learning improves model performance.

Peng Lai, He Zhu, Zhiwen Ruan +6

Southern University of Science and Technology · Peking University · MSRA +2

Data Selection Preference Alignment

Ask PDF

Oct 25, 2026

"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics

"Data quality" is widely invoked in instruction tuning, but the term is often underspecified. We study it as an operational property by asking which simple example-level metrics, drawn from prior work on data selection, text quality, and instruction tuning, best predict training utility under a fixed finetuning budget. For each metric, we sort instruction-response examples by score, partition them into low, middle, and high bins with roughly equal token mass, finetune separately on each bin, and compare validation perplexity and LLM-as-judge scores for the models' generated answers as a function of cumulative training tokens. Our results suggest that answer-dense supervision, as captured by response ratio, is a stronger and more portable predictor of instruction-tuning utility than commonly used alternative proxies.

Vladimir Necula, David Gvadzabia, Sofia Serrano

University of Cambridge Department of Physics · Lafayette College Department of Computer Science

Instruction-Tuning Data Selection Instruction Tuning

Ask PDF

Abstract

Preference Learning

Explore similar work

Cross-Relational Preference Learning for Better LLM Instruction Following

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection

"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics