Instruction Data Selection via Answer Divergence

Bo Li; Mingda Wang; Shikun Zhang; Wei Ye

cs.CLApr 12, 2026

Instruction Data Selection via Answer Divergence

Authors: Bo Li, Mingda Wang, Shikun Zhang, Wei Ye

Organizations: National Engineering Research Center for Software Engineering, Peking University · School of Computer Science, Peking University · PKU-CMCC(Hubei) Joint Research Lab for LLM Industrial Applications · School of Health Sciences and Biomedical Engineering, Hebei University of Technology

Abstract

Instruction tuning relies on large instruction-response corpora whose quality and composition strongly affect downstream performance. We propose Answer Divergence-Guided Selection (ADG), which selects instruction data based on the geometric structure of multi-sample outputs. ADG draws several high-temperature generations per instruction, maps responses into an embedding space, and computes an output divergence score that jointly encodes dispersion magnitude and shape anisotropy. High scores correspond to instructions whose answers are both far apart and multi-modal, rather than clustered paraphrases along a single direction. Across two backbones and three public instruction pools, fine-tuning on only 10K ADG-selected examples consistently outperforms strong selectors on six benchmarks spanning reasoning, knowledge, and coding. Analyses further show that both dispersion magnitude and shape anisotropy are necessary, supporting answer divergence as a practical signal for instruction data selection. Code and appendix are included in the supplementary materials.

Figures & tables

Figure 1: Overview of ADG: sample $K$ answers per instruction, embed answers from output hidden states, score with $D(x)$ (dispersion magnitude) and $I(x)$ (shape anisotropy), and select top examples within semantic bins.

Backbone: LLaMA3-8B
	Alpaca-GPT4				WizardLM				CoT
	Reason.	Know.	Code.	Avg.Score	Reason.	Know.	Code.	Avg.Score	Reason.	Know.	Code.	Avg.Score
All Data	45.95	57.63	37.87	47.15	51.60	54.20	42.85	49.55	46.18	47.60	26.88	40.22
Random	42.81	57.07	38.30	46.06	48.38	56.89	38.81	48.03	39.58	47.00	29.38	38.65
IFD	43.19	59.65	43.93	48.92	50.34	57.73	42.52	50.19	37.09	56.63	39.19	44.30
SuperFiltering	46.80	58.46	42.93	49.39	51.09	57.87	42.11	50.35	44.89	49.04	38.19	44.03

Table 1: Main results with a fixed selection budget (

M{=}10

K) for selection methods across three selection datasets (Alpaca-GPT4, WizardLM, and CoT) and two backbones (LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct). Reason. , Know. , and Code. denote the average performance on reasoning (BBH, GSM8K), knowledge (MMLU, TruthfulQA), and code generation (HumanEval, MBPP). Avg.Score is computed as the unweighted average of the three capability scores. All selection methods select 10K instances from the same pool, while All Data uses the full pool. All models are fine-tuned and evaluated with identical training and decoding configurations, and we report the mean over five random seeds to reduce randomness.

Figure 2: Effect of the fusion weight

\lambda

on ADG. We report the mean Avg.Score over five runs, with the shaded band indicating standard deviation across five runs.

(A) Score-rank
Ablation Setting	Avg.Score
Top 10K (default)	50.85
Middle 10K	42.69
Tail 10K	37.93
(B) Sampling temperature for scoring (top- $p{=}0.9$ )
$T{=}1.0$	48.49

Table 2: Ablations on key components and hyperparameters of ADG: (A) selecting 10K examples from different score-rank segments, (B) the temperature used for multi-sample answer generation, (C) the number of bins (

B

) for stratified top selection, and (D) the necessity of combining the two complementary scoring components. We report mean Avg.Score over five runs.

Figure 3: Task-type composition shift of ADG-selected data. We report

\Delta

proportion (percentage-point change) between ADG and four reference sets (All Data, SuperFiltering, ZIP, and ADG (Bottom)) on Alpaca-GPT4. Task types are ordered by

\Delta(\text{ADG}-\text{SuperFiltering})

from high to low.

Fine-tune	Selection	Reason.	Know.	Code.	Avg.Score
Qwen2.5	Qwen2.5	75.43	67.36	64.40	69.06
Qwen2.5	LLaMA3	76.09	67.11	62.69	68.63
LLaMA3	LLaMA3	50.11	60.31	42.14	50.85
LLaMA3	Qwen2.5	49.84	59.61	41.11	50.19

Table 3: Cross-backbone generalization of ADG on Alpaca-GPT4 dataset with a 10K budget: we score and select data using the specified selection backbone, then fine-tune the specified backbone.

Selector	Time (h) $\downarrow$	Ex/s $\uparrow$
SuperFiltering	1.05	13.76
ADG (ours)	1.78	8.10
ZIP	3.17	4.56
MIG	3.92	3.69

Table 4: End-to-end selection efficiency on Alpaca-GPT4 (

N{=}52{,}002

) for producing a 10K subset using 4 GPUs. Ex/s is computed as

N/(\text{time}\times 3600)

Layer	Reason.	Know.	Code.	Avg.Score
0–3	44.46	57.52	36.87	46.28
4–7	42.81	57.54	36.59	45.65
8–11	38.37	57.11	35.97	43.82
12–15	42.74	56.92	38.08	45.91
16–19	47.62	59.27	40.41	49.10
20–23	50.06	54.82	39.80	48.23

Table 5: Robustness of ADG to different 4-layer output representation windows on LLaMA3-8B-Instruct with Alpaca-GPT4 under a fixed 10K selection budget. Reason., Know., and Code. denote the average performance on reasoning (BBH, GSM8K), knowledge (MMLU, TruthfulQA), and code generation (HumanEval, MBPP), respectively. Avg.Score is the unweighted average of the three capability scores.

Figure 4: A quadrant-based case study of ADG using $D(x)$ (Dispersion magnitude) and $I(x)$ (Shape anisotropy). Each quadrant shows one instruction with five sampled outputs, illustrating distinct disagreement patterns; ADG favors the high- $D(x)$ and high- $I(x)$ region (top-right).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Number Instances
Training
Alpaca-GPT4	52,002
WizardLM	70,000
CoT	100,000
Alpaca-GPT4-ZH	42,677
Evaluation

Appendix

Table 6: Datasets used in our experiments. We report the training selection pools and evaluation benchmarks, together with the number of instances for each dataset.

Figure 5: Task-Type classification prompt used in this paper.

Selector	Avg.Score	CMMLU	C-Eval
All Data	40.84	41.11	40.56
Random	41.32	42.08	40.56
SuperFiltering	44.50	44.02	44.97
Rethinking	42.40	42.53	42.27
ADG (Ours)	45.54	45.09	45.99

Appendix

Table 7: Results on Chinese benchmarks. We select 10K instances from Alpaca-GPT4-ZH, fine-tune LLaMA3-8B-Instruct and evaluate on CMMLU and C-Eval. We report the mean over five runs.

Method	Cognitive	Instruction	Response	Educational	Scalability
	Complexity	Precision	Quality	Value	Potential
Random	3.94	8.10	7.64	5.00	5.56
SuperFiltering	5.79	8.56	8.07	6.83	7.27
ADG_Bottom	3.02	8.02	7.74	4.13	4.80
ADG_Top(default)	6.37	8.64	8.16	7.11	7.48

Appendix

Table 8: LLM-based rating scores (GPT-4o) on five sample-quality dimensions. Each value is the mean over 1,000 randomly sampled instruction–response pairs from the corresponding subset.

$K$	Avg.Score	BBH	GSM8K	MMLU	TruthfulQA	HumanEval	MBPP
3	49.32	53.05	42.00	62.40	54.32	37.80	46.40
5 (default)	50.85	57.38	42.84	62.67	57.94	42.07	42.20
7	46.79	54.31	34.87	61.24	54.77	34.76	40.80
10	45.72	50.38	37.15	59.93	55.73	33.54	37.60

Appendix

Table 9: Ablation on the number of sampled answers

K

for ADG scoring and selection, evaluated with a LLaMA3-8B-Instruct backbone on the Alpaca-GPT4 pool. Bold indicates the best score per column.

Figure 6: Average token length of the selected subsets, reporting instruction (input) and response (output) lengths under the same tokenizer.

$M$	Method	BBH	MMLU	HumEval	Avg.Score
3K	Random	51.37	61.26	34.76	49.13
	SuperFiltering	51.94	61.65	42.68	52.09
	ADG	52.46	62.92	43.29	52.89
5K	Random	51.26	61.72	35.80	49.59
	SuperFiltering	54.04	59.97	37.80	50.60
	ADG	53.68	61.06	43.34	52.69

Appendix

Table 10: Budget-scaling results on LLaMA3-8B-Instruct with Alpaca-GPT4 as the selection pool. Avg.Score is the unweighted average of the three representative benchmarks.

Figure 7: Trade-off of ADG scoring w.r.t. response sampling length on Alpaca-GPT4 dataset.

	Reasoning		Knowledge		Coding		Avg.Score
	BBH	GSM8K	MMLU	TruthfulQA	HumanEval	MBPP	Avg.Score
Alpaca-GPT4(52K)
All Data	55.28	36.62	60.42	54.84	32.93	42.80	47.15
Random	52.35	33.28	60.48	53.65	37.8	38.80	46.06
IFD	40.35	46.02	63.21	56.08	41.46	46.40	48.92
SuperFiltering	53.86	39.73	60.13	56.79	40.85	45.00	49.39

Appendix

Table 11: Detailed results on the LLaMA3-8B-Instruct backbone.

	Reasoning		Knowledge		Coding		Avg.Score
	BBH	GSM8K	MMLU	TruthfulQA	HumanEval	MBPP	Avg.Score
Alpaca-GPT4(52K)
All Data	60.21	78.70	71.79	58.97	53.66	60.80	64.02
Random	62.17	81.20	73.04	60.56	61.59	62.00	66.76
IFD	63.23	83.85	73.55	59.20	60.98	62.60	67.24
SuperFiltering	63.71	82.56	73.29	60.73	61.59	66.40	68.05

Appendix

Table 12: Detailed results on the Qwen2.5-7B-Instruct backbone.

	Reasoning		Knowledge		Coding		Avg.Score
	BBH	GSM8K	MMLU	TruthfulQA	HumanEval	MBPP	Avg.Score
Alpaca-GPT4(52K)
All Data	29.67	5.38	29.53	44.91	17.07	21.80	24.73
Random	31.07	6.75	31.76	38.43	17.07	22.20	24.55
IFD	31.98	7.28	30.52	39.12	18.9	24.20	25.33
SuperFiltering	31.77	6.82	31.8	41.92	17.68	26.20	26.03

Appendix

Table 13: Detailed results under a small backbone (LLaMA3.2-1B-Instruct). We report reasoning, knowledge, and coding performance on three selection pools (Alpaca-GPT4, WizardLM, and CoT) with a fixed 10K budget. ADG consistently improves the average score over strong selection baselines, indicating that divergence-guided selection remains effective even for a lightweight model.

Explore similar work

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Apr 15, 2026Tatsuya Ichinose, Youmi Ma, Masanari Oi +2Instruction Tuning Synthetic Data Generation for Language Models
Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII
Token Utility Is Selection-Conditioned: Coupled Selection of Prompt Context and Response Supervision for Instruction Tuning
Sep 19, 2026Can Wu, Xinrui Chen, Ou Wu +1LLM Fine-Tuning Instruction-Tuning Data Selection
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Feb 16, 2026Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund +2Instruction-Tuning Data Selection Training Data Selection
Harvard University · MIT · Kempner Institute

Apr 15, 2026J/K move · Enter open · S save

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike+1

Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII

Instruction Tuning Synthetic Data Generation for Language Models Contrastive Decoding Language Model Distillation

Open paper AskPDF arXiv

Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.

Abstract

Explore similar work

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Token Utility Is Selection-Conditioned: Coupled Selection of Prompt Context and Response Supervision for Instruction Tuning

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)