Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets, models, and candidate pools. While no single method dominates, gradient-based representations paired with greedy round-robin selection often perform best on average at low budgets, but these gains diminish at larger budgets. Finally, we unify several existing selection algorithms as forms of approximate distance minimization between the selected subset and the query set, and support this view with new generalization bounds. More broadly, our findings provide critical insights and a foundation for more principled data selection in LLM fine-tuning. The code is available at https://github.com/dcml-lab/targeted-instruction-selection.
Figures & tables
Figure 1: Disentangled view of targeted instruction selection. First, the query set (stars) and candidate pool (dots) are encoded as data representations. Then, for a given budget, using the data representations for the query and candidates, we perform targeted selection (denoted by the dotted line) using a selection algorithm such as greedy round-robin.
Figure 2: Query loss vs. subset-query distance quantile. We stratify candidates into 10 distance quantiles (1 = closest, 10 = farthest) using each representation, select 500 examples per quantile using the RR selection algorithm, and train the Llama 2 7B model. We report query-set cross-entropy loss and Spearman correlation per target task. LESS (RR) exhibits a strong monotonic increase in loss with distance (high positive Spearman correlation), whereas RDS+ (RR) and EMBED (RR) show weak or inconsistent correlations.
Figure 3: Downstream performance vs. subset-query distance quantile. Using the same quantile construction and training protocol as Figure 2 , we evaluate downstream task performance across distance quantiles and report Spearman correlation per target task. LESS (RR) shows a strong negative correlation across most target tasks, while RDS+ (RR) and EMBED (RR) exhibit weaker, less consistent trends.
Figure 4: Query loss vs. budget for different data representations (fixed selection algorithm). Using greedy round-robin selection and the query-candidate pool similarity, we select subsets of size B∈{500,1000,2500,5000,10000} , train Llama 2 7B on them, and report average cross entropy loss averaged across three seeds and the standard error. Random averages over three uniformly sampled subsets from the candidate pool. LESS (RR) achieves the lowest loss across target tasks, while RDS+ (RR) and EMBED (RR) can underperform Random at larger budgets.
Figure 5: Downstream performance vs. budget for different data representations (fixed selection algorithm). With the same greedy round-robin selection and budgets as Figure 4 , we report the downstream performance for different data representations averaged across three random seeds and the standard error. LESS (RR) performs best on BBH, TyDiQA, and MMLU-Pro, whereas RDS+ (RR) performs the best on GSM8K and is competitive with Random on Codex.
Figure 6: Query loss vs. budget for different selection algorithms (fixed data representation). Using LESS representations and the query-candidate pool cosine similarity (or distance), we select subsets of size B∈{500,1000,2500,5000,10000} with each selection algorithm, train Llama 2 7B on them, and report average cross entropy loss on the query set averaged across three seeds and the standard error. Random averages over three uniformly sampled subsets from the candidate pool. UOT achieves the lowest loss on three of the five datasets and remains competitive on the others, while DG often underperforms, yielding the highest loss on three datasets.
Figure 7: Downstream performance vs. budget for different selection algorithms (fixed data representation). With the same data representation and the budgets as Figure 6 , we report downstream performance for different selection algorithms averaged across three seeds and the standard error. RR tends to perform best at smaller budgets, whereas UOT and KNN-KDE perform better at larger budgets; DG consistently underperforms across three of the five datasets.
Figure 8: Convergence of LESS variants toward random sampling as budget increases. We plot the loss gap between random sampling and the LESS variants as a function of budget B (log-scale) on MMLU-Pro. The gray reference line indicates the B−1/d decay predicted by Theorem 6.2 . We report the average difference in loss across one seed of LESS and three randomly sampled multisets of size B , along with the standard error. For visual comparison of decay rates, we apply a constant offset to each LESS curve so that all methods start at B0−1/d where B0=500 and d=8192 .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Existing Work
Data Representation
Similarity
Selection Algorithm
Candidate Pool
Budget
Xia et al. (2024)
LESS
Cosine sim.
Doubly Greedy (DG)
Tulu V1 (Subset)
13,533
Liu et al. (2024b)
LESS
L2 dist.
KNN-KDE, KNN-Unif.
Tulu V1 (Subset)
Multiple
Ivison et al. (2025)
RDS+
Cosine sim.
Round-Robin (RR)
Tulu V2
10,000
Appendix
Table 1: Overview of the fragmented literature on targeted instruction selection. We summarize prior work and highlight differences in data representation, similarity between data representations, selection algorithm, candidate pool, and selection budget, which prevent us from systematically understanding which of these key factors contributes to performance. Note that Xia et al. (2024) uses 5% of the candidate pool as the budget, resulting in 13,533 samples whereas Liu et al. (2024b) uses 0.5%, 1.0%, and 5.0% of the candidate pool as the budget.
Target Task
Num. Query Set Samples
Num. Test Set Samples
Target Task Evaluation
BBH (BIG-Bench Hard)
81
6,511
3 -Shot
Codex
16
148
0 -Shot
GSM8K
8
1,319
8 -Shot
TyDiQA
9
5,077
1 -Shot
MMLU Pro
70
12,032
0 -Shot
Appendix
Table 2: Statistics of target tasks used in our experiments. The query set is used to select samples from the candidate pool to train the base model, and then evaluated on the test set from the target task. Each target task is evaluated using 0 to N few-shot in-context examples, and we reuse the query set samples as these few-shot examples if mentioned in Appendix B .
Hyperparameters
Values
Learning rate
2e-5
Learning rate scheduler
linear
Number of epochs
2
Warmup ratio
0.03
Optimizer
AdamW
Adam betas
(0.9, 0.999)
Appendix
Table 3: Hyperparameters used to train the base models on the selected instructions.
Figure 9: Query loss across subset-query distance sub-quantiles and Spearman correlation. We further stratify the first distance quantile from Section 5.1 into 10 sub-quantiles (1 = closest, 10 = farthest), select 500 examples per sub-quantile, and train the Llama 2 7B model. We report loss on the query set and Spearman correlation per dataset. LESS (RR) maintains a strong monotonic increase in loss with distance (high Spearman correlation), whereas RDS+ (RR) and EMBED (RR) show weak or inconsistent correlations.
Figure 10: Downstream performance across subset-query distance sub-quantiles and Spearman correlation. Using the same sub-quantile construction and training protocol as Figure 9 , we evaluate downstream task performance across sub-quantiles and report Spearman correlation per dataset (more negative is better). LESS (RR) shows a stronger negative correlation on average, but the performance differences are small, suggesting that many subsets within the closest quantile result in similar downstream performance.
Figure 11: Average token length of the query set and the selected subsets with different data representations. We find that LESS (RR) is biased towards shorter sequences, whereas RDS+ (RR) and EMBED (RR) select subsets with longer sequences.
Figure 12: Jaccard index between selected subsets created using different data representations. We find that model-based representations (RDS+ and EMBED) have a higher Jaccard index compared to LESS.
Figure 13: Cheaper proxies for LESS (fixed selection algorithm). With the same greedy round-robin selection procedure and the budgets from Section 5.2 , we report downstream performance of Llama 2 7B when LESS representations are computed using proxy models, averaged across three seeds and the standard error. SmolLM-135M and SmolLM2-135M consistently match or outperform the Random baseline across target tasks, whereas Pythia-160M often matches Random on several tasks. Llama 3.2 3B matches or outperforms LESS computed with Llama 2 7B on multiple target tasks, suggesting a trade-off between proxy and target model size when approximating instruction selection.
Figure 14: KNN-Uniform and KNN-KDE with L2 distance (fixed data representation). With the same Llama 2 7B LESS representations and the budgets from Section 5.3 , we report downstream performance for KNN-Uniform and KNN-KDE when distances are computed with L2 (following Liu et al. (2024b) ), averaged across three seeds and the standard error. We compare against cosine distance variants and find similar performance trends across budgets.
Figure 15: Ablation experiments with Llama 3.2 3B.
Figure 16: Ablation experiments with SmolLM3 3B.
Figure 17: Ablation experiments with Qwen3 4B.
Figure 18: Ablation experiments with Olmo 3 7B.
Figure 19: Multitask instruction selection experiments with Llama 3.2 3B
Figure 20: Ablation experiments with Llama 3.2 3B using Dolci Instruct as the candidate pool.
Figure 21: Ablation experiments with Llama 3.2 3B using Dolci Instruct as the candidate pool.
Figure 22: Ablation experiments with SmolLM3 3B using Dolci Instruct as the candidate pool.
Figure 23: Ablation experiments with Qwen3 4B using Dolci Instruct as the candidate pool.
Figure 24: Ablation experiments with Olmo 3 7B using Dolci Instruct as the candidate pool.
Instruction fine-tuning is employed to enhance the instruction-following ability of large language models (LLMs). As the amount of instruction fine-tuning data increases, selecting the optimal core set becomes particularly important. However, ensuring the diversity of the core set remains a significant challenge. Existing methods predominantly distinguish different training data based on the text features themselves, decoupled from LLMs' own understanding and representation of the data. To address this issue, we propose a Model-Aware Diverse Core Set Selection method, which distinguishes data features based on the neural activation states during LLM inference. This approach serves as an efficient instantiation of coverage-based selection using model-intrinsic activation features to ensure the diversity in the core set. We extensively evaluate our method on six benchmarks that cover five distinct tasks. In our method, the core set selected by the 3B-parameter LLM performs effectively when utilized to fine-tune larger models with 7B, 8B, and 13B parameters. Experimental results on the Alpaca-GPT4 dataset, which comprises 52K instruction-response pairs, show that the core set, sized at 15% of the original dataset and selected by Llama-3.2-3B-Instruct, achieves an average improvement of 2.5% when fine-tuning four larger base models compared with training on the full dataset. The experimental results demonstrate that our method enhances model performance on multiple downstream tasks while reducing data requirements.
Yi Bai, Wenhao Zhang, Yao Chen +3
Shandong University, Qingdao, China · Inspurcloud, Jinan, China
With Large Language Model (LLM) pre-training and fine-tuning shifting its focus from data volume to data quality, quality data selection has emerged as a critical research topic. Existing online data selection methods for LLM training are typically "batch-constrained", limiting optimization to local utility within random batches. To overcome this, we propose GAIA (Global Adaptive Instruction tuning via GAussian processes), a framework that formulates data valuation as a global estimation process. GAIA employs Gaussian Process regression to model continuous utility manifolds across the semantic space, utilizing an adaptive strategy fusion mechanism to dynamically prioritize high-utility samples. By casting the strategy-posterior update as an instance of the classical fixed-share Hedge framework for tracking the best expert, we inherit a dynamic-regret guarantee that characterizes GAIA's robustness under non-stationary quality scores during training. Empirical evaluations on three datasets demonstrate that GAIA significantly outperforms state-of-the-art baselines like \greats, establishing our method as a scalable and robust solution for efficient instruction tuning.
Instruction tuning relies on large instruction-response corpora whose quality and composition strongly affect downstream performance. We propose Answer Divergence-Guided Selection (ADG), which selects instruction data based on the geometric structure of multi-sample outputs. ADG draws several high-temperature generations per instruction, maps responses into an embedding space, and computes an output divergence score that jointly encodes dispersion magnitude and shape anisotropy. High scores correspond to instructions whose answers are both far apart and multi-modal, rather than clustered paraphrases along a single direction. Across two backbones and three public instruction pools, fine-tuning on only 10K ADG-selected examples consistently outperforms strong selectors on six benchmarks spanning reasoning, knowledge, and coding. Analyses further show that both dispersion magnitude and shape anisotropy are necessary, supporting answer divergence as a practical signal for instruction data selection. Code and appendix are included in the supplementary materials.
Bo Li, Mingda Wang, Shikun Zhang +1
National Engineering Research Center for Software Engineering, Peking University · School of Computer Science, Peking University · PKU-CMCC(Hubei) Joint Research Lab for LLM Industrial Applications +1