Recent research demonstrates that pretrained large language models (LLMs) require minimal but carefully curated data for instruction-tuning. Building on LLMs, multi-modal models further contend with far larger data volumes, heterogeneous sources, and cross-modal alignment, amplifying both the necessity and the complexity of data selection. However, existing methods rely on deeply extracted embeddings, gradients or narrow quality scores that are time-consuming to compute and limited in expressiveness, frequently failing to surpass a simple random-sampling baseline. To overcome these limitations, we step back from complex embedding spaces and target the fundamental objective of instruction tuning: how and what knowledge should be elicited from the pretrained model and exposed to users. We leverage interaction Styles as a proxy for structural diversity, and decompose data quality into capability-specific Scores, yielding an explicit yet Rich (SSR) profile for each sample. We further cast the selection as submodular coverage maximization over the capability--style grid, providing an approximation guarantee. SSR consistently delivers superior performance across two distinct architecture stacks, LLaVA-1.5 (Vicuna + CLIP) and LLaVA-OneVision (Qwen2 + SigLIP), two large-scale data pools, six data budget settings, and 14 multi-modal benchmarks. To our knowledge, SSR is the first data selection method validated at million-scale, achieving 99.1% of full-data performance using only 30% of a 2.6M pool. Furthermore, the SSR profile models and selected subsets transfer training-free across data pools and model architectures with capability extensibility.
Figures & tables
Figure 1: Million-scale selection exposes the failure of prior methods. Failing to beat random sampling across 5%, 10%, and 30% budgets on the 2.6M data pool, existing methods expose a scalability gap that SSR closes with consistent gains.
Figure 2: Overview of the SSR framework compared to existing paradigms. Full fine-tuning indiscriminately processes all data (Top Right), while previous selection methods (Left) rely on model-centric signals (e.g. , latent embeddings). In contrast, our proposed SSR (Middle) adopts a data-centric profile: rich and explicit capability scores and an interaction style, which together drive submodular coverage maximization over the capability–style grid, yielding an efficient subset with superior performance, training-free transfer across data pools and target models, and seamless capability extensibility.
R scr
R sty
MMBench en-v1.1
MMStar
MMMU
MMVet
BLINK
MMT-Bench
MME
AI2D
ScienceQA
MathVista mini
> Rand
/FULL
Budget: 5%
Random
73.74
47.98
43.70
42.34
50.61
58.87
2004.50
73.07
81.52
45.47
-
89.29
SP GPT-4o
75.85
51.27
42.97
44.27
51.95
58.14
1940.27
73.61
81.46
45.00
5/10
90.14
ScrR GPT-4o
✓
74.23
49.07
43.07
41.56
52.08
57.56
1996.27
74.77
80.12
44.80
4/10
89.17
StyR GPT-4o
✓
76.48
52.52
43.85
41.58
52.46
58.89
1945.68
76.64
86.88
49.87
8/10
92.07
SSR Qwen2-VL
✓
✓
75.08
51.00
45.16
42.57
52.71
57.37
1955.78
74.74
84.88
48.90
8/10
91.37
Table 2: Richness drives generality, robustly across annotator sources. Both rich scores (ScrR) and styles (StyR) are necessary for accuracy and generality: dropping either degrades coverage, and the single-criterion, i.e. , quality SP(oor) baseline fails to cover the diverse axes of evaluation. {method} {source} indicates querying scores and styles from {source} for {method}. With either distilled proxies (SSR GPT-4o ) or direct open-weight model prompting (SSR Qwen2-VL , SSR LLaVA-OVSI ), SSR consistently beats prior baselines, confirming the efficacy stems from the explicit, rich SSR formulation that is robust to annotators.
Figure 4
Cambrian-7M
5%
10%
30%
Random / FULL
90.54
93.57
97.85
SSR (Ours) / FULL
92.40
94.35
98.19
Table 3: Zero-shot transfer to Cambrian-1-8B on the Cambrian-7M pool. LLaVA-OVSI-distilled SSR models profile the unseen pool without adaptation; /FULL (%) on the benchmark suite of Tab. 1 .
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Vision-Flan
LLaVA-1.5
LLaVA-OVSI
COINCIDE Lee et al. (2024)
K=5,000,τ=0.1
K=10,000,τ=0.1
K=20,000,τ=0.1
ICONS Wu et al. (2024)
warm-up 5% , p=0.2
warm-up 5% , p=0.2
warm-up 5% , p=0.2
PreSel Safaei et al. (2025)
k=5
k=5
k=5
PRISM Bi et al. (2025)
Layer 1
Layer 1
Layer 1
Appendix
Table 4: Baseline configurations. Settings reported in the original papers (Vision-Flan, LLaVA-1.5) and those used on LLaVA-OVSI; symbols follow each method’s original notation.
multi-modal Capabilities
Definitions
Examples
activity recognition
actions or behaviors of humans, animals, or objects
Fig. 15
causal reasoning
cause-and-effect relationships between events or variables to predict outcomes and explain phenomena
Fig. 16
humanities
history, literature, philosophy, art, and culture to understand human experiences and societal developments
Fig. 17
STEM knowledge
science, technology, engineering, and mathematics, chemistry, economics etc
Fig. 18
comparative analysis
compare multiple entities, concepts, or datasets to identify similarities, differences, and relationships
Fig. 19
data understanding
documents, tables, charts, graphics, infographics
Fig. 20
Appendix
Table 5: Criteria we recognize as the foundational pillars for developing vision perception and reasoning capabilities within MLLMs, and the interaction styles we identify from instructional multi-modal data.
Figure 8Figure 9
Figure 6: The mean absolute error of SSR scorer predictions against GPT-4o judgment over capabilities.
Figure 7: The precision and recall of SSR styler predictions against GPT-4o judgment.
Figure 8: Score distribution of capabilities across the LLaVA-OVSI dataset inferred by SSR.
Figure 9: Scalability along both quantity and capability axes. SSR consistently surpasses random across six data quantities (1% → 5% → 10% → 30% → 40% → 50%); adding OCR as a held-out capability (13 → 13 + OCR) extends the criteria set without degrading general performance, evidencing modular capability scalability. Legend: Random, SSR without OCR, SSR with OCR; 14 benchmarks.
Figure 10: Comparison of selection methods on average performance and total time consumption.
MMBench en-v1.1-test
MMMU test
MMT-Bench all
> Rand
/FULL
Budget: 5%
Random
73.45
40.37
59.98
-
96.32%
Deita
73.97
36.30
56.57
1/3
91.40%
COINCIDE
73.09
40.30
57.47
0/3
94.74%
ICONS
69.04
37.30
57.49
0/3
90.63%
SSR
75.84
41.30
60.10
3/3
98.15 %
Appendix
Table 6: Performance comparison on the benchmarks with online test splits conducted across varying budgets of 5%, 10% and 30% of LLaVA-OVSI. We highlight the best result in boldface and underline the result if it beats the random baseline. The column > Rand presents the number of benchmarks where the method exceeds random sampling, and /FULL compares the performance of sampled data with that of the FULL dataset.
Figure 11: Sample-level score distribution of the selected SSR-10% and SSR-30%.
Figure 12: Comparison of data style statistics between our selection and those of competitors.
Figure 13: Data source statistics of the original LLaVA-OVSI data pool (L) and our SSR-10% (R).
Figure 14: Comparison of data source statistics between our selection and those of competitors. For brevity, the figure displays only a small subset of the legends. Data sources of the same category are represented by shared color schemes, in accordance with Fig. 13 .
Figure 15: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the activity recognition capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 16: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the causal reasoning capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 17: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the humanities capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 18: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the stem knowledge capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 19: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the comparative analysis capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 20: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the data understanding capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 21: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the object spatial understanding capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 22: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the attribute identification capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 23: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the logical deduction capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 24: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the scene understanding capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 25: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the fine-grained recognition capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 26: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the language generation capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 27: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the in-context learning capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 28: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the optical character recognition capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Multimodal instruction tuning is widely used to adapt multimodal large language models (MLLMs), yet the large-scale image-text datasets it relies on are often highly redundant. Existing data selection methods are commonly tied to specific datasets, target models, or training states, requiring retraining or recomputation when transferred to new settings. We ask whether a selection signal can instead be learned once and reused across datasets and models. We propose OnceSelect, a reusable data selector trained once and directly applied to unseen datasets. OnceSelect encodes image-instruction pairs in a frozen joint multimodal space, clusters them into pseudo-labels capturing coarse semantic structure, and trains a lightweight selector using a fixed validation macro-accuracy criterion. Low-confidence samples under the learned partition are treated as informative candidates. As the selector is independent of the downstream MLLM, it can be transferred across datasets without retraining or model-dependent scoring, while selected subsets can be reused across model architectures. Using only 15% of LLaVA-625K, OnceSelect retains 99.2% of full-data aggregate performance across nine evaluation metrics. Without retraining, it achieves 103.4% and 102.5% relative performance on unseen Vision-Flan-186K and LRV-Sub-180K. The same 93.7K LLaVA subset further achieves 99.9-106.7% relative performance across Qwen3-VL and InternVL3 models without model-specific scoring or reselection. These results demonstrate efficient and reusable multimodal data selection across datasets and model architectures. Code is available at https://github.com/DMK041218/OnceSelect
Mingkang Dong, Muxin Pu, Hongyi Cai +5
Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China
Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets, models, and candidate pools. While no single method dominates, gradient-based representations paired with greedy round-robin selection often perform best on average at low budgets, but these gains diminish at larger budgets. Finally, we unify several existing selection algorithms as forms of approximate distance minimization between the selected subset and the query set, and support this view with new generalization bounds. More broadly, our findings provide critical insights and a foundation for more principled data selection in LLM fine-tuning. The code is available at https://github.com/dcml-lab/targeted-instruction-selection.
Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund +2
Data selection is a key component of efficient instruction tuning for large language models, as recent work has shown that data quality often matters more than data quantity. Accordingly, prior studies have introduced various multi-dimensional heuristics to evaluate and filter instruction data. However, most existing methods rely on static task-agnostic and model-agnostic weighting schemes, which overlook the varying requirements of specific downstream tasks and the differing pre-existing capabilities of models. In this paper, we propose a framework for learning multi-indicator weights that jointly adapts data selection to both the downstream task and the specific model. Our method identifies optimal weight configurations without full-scale fine-tuning by utilizing in-context learning (ICL) signals on compact tiny-validation sets. These signals serve as efficient performance proxies that ensure high-fidelity evaluation at minimal computational cost. Experiments across multiple benchmarks and model families, including Mistral, Qwen, and Llama, show that the approach achieves performance comparable to or exceeding full-dataset tuning while using only 30% of the training samples on GSM8K. Furthermore, our analysis reveals a trade-off between semantic diversity and logical complexity in reasoning tasks, highlighting the necessity of joint task-model adaptation.
Jingze Song, Zihao Chen, Wenqing Chen +1
School of Software Engineering, Sun Yat-sen University