cs.CVMar 17, 2025
SaveAn Embarrassingly Simple Data Selection Strategy for Million-Scale Multi-Modal Instruction-Tuning
Organizations: Tsinghua University · BNRist · Bytedance (work as an intern)
Abstract
Recent research demonstrates that pretrained large language models (LLMs) require minimal but carefully curated data for instruction-tuning. Building on LLMs, multi-modal models further contend with far larger data volumes, heterogeneous sources, and cross-modal alignment, amplifying both the necessity and the complexity of data selection. However, existing methods rely on deeply extracted embeddings, gradients or narrow quality scores that are time-consuming to compute and limited in expressiveness, frequently failing to surpass a simple random-sampling baseline. To overcome these limitations, we step back from complex embedding spaces and target the fundamental objective of instruction tuning: how and what knowledge should be elicited from the pretrained model and exposed to users. We leverage interaction Styles as a proxy for structural diversity, and decompose data quality into capability-specific Scores, yielding an explicit yet Rich (SSR) profile for each sample. We further cast the selection as submodular coverage maximization over the capability--style grid, providing an approximation guarantee. SSR consistently delivers superior performance across two distinct architecture stacks, LLaVA-1.5 (Vicuna + CLIP) and LLaVA-OneVision (Qwen2 + SigLIP), two large-scale data pools, six data budget settings, and 14 multi-modal benchmarks. To our knowledge, SSR is the first data selection method validated at million-scale, achieving 99.1% of full-data performance using only 30% of a 2.6M pool. Furthermore, the SSR profile models and selected subsets transfer training-free across data pools and model architectures with capability extensibility.
Figures & tables
Figure 1: Million-scale selection exposes the failure of prior methods. Failing to beat random sampling across 5%, 10%, and 30% budgets on the 2.6M data pool, existing methods expose a scalability gap that SSR closes with consistent gains.
Figure 2: Overview of the SSR framework compared to existing paradigms. Full fine-tuning indiscriminately processes all data (Top Right), while previous selection methods (Left) rely on model-centric signals (e.g. , latent embeddings). In contrast, our proposed SSR (Middle) adopts a data-centric profile: rich and explicit capability scores and an interaction style, which together drive submodular coverage maximization over the capability–style grid, yielding an efficient subset with superior performance, training-free transfer across data pools and target models, and seamless capability extensibility.
| R scr | R sty | MMBench en-v1.1 | MMStar | MMMU | MMVet | BLINK | MMT-Bench | MME | AI2D | ScienceQA | MathVista mini | Rand | /FULL | |
| Budget: 5% | ||||||||||||||
| Random | 73.74 | 47.98 | 43.70 | 42.34 | 50.61 | 58.87 | 2004.50 | 73.07 | 81.52 | 45.47 | - | 89.29 | ||
| SP GPT-4o | 75.85 | 51.27 | 42.97 | 44.27 | 51.95 | 58.14 | 1940.27 | 73.61 | 81.46 | 45.00 | 5/10 | 90.14 | ||
| ScrR GPT-4o | ✓ | 74.23 | 49.07 | 43.07 | 41.56 | 52.08 | 57.56 | 1996.27 | 74.77 | 80.12 | 44.80 | 4/10 | 89.17 | |
| StyR GPT-4o | ✓ | 76.48 | 52.52 | 43.85 | 41.58 | 52.46 | 58.89 | 1945.68 | 76.64 | 86.88 | 49.87 | 8/10 | 92.07 | |
| SSR Qwen2-VL | ✓ | ✓ | 75.08 | 51.00 | 45.16 | 42.57 | 52.71 | 57.37 | 1955.78 | 74.74 | 84.88 | 48.90 | 8/10 | 91.37 |
Table 2: Richness drives generality, robustly across annotator sources. Both rich scores (ScrR) and styles (StyR) are necessary for accuracy and generality: dropping either degrades coverage, and the single-criterion, i.e. , quality SP(oor) baseline fails to cover the diverse axes of evaluation. {method} {source} indicates querying scores and styles from {source} for {method}. With either distilled proxies (SSR GPT-4o ) or direct open-weight model prompting (SSR Qwen2-VL , SSR LLaVA-OVSI ), SSR consistently beats prior baselines, confirming the efficacy stems from the explicit, rich SSR formulation that is robust to annotators.
| Cambrian-7M | 5% | 10% | 30% |
| Random / FULL | 90.54 | 93.57 | 97.85 |
| SSR (Ours) / FULL | 92.40 | 94.35 | 98.19 |
Table 3: Zero-shot transfer to Cambrian-1-8B on the Cambrian-7M pool. LLaVA-OVSI-distilled SSR models profile the unseen pool without adaptation; /FULL (%) on the benchmark suite of Tab. 1 .
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Vision-Flan | LLaVA-1.5 | LLaVA-OVSI |
| COINCIDE Lee et al. (2024) | |||
| ICONS Wu et al. (2024) | warm-up , | warm-up , | warm-up , |
| PreSel Safaei et al. (2025) | |||
| PRISM Bi et al. (2025) | Layer 1 | Layer 1 | Layer 1 |
Table 4: Baseline configurations. Settings reported in the original papers (Vision-Flan, LLaVA-1.5) and those used on LLaVA-OVSI; symbols follow each method’s original notation.
| multi-modal Capabilities | Definitions | Examples |
| activity recognition | actions or behaviors of humans, animals, or objects | Fig. 15 |
| causal reasoning | cause-and-effect relationships between events or variables to predict outcomes and explain phenomena | Fig. 16 |
| humanities | history, literature, philosophy, art, and culture to understand human experiences and societal developments | Fig. 17 |
| STEM knowledge | science, technology, engineering, and mathematics, chemistry, economics etc | Fig. 18 |
| comparative analysis | compare multiple entities, concepts, or datasets to identify similarities, differences, and relationships | Fig. 19 |
| data understanding | documents, tables, charts, graphics, infographics | Fig. 20 |
Table 5: Criteria we recognize as the foundational pillars for developing vision perception and reasoning capabilities within MLLMs, and the interaction styles we identify from instructional multi-modal data.
Figure 6: The mean absolute error of SSR scorer predictions against GPT-4o judgment over capabilities.
Figure 7: The precision and recall of SSR styler predictions against GPT-4o judgment.
Figure 8: Score distribution of capabilities across the LLaVA-OVSI dataset inferred by SSR.
Figure 9: Scalability along both quantity and capability axes. SSR consistently surpasses random across six data quantities (1% 5% 10% 30% 40% 50%); adding OCR as a held-out capability (13 13 + OCR) extends the criteria set without degrading general performance, evidencing modular capability scalability. Legend: Random, SSR without OCR, SSR with OCR; 14 benchmarks.
Figure 10: Comparison of selection methods on average performance and total time consumption.
| MMBench en-v1.1-test | MMMU test | MMT-Bench all | Rand | /FULL | |
| Budget: 5% | |||||
| Random | 73.45 | 40.37 | 59.98 | - | 96.32% |
| Deita | 73.97 | 36.30 | 56.57 | 1/3 | 91.40% |
| COINCIDE | 73.09 | 40.30 | 57.47 | 0/3 | 94.74% |
| ICONS | 69.04 | 37.30 | 57.49 | 0/3 | 90.63% |
| SSR | 75.84 | 41.30 | 60.10 | 3/3 | 98.15 % |
Table 6: Performance comparison on the benchmarks with online test splits conducted across varying budgets of 5%, 10% and 30% of LLaVA-OVSI. We highlight the best result in boldface and underline the result if it beats the random baseline. The column Rand presents the number of benchmarks where the method exceeds random sampling, and /FULL compares the performance of sampled data with that of the FULL dataset.
Figure 11: Sample-level score distribution of the selected SSR-10% and SSR-30%.
Figure 12: Comparison of data style statistics between our selection and those of competitors.
Figure 13: Data source statistics of the original LLaVA-OVSI data pool (L) and our SSR-10% (R).
Figure 14: Comparison of data source statistics between our selection and those of competitors. For brevity, the figure displays only a small subset of the legends. Data sources of the same category are represented by shared color schemes, in accordance with Fig. 13 .
Figure 15: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the activity recognition capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 16: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the causal reasoning capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 17: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the humanities capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 18: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the stem knowledge capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 19: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the comparative analysis capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 20: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the data understanding capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 21: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the object spatial understanding capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 22: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the attribute identification capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 23: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the logical deduction capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 24: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the scene understanding capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 25: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the fine-grained recognition capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 26: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the language generation capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 27: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the in-context learning capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
Figure 28: Randomly selected samples from the LLaVA-OVSI data pool within each score group for the optical character recognition capability, ranging from 5 to 0, as evaluated by GPT-4o, along with corresponding score explanations and SSR predictions.
More in this topic
Instruction-Tuning Data Selection
Instruction-tuning data selection chooses and organizes training examples to improve model behavior while reducing fine-tuning cost.
See all 45 papers