Visual instruction tuning is crucial for improving vision-language large models (VLLMs). However, many samples can be solved via linguistic patterns or common-sense shortcuts, without genuine cross-modal reasoning, limiting the effectiveness of multimodal learning. Prior data selection methods often rely on costly proxy model training and focus on difficulty or diversity, failing to capture a sample's true contribution to vision-language joint reasoning. In this paper, we propose CVS, a training-free data selection method based on the insight that, for high-quality multimodal samples, introducing the question should substantially alter the model's assessment of answer validity given an image. CVS leverages a frozen VLLM as an evaluator and measures the discrepancy in answer validity with and without conditioning on the question, enabling the identification of samples that require vision-language joint reasoning while filtering semantic-conflict noise. Experiments on Vision-Flan and The Cauldron show that CVS achieves solid performance across datasets. On Vision-Flan, CVS outperforms full-data training by 3.5% and 4.8% using only 10% and 15% of the data, respectively, and remains robust on the highly heterogeneous Cauldron dataset. Moreover, CVS reduces computational cost by 17.3% and 44.4% compared to COINCIDE and XMAS.
Figures & tables
Figure 1: Overview of CVS. (a) Prompt construction. (b) CVS uses a frozen vision-language model to estimate conditional verdict shifts, filter noisy samples, and prioritize informative samples near the decision boundary.
Figure 2: Performance comparison between CVS and baseline methods on Vision-Flan and The Cauldron. CVS performs best at 10% and 15% sampling ratios on Vision-Flan, even surpassing full-data training, and remains robust across sampling ratios on The Cauldron.
Figure 3: Effect of CVS score ranges on model performance. The low-score strategy consistently yields the largest gains across all budgets.
Figure 4: Effect of evaluator architecture and scale on CVS performance.
Table 5
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Prompt templates for CVS computation.
Table 7
Method
Settings
CLIP-Score
high score selected
EL2N
medium score selected
SemDeDup
K:5,000
D2 Pruning
k:5,γr:0.4,γf:1.0
COINCIDE
K:5,000,τ:0.1
XMAS
K:1,000,T:7
Appendix
Table 6: Hyperparameter configurations.
Table 7: Main experimental results on Vision-Flan. We compare different data selection methods on Vision-Flan using 5%, 10%, and 15% data subsets. The best result among subset-selection methods in each column is highlighted in bold.
Table 8: Main experimental results on The Cauldron. We compare different data selection methods on The Cauldron using 10%, 20%, 30%, and 50% data subsets. The best result in each column is highlighted in bold.
Table 9: Detailed results on Vision-Flan for the effect of CVS score ranges. We evaluate different CVS score selection strategies (Low / High / No) on the Vision-Flan using data subsets of 5%, 10%, and 15%. The best result in each column is highlighted in bold.
Table 10: Detailed results on Vision-Flan under different evaluator architectures. We compare different evaluator architectures on Vision-Flan using data subsets of 5%, 10%, and 15%. Qwen, InternVL, and LLaVA-OV correspond to Qwen2.5-VL-7B-Instruct, InternVL3-8B, and LLaVA-OneVision-1.5-4B-Instruct, respectively. The best result among subset-selection methods in each column is highlighted in bold.
Table 11: Detailed results under different evaluator scales on Vision-Flan. We compare evaluators of different scales within the Qwen2.5-VL family, where CVS (7B) and CVS (3B) denote using Qwen2.5-VL-7B-Instruct and Qwen2.5-VL-3B-Instruct as evaluators, respectively. Experiments are conducted using 5%, 10%, and 15% data subsets. The best result in each column is highlighted in bold.
Table 12: Detailed results on Vision-Flan for robustness to the target model (Qwen2-VL-2B). With the evaluator fixed to Qwen2-VL-7B-Instruct, we replace the target model from LLaVA-1.5-7B to Qwen2-VL-2B to evaluate the robustness of CVS under target model changes. Experiments are conducted using data subsets of 5%, 10%, and 15%. The best result in each column is highlighted in bold.
Table 13: Detailed results on Vision-Flan for robustness to the target model (LLaVA-1.5-13B). With the evaluator fixed to Qwen2-VL-7B-Instruct, we replace the target model from LLaVA-1.5-7B to LLaVA-1.5-13B to evaluate the robustness of CVS under target model changes. Experiments are conducted using data subsets of 5%, 10%, and 15%. The best result in each column is highlighted in bold.
Table 14: Detailed results on Vision-Flan for visual anchoring. To evaluate the necessity of the visual anchoring term in CVS, we remove the image information I from the conditional probability P(Y∣I,A) in the denominator and compare the resulting variant with random sampling using data subsets of 5%, 10%, and 15%. The best result in each column is highlighted in bold.
Figure 6: Representative examples selected by the Low strategy.
Figure 7: Representative examples selected by the High strategy.
Figure 8: Representative examples selected by the No strategy.
Multimodal instruction tuning is widely used to adapt multimodal large language models (MLLMs), yet the large-scale image-text datasets it relies on are often highly redundant. Existing data selection methods are commonly tied to specific datasets, target models, or training states, requiring retraining or recomputation when transferred to new settings. We ask whether a selection signal can instead be learned once and reused across datasets and models. We propose OnceSelect, a reusable data selector trained once and directly applied to unseen datasets. OnceSelect encodes image-instruction pairs in a frozen joint multimodal space, clusters them into pseudo-labels capturing coarse semantic structure, and trains a lightweight selector using a fixed validation macro-accuracy criterion. Low-confidence samples under the learned partition are treated as informative candidates. As the selector is independent of the downstream MLLM, it can be transferred across datasets without retraining or model-dependent scoring, while selected subsets can be reused across model architectures. Using only 15% of LLaVA-625K, OnceSelect retains 99.2% of full-data aggregate performance across nine evaluation metrics. Without retraining, it achieves 103.4% and 102.5% relative performance on unseen Vision-Flan-186K and LRV-Sub-180K. The same 93.7K LLaVA subset further achieves 99.9-106.7% relative performance across Qwen3-VL and InternVL3 models without model-specific scoring or reselection. These results demonstrate efficient and reusable multimodal data selection across datasets and model architectures. Code is available at https://github.com/DMK041218/OnceSelect
Mingkang Dong, Muxin Pu, Hongyi Cai +5
Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China
Instruction tuning of large vision-language models (LVLMs) increasingly depends on massive multimodal corpora, yet these datasets contain samples with substantial redundancy, low visual dependency, and highly imbalanced coverage of multimodal reasoning behaviors. As a result, uniform subsampling or naive score-based selection often yields suboptimal training subsets. We introduce MAGIC, a training-free, forward-only coreset selection method designed to construct compact yet behaviorally faithful subsets for multimodal instruction tuning. MAGIC is built on three intrinsic signals extracted from a pretrained VLM: Multimodal Gain, which measures the likelihood improvement obtained from visual input; Bridging Relevance, which captures the sharpness of answer-token grounding over visual tokens; and Skill-Neuron Signatures, which characterize the functional computation elicited by each sample via top-activated feed-forward neurons. MAGIC combines these signals in a three-stage pipeline: filtering low-gain examples, ranking candidates by a normalized quality objective, and performing bucket-wise budget allocation over discrete neuron signatures to preserve latent multimodal skill coverage. This formulation avoids backpropagation, auxiliary selector training, and expensive clustering in continuous activation spaces, while remaining efficient and easily deployable in existing VLMs. Across LLaVA-665K and Vision-Flan datasets, and transfer settings to large target models, LLaVA-1.5-7B and -13B, MAGIC consistently improves over strong baselines under matched 20% budgets: it achieves 100.3% relative performance to full finetuning on LLaVA-665K and 101.6% relative performance on Vision-Flan-186K, while yielding a 73.7% reduction in wall-clock run time.
Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.
Matteo Farina, Vishaal Udandarao, Thao Nguyen +33
University of Trento · 2Tübingen AI Center, University of Tübingen · University of Cambridge +12