Visual instruction tuning is crucial for improving vision-language large models (VLLMs). However, many samples can be solved via linguistic patterns or common-sense shortcuts, without genuine cross-modal reasoning, limiting the effectiveness of multimodal learning. Prior data selection methods often rely on costly proxy model training and focus on difficulty or diversity, failing to capture a sample's true contribution to vision-language joint reasoning. In this paper, we propose CVS, a training-free data selection method based on the insight that, for high-quality multimodal samples, introducing the question should substantially alter the model's assessment of answer validity given an image. CVS leverages a frozen VLLM as an evaluator and measures the discrepancy in answer validity with and without conditioning on the question, enabling the identification of samples that require vision-language joint reasoning while filtering semantic-conflict noise. Experiments on Vision-Flan and The Cauldron show that CVS achieves solid performance across datasets. On Vision-Flan, CVS outperforms full-data training by 3.5% and 4.8% using only 10% and 15% of the data, respectively, and remains robust on the highly heterogeneous Cauldron dataset. Moreover, CVS reduces computational cost by 17.3% and 44.4% compared to COINCIDE and XMAS.
Figures & tables
Figure 1: Overview of CVS. (a) Prompt construction. (b) CVS uses a frozen vision-language model to estimate conditional verdict shifts, filter noisy samples, and prioritize informative samples near the decision boundary.
Figure 2: Performance comparison between CVS and baseline methods on Vision-Flan and The Cauldron. CVS performs best at 10% and 15% sampling ratios on Vision-Flan, even surpassing full-data training, and remains robust across sampling ratios on The Cauldron.
Figure 3: Effect of CVS score ranges on model performance. The low-score strategy consistently yields the largest gains across all budgets.
Figure 4: Effect of evaluator architecture and scale on CVS performance.
Table 5
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Prompt templates for CVS computation.
Table 7
Method
Settings
CLIP-Score
high score selected
EL2N
medium score selected
SemDeDup
K:5,000
D2 Pruning
k:5,γr:0.4,γf:1.0
COINCIDE
K:5,000,τ:0.1
XMAS
K:1,000,T:7
Appendix
Table 6: Hyperparameter configurations.
Table 7: Main experimental results on Vision-Flan. We compare different data selection methods on Vision-Flan using 5%, 10%, and 15% data subsets. The best result among subset-selection methods in each column is highlighted in bold.
Table 8: Main experimental results on The Cauldron. We compare different data selection methods on The Cauldron using 10%, 20%, 30%, and 50% data subsets. The best result in each column is highlighted in bold.
Table 9: Detailed results on Vision-Flan for the effect of CVS score ranges. We evaluate different CVS score selection strategies (Low / High / No) on the Vision-Flan using data subsets of 5%, 10%, and 15%. The best result in each column is highlighted in bold.
Table 10: Detailed results on Vision-Flan under different evaluator architectures. We compare different evaluator architectures on Vision-Flan using data subsets of 5%, 10%, and 15%. Qwen, InternVL, and LLaVA-OV correspond to Qwen2.5-VL-7B-Instruct, InternVL3-8B, and LLaVA-OneVision-1.5-4B-Instruct, respectively. The best result among subset-selection methods in each column is highlighted in bold.
Table 11: Detailed results under different evaluator scales on Vision-Flan. We compare evaluators of different scales within the Qwen2.5-VL family, where CVS (7B) and CVS (3B) denote using Qwen2.5-VL-7B-Instruct and Qwen2.5-VL-3B-Instruct as evaluators, respectively. Experiments are conducted using 5%, 10%, and 15% data subsets. The best result in each column is highlighted in bold.
Table 12: Detailed results on Vision-Flan for robustness to the target model (Qwen2-VL-2B). With the evaluator fixed to Qwen2-VL-7B-Instruct, we replace the target model from LLaVA-1.5-7B to Qwen2-VL-2B to evaluate the robustness of CVS under target model changes. Experiments are conducted using data subsets of 5%, 10%, and 15%. The best result in each column is highlighted in bold.
Table 13: Detailed results on Vision-Flan for robustness to the target model (LLaVA-1.5-13B). With the evaluator fixed to Qwen2-VL-7B-Instruct, we replace the target model from LLaVA-1.5-7B to LLaVA-1.5-13B to evaluate the robustness of CVS under target model changes. Experiments are conducted using data subsets of 5%, 10%, and 15%. The best result in each column is highlighted in bold.
Table 14: Detailed results on Vision-Flan for visual anchoring. To evaluate the necessity of the visual anchoring term in CVS, we remove the image information I from the conditional probability P(Y∣I,A) in the denominator and compare the resulting variant with random sampling using data subsets of 5%, 10%, and 15%. The best result in each column is highlighted in bold.
Figure 6: Representative examples selected by the Low strategy.
Figure 7: Representative examples selected by the High strategy.
Figure 8: Representative examples selected by the No strategy.
Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China