Multimodal large language models (MLLMs) perform well on many vision-language tasks but often struggle with vision-centric problems that require fine-grained visual reasoning. Recent evidence suggests that this limitation arises not from weak visual representations, but from under-utilization of visual information during instruction tuning, where many tasks can be partially solved using language priors alone. We propose a simple and lightweight approach that augments visual instruction tuning with a small number of visually grounded self-supervised tasks expressed as natural language instructions. By reformulating classical self-supervised pretext tasks, such as rotation prediction, color matching, and cross-view correspondence, as image-instruction-response triplets, we introduce supervision that cannot be solved without relying on visual evidence. Our approach requires no human annotations, no architectural modifications, and no additional training stages. Across multiple models, training regimes, and benchmarks, injecting only a small fraction (3-10%) of such visually grounded instructions consistently improves performance on vision-centric evaluations. Our findings highlight instruction tuning with visually grounded SSL tasks as a powerful lever for improving visual reasoning in MLLMs through simple adjustments to the training data distribution. Code available at: https://github.com/sirkosophia/V-GIFT
Figures & tables
Figure 1: V isually G rounded I nstruction F ine- T uning V-GIFT . We enhance visual instruction tuning by injecting visually grounded self-supervised tasks as additional instruction-following examples sampled from the instruction-tuning data (left; rotation prediction shown). This simple modification encourages better use of visual information and yields consistent gains on vision-centric benchmarks (right; CVB-2D, POPE, MMStar, BLINK) across model variants.
Figure 2: Visually grounded instruction-following tasks reformulated from self-supervised learning (SSL) pretext tasks. (a) Rotation prediction: the model must recognize object orientations and relate it to canonical poses. (b) Point-wise colorization: the model must match grayscale points to their original colors, requiring fine-grained visual discrimination, spatial grounding, and reasoning over local and global image context. (c) Point correspondence: the model must identify corresponding points across views, requiring cross-view feature matching and spatial reasoning. Collectively, these tasks compel the model to integrate local visual cues with global structure and rely on visual evidence rather than language priors.
Model
Method
CVB-2D
POPE
MMStar
BLINK
Avg.
LLaVA-1.5-Vicuna-7B
Baseline
55.9
87.0
33.5
38.2
53.6
V-GIFT
58.5 (+2.6)
87.2 (+0.2)
34.6 (+1.1)
37.8 (-0.4)
54.5 (+0.9)
LLaVA-1.5-Qwen2.5-7B
Baseline
61.1
87.2
42.7
40.1
57.8
V-GIFT
62.2 (+1.1)
87.7 (+0.5)
43.2 (+0.5)
41.8 (+1.7)
58.7 (+0.9)
LLaVA-OneVision-1.5
Baseline
70.0
88.6
55.3
48.8
65.7
V-GIFT
71.0 (+1.0)
88.9 (+0.3)
55.5 (+0.2)
52.2 (+3.4)
66.9 (+1.2)
Table 1: Main results of incorporating visually grounded SSL tasks on three MLLM models (LLaVA-1.5-Vicuna-7B, LLaVA-1.5-Qwen2.5-7B, LLaVA-OneVision-1.5). Performance is reported on vision-centric benchmarks (CVB-2D, POPE, MMStar, BLINK). Numbers in parentheses indicate improvement over the baseline.
Model
Method
CVB-2D
POPE ∗
MMStar
BLINK
Avg.
LLaVA-1.5-Qwen2.5-7B (LoRA)
Baseline
59.9
87.7
42.6
40.0
57.4
V-GIFT
63.8 (+3.9)
88.5 (+0.8)
43.7 (+1.1)
43.3 (+3.3)
59.6 (+2.2)
VIRAL [ 79 ]
60.5
88.3
39.2
–
–
Table 2: Effect of visually grounded SSL tasks with LoRA training on LLaVA-1.5-Qwen2.5-7B. Performance is reported on vision-centric benchmarks (CVB-2D, POPE, MMStar, BLINK). Numbers in parentheses indicate improvement over the baseline. Baseline results are reproduced, while VIRAL results are reported from [ 79 ] . POPE ∗ represents average accuracy across the “random” and “popular” subsets.
Model
Method
MathVista
OCRBench
RealWorldQA
LLaVA-1.5-Qwen2.5-7B
Baseline
12.2
312.0
56.4
V-GIFT
15.3
311.3
59.0
LLaVA-1.5-Vicuna-7B
Baseline
12.9
319.3
54.2
V-GIFT
12.8
318.3
55.4
LLaVA-OneVision-1.5
Baseline
22.9
627.0
66.1
V-GIFT
22.6
634.0
66.4
Table 3: General benchmark results. Comparing V-GIFT to baselines for LLaVA-1.5-Vicuna-7B, LLaVA-1.5-Qwen2.5-7B, and LLaVA-OneVision-1.5. Performance is reported on MathVista, OCRBench, and RealWorldQA.
SSL task
Benchmark
Rot.
Col.
Corr.
CVB-2D
POPE
MMStar
BLINK
Avg.
61.1
87.2
42.7
40.1
57.8
✓
62.4
87.2
42.5
40.7
58.2
✓
62.3
87.5
41.8
41.7
58.3
✓
61.9
87.3
41.7
42.0
58.2
✓
✓
✓
62.2
87.7
43.2
41.8
58.7
Table 4: Impact of individual and combined SSL pretext tasks on LLaVA-1.5-Qwen2.5-7B ( ρ=1% per task). Each task, rotation (Rot.), colorization (Col.), and correspondence (Corr.), independently improves the average performance over the baseline, while combining all three yields the strongest and most consistent gains across benchmarks.
Figure 3: Effect of the SSL injection ratio ρ on vision-centric instruction-following performance for LLaVA-1.5-Qwen2.5-7B (left) and LLaVA-OneVision-1.5 (right).
Training Setup
CVB-2D
POPE
MMStar
BLINK
Avg.
LLaVA-OneVision-1.5
Baseline (100% instruct)
70.0
88.6
55.3
48.8
65.7
Baseline (+3% instruct only)
68.2
89.0
55.4
49.3
65.6
V-GIFT (+3% SSL)
71.0
88.9
55.5
52.2
66.9
Table 5: Controlling for training compute. We compare our method against baselines trained with the same proportional increase in instruction tuning iterations, but without SSL tasks. Increasing compute alone does not improve performance; gains arise only when the additional data consists of visually grounded SSL tasks.
SSL Injection Stage
CVB-2D
POPE
MMStar
BLINK
Avg.
Baseline
61.1
87.2
42.7
40.1
57.8
SSL before IT
60.1
87.1
42.6
40.8
57.7
SSL during IT (ours)
62.2
87.7
43.2
41.8
58.7
SSL after IT
36.5
50.2
39.5
33.2
39.9
Table 6: Effect of SSL injection stage on LLaVA-1.5-Qwen2.5-7B. Training after the instruction tuning is done using LoRA to avoid catastrophic forgetting. Improvements are obtained only when SSL tasks are integrated during instruction tuning.
SSL Image Source
CVB-2D
POPE
MMStar
BLINK
Avg.
Baseline (no SSL)
61.1
87.2
42.7
40.1
57.8
Single image (augmented views)
62.7
86.9
42.5
41.5
58.4
COCO (reused images)
62.4
87.2
42.5
40.7
58.2
Table 7: Effect of SSL image source (rotation prediction, ρ=1% ) on LLaVA-1.5-Qwen2.5-7B. Both COCO and a single-image source yield improvements over the baseline, indicating that visually grounded supervision, rather than dataset scale, drives the gains.
Model
Method
CVB-2D
MMStar
LLaVA-1.5
Baseline
0.1238
0.1426
LLaVA-1.5
V-GIFT
0.1368
0.1430
Table 8: Mean TVI ↑ [ 53 ] comparison across datasets on LLaVA-1.5 Vicuna-7B.
Figure 4: Attention map from the Baseline (LLaVA-1.5-Vicuna-7B) and V-GIFT on CV-Bench2D examples. V-GIFT produces more focused and better localized attention on task-relevant objects.
Figure 5: Qualitative examples. We present a few qualitative examples comparing LLaVA-1.5 Qwen-2.5-7B baseline against V-GIFT . Our SSL-inspired tasks yield improvements on the variety of vision oriented skills such as counting, multi-view reasoning and visual reasoning.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Examples of the visually grounded self-supervised tasks used during training: colorization point matching, point correspondence, and rotation prediction.
Injection ratio ρ
CVB-2D
POPE
MMStar
BLINK
Avg.
1%
69.4
88.5
55.5
49.9
65.8
3%
71.0
88.9
55.5
52.2
66.9
5%
69.4
88.3
54.7
52.3
66.2
10%
69.5
88.3
54.1
53.7
66.4
30%
68.9
88.2
55.3
52.9
66.3
Appendix
Table 9: Effect of the SSL injection ratio ρ on vision-centric instruction-following performance for LLaVA-OneVision-1.5.
Reinforcement learning (RL) with verifiable rewards (RLVR) has demonstrated the great potential of enhancing the reasoning abilities in multimodal large language models (MLLMs). However, the reliance on language-centric priors and expensive manual annotations prevents MLLMs' intrinsic visual understanding and scalable reward designs. In this work, we introduce SSL-R1, a generic self-supervised RL framework that derives verifiable rewards directly from images. To this end, we revisit self-supervised learning (SSL) in visual domains and reformulate widely-used SSL tasks into a set of verifiable visual puzzles for RL post-training, requiring neither human nor external model supervision. Training MLLMs on these tasks substantially improves their performance on multimodal understanding and reasoning benchmarks, highlighting the potential of leveraging vision-centric self-supervised tasks for MLLM post-training. We think this work will provide useful experience in devising effective self-supervised verifiable rewards to enable RL at scale. Project page: https://github.com/Jiahao000/SSL-R1.
The effectiveness of multimodal instruction tuning depends not only on dataset scale, but critically on whether training samples genuinely require visual reasoning. However, existing instruction datasets often contain a substantial portion of visually redundant samples (solvable from text alone), as well as multimodally misaligned supervision that can degrade learning. To address this, we propose VisNec (Visual Necessity Score), a principled data selection framework that measures the marginal contribution of visual input during instruction tuning. By comparing predictive loss with and without visual context, VisNec identifies whether a training instance is vision-critical, redundant, or misaligned. To preserve task diversity, we combine VisNec with semantic clustering and select high-necessity samples within each cluster. Across 10 downstream benchmarks, training on only 15% of the LLaVA-665K dataset selected by VisNec achieves 100.2% of full-data performance. On the smaller Vision-Flan-186K dataset, our selection not only further reduces data size but also surpasses full-data training by 15.8%. These results demonstrate that measuring and leveraging visual necessity provides an effective solution for both efficient and robust multimodal instruction tuning. Codes and selected subsets will be released upon acceptance.
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences