Multimodal large language models (MLLMs) perform well on many vision-language tasks but often struggle with vision-centric problems that require fine-grained visual reasoning. Recent evidence suggests that this limitation arises not from weak visual representations, but from under-utilization of visual information during instruction tuning, where many tasks can be partially solved using language priors alone. We propose a simple and lightweight approach that augments visual instruction tuning with a small number of visually grounded self-supervised tasks expressed as natural language instructions. By reformulating classical self-supervised pretext tasks, such as rotation prediction, color matching, and cross-view correspondence, as image-instruction-response triplets, we introduce supervision that cannot be solved without relying on visual evidence. Our approach requires no human annotations, no architectural modifications, and no additional training stages. Across multiple models, training regimes, and benchmarks, injecting only a small fraction (3-10%) of such visually grounded instructions consistently improves performance on vision-centric evaluations. Our findings highlight instruction tuning with visually grounded SSL tasks as a powerful lever for improving visual reasoning in MLLMs through simple adjustments to the training data distribution. Code available at: https://github.com/sirkosophia/V-GIFT
Figures & tables
Figure 1: V isually G rounded I nstruction F ine- T uning V-GIFT . We enhance visual instruction tuning by injecting visually grounded self-supervised tasks as additional instruction-following examples sampled from the instruction-tuning data (left; rotation prediction shown). This simple modification encourages better use of visual information and yields consistent gains on vision-centric benchmarks (right; CVB-2D, POPE, MMStar, BLINK) across model variants.
Figure 2: Visually grounded instruction-following tasks reformulated from self-supervised learning (SSL) pretext tasks. (a) Rotation prediction: the model must recognize object orientations and relate it to canonical poses. (b) Point-wise colorization: the model must match grayscale points to their original colors, requiring fine-grained visual discrimination, spatial grounding, and reasoning over local and global image context. (c) Point correspondence: the model must identify corresponding points across views, requiring cross-view feature matching and spatial reasoning. Collectively, these tasks compel the model to integrate local visual cues with global structure and rely on visual evidence rather than language priors.
Model
Method
CVB-2D
POPE
MMStar
BLINK
Avg.
LLaVA-1.5-Vicuna-7B
Baseline
55.9
87.0
33.5
38.2
53.6
V-GIFT
58.5 (+2.6)
87.2 (+0.2)
34.6 (+1.1)
37.8 (-0.4)
54.5 (+0.9)
LLaVA-1.5-Qwen2.5-7B
Baseline
61.1
87.2
42.7
40.1
57.8
V-GIFT
62.2 (+1.1)
87.7 (+0.5)
43.2 (+0.5)
41.8 (+1.7)
58.7 (+0.9)
LLaVA-OneVision-1.5
Baseline
70.0
88.6
55.3
48.8
65.7
V-GIFT
71.0 (+1.0)
88.9 (+0.3)
55.5 (+0.2)
52.2 (+3.4)
66.9 (+1.2)
Table 1: Main results of incorporating visually grounded SSL tasks on three MLLM models (LLaVA-1.5-Vicuna-7B, LLaVA-1.5-Qwen2.5-7B, LLaVA-OneVision-1.5). Performance is reported on vision-centric benchmarks (CVB-2D, POPE, MMStar, BLINK). Numbers in parentheses indicate improvement over the baseline.
Model
Method
CVB-2D
POPE ∗
MMStar
BLINK
Avg.
LLaVA-1.5-Qwen2.5-7B (LoRA)
Baseline
59.9
87.7
42.6
40.0
57.4
V-GIFT
63.8 (+3.9)
88.5 (+0.8)
43.7 (+1.1)
43.3 (+3.3)
59.6 (+2.2)
VIRAL [ 79 ]
60.5
88.3
39.2
–
–
Table 2: Effect of visually grounded SSL tasks with LoRA training on LLaVA-1.5-Qwen2.5-7B. Performance is reported on vision-centric benchmarks (CVB-2D, POPE, MMStar, BLINK). Numbers in parentheses indicate improvement over the baseline. Baseline results are reproduced, while VIRAL results are reported from [ 79 ] . POPE ∗ represents average accuracy across the “random” and “popular” subsets.
Model
Method
MathVista
OCRBench
RealWorldQA
LLaVA-1.5-Qwen2.5-7B
Baseline
12.2
312.0
56.4
V-GIFT
15.3
311.3
59.0
LLaVA-1.5-Vicuna-7B
Baseline
12.9
319.3
54.2
V-GIFT
12.8
318.3
55.4
LLaVA-OneVision-1.5
Baseline
22.9
627.0
66.1
V-GIFT
22.6
634.0
66.4
Table 3: General benchmark results. Comparing V-GIFT to baselines for LLaVA-1.5-Vicuna-7B, LLaVA-1.5-Qwen2.5-7B, and LLaVA-OneVision-1.5. Performance is reported on MathVista, OCRBench, and RealWorldQA.
SSL task
Benchmark
Rot.
Col.
Corr.
CVB-2D
POPE
MMStar
BLINK
Avg.
61.1
87.2
42.7
40.1
57.8
✓
62.4
87.2
42.5
40.7
58.2
✓
62.3
87.5
41.8
41.7
58.3
✓
61.9
87.3
41.7
42.0
58.2
✓
✓
✓
62.2
87.7
43.2
41.8
58.7
Table 4: Impact of individual and combined SSL pretext tasks on LLaVA-1.5-Qwen2.5-7B ( ρ=1% per task). Each task, rotation (Rot.), colorization (Col.), and correspondence (Corr.), independently improves the average performance over the baseline, while combining all three yields the strongest and most consistent gains across benchmarks.
Figure 3: Effect of the SSL injection ratio ρ on vision-centric instruction-following performance for LLaVA-1.5-Qwen2.5-7B (left) and LLaVA-OneVision-1.5 (right).
Training Setup
CVB-2D
POPE
MMStar
BLINK
Avg.
LLaVA-OneVision-1.5
Baseline (100% instruct)
70.0
88.6
55.3
48.8
65.7
Baseline (+3% instruct only)
68.2
89.0
55.4
49.3
65.6
V-GIFT (+3% SSL)
71.0
88.9
55.5
52.2
66.9
Table 5: Controlling for training compute. We compare our method against baselines trained with the same proportional increase in instruction tuning iterations, but without SSL tasks. Increasing compute alone does not improve performance; gains arise only when the additional data consists of visually grounded SSL tasks.
SSL Injection Stage
CVB-2D
POPE
MMStar
BLINK
Avg.
Baseline
61.1
87.2
42.7
40.1
57.8
SSL before IT
60.1
87.1
42.6
40.8
57.7
SSL during IT (ours)
62.2
87.7
43.2
41.8
58.7
SSL after IT
36.5
50.2
39.5
33.2
39.9
Table 6: Effect of SSL injection stage on LLaVA-1.5-Qwen2.5-7B. Training after the instruction tuning is done using LoRA to avoid catastrophic forgetting. Improvements are obtained only when SSL tasks are integrated during instruction tuning.
SSL Image Source
CVB-2D
POPE
MMStar
BLINK
Avg.
Baseline (no SSL)
61.1
87.2
42.7
40.1
57.8
Single image (augmented views)
62.7
86.9
42.5
41.5
58.4
COCO (reused images)
62.4
87.2
42.5
40.7
58.2
Table 7: Effect of SSL image source (rotation prediction, ρ=1% ) on LLaVA-1.5-Qwen2.5-7B. Both COCO and a single-image source yield improvements over the baseline, indicating that visually grounded supervision, rather than dataset scale, drives the gains.
Model
Method
CVB-2D
MMStar
LLaVA-1.5
Baseline
0.1238
0.1426
LLaVA-1.5
V-GIFT
0.1368
0.1430
Table 8: Mean TVI ↑ [ 53 ] comparison across datasets on LLaVA-1.5 Vicuna-7B.
Figure 4: Attention map from the Baseline (LLaVA-1.5-Vicuna-7B) and V-GIFT on CV-Bench2D examples. V-GIFT produces more focused and better localized attention on task-relevant objects.
Figure 5: Qualitative examples. We present a few qualitative examples comparing LLaVA-1.5 Qwen-2.5-7B baseline against V-GIFT . Our SSL-inspired tasks yield improvements on the variety of vision oriented skills such as counting, multi-view reasoning and visual reasoning.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Examples of the visually grounded self-supervised tasks used during training: colorization point matching, point correspondence, and rotation prediction.
Injection ratio ρ
CVB-2D
POPE
MMStar
BLINK
Avg.
1%
69.4
88.5
55.5
49.9
65.8
3%
71.0
88.9
55.5
52.2
66.9
5%
69.4
88.3
54.7
52.3
66.2
10%
69.5
88.3
54.1
53.7
66.4
30%
68.9
88.2
55.3
52.9
66.3
Appendix
Table 9: Effect of the SSL injection ratio ρ on vision-centric instruction-following performance for LLaVA-OneVision-1.5.