Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
Authors: Seulbi Lee, Sangheum Hwang
Organizations: Department of Data Science, Seoul National University of Science and Technology, Seoul, Republic of Korea. · Currently with Tomocube, Daejeon, Republic of Korea.
Large Vision Language Models (LVLMs) have achieved remarkable progress, yet they often suffer from language bias, producing answers without relying on visual evidence. While prior work attempts to mitigate this issue through decoding strategies, architectural modifications, or curated instruction data, they typically lack a quantitative measure of how much individual training samples or tokens actually benefit from the image. In this work, we introduce Visual Information Gain (VIG), a perplexity-based metric that measures the reduction in prediction uncertainty provided by visual input. VIG enables fine-grained analysis at both sample and token levels, effectively highlighting visually grounded elements such as colors, spatial relations, and attributes. Leveraging this, we propose a VIG-guided selective training scheme that prioritizes high-VIG samples and tokens. This approach improves visual grounding and mitigates language bias, achieving superior performance with significantly reduced supervision by focusing exclusively on visually informative samples and tokens.
Figures & tables
Figure 1: Examples of LLaVA-1.5 instruction tuning data. The dataset includes both samples and tokens with very different levels of visual dependency: some questions can be answered without looking at the image, whereas others need fine-grained visual details (highlighted in green).
Table 1: VIG’s sensitivity to the degree of the visual grounding. Examples from the MS-COCO ( Lin et al., 2014 ) validation set show that VIG quantitatively captures the strength of visual support: high positive for a perfect match, moderate positive for partial grounding, and negative for a conflicting image.
Figure 2: VIG distribution across benchmarks. Blue benchmarks (COCO, CV-Bench, POPE) show stronger multimodal interaction, while red benchmarks (GQA, SQA) exhibit weaker visual dependency.
Figure 3: Visualizing the token-level VIGs. Each point shows a token’s prediction loss with ( x -axis) and without ( y -axis) visual input. The color encodes the token-level loss difference ( y−x ).
Red
white (3.59), black (6.08), lying (5.99), flying (5.73) sitting (4.22), standing (5.10), reading (5.974), crowd (3.30)
Blue
a (0.03), of (0.01), the (0.03), ize (0.00), which (0.00), are (-0.02), The (0.04), - (-1.10)
Table 2: Tokens and their loss differences in LLaVA-1.5 instruction-tuning data. Red regions show large positive loss differences (strong visual grounding), while blue regions exhibit near-zero or negative differences (weak visual contribution).
Figure 4: Average VIG by POS category. Visual relevant categories show higher VIG than function words, suggesting that VIG emphasizes linguistically informative tokens for visual grounding.
Model
# Sample Tokens
# Active Tokens
Vision Understanding
Hallucination
LLaVA W
MMVet
MMBench
CV-Bench
DocVQA
POPE
CHAIR
MMHal
Score ↑
Acc. ↑
F1 ↑
Acc. ↑
CS↓
CI↓
Score ↑
Hall. ↓
LLaVA-1.5 7B
58.61M
58.61M
59.02
28.62
65.46
59.18
22.31
85.90
87.08
52.93
14.99
1.71
71.25
+ VIG training
51.17M
38.45M
61.22
32.71
66.33
62.48
22.51
85.93
87.47
47.00
12.80
2.23
62.78
Δ
-13%
-34%
+2.20
+4.09
+0.87
+3.30
+0.20
+0.03
+0.39
+5.93
+2.19
+0.52
+8.47
LLaVA-1.5 13B
58.61M
58.61M
72.01
36.19
67.52
60.16
24.08
85.72
87.05
51.96
13.22
2.05
67.09
Table 3: Quantitative comparison on vision understanding and hallucination benchmarks. We report results for four LVLMs: LLaVA-1.5 7B/13B, ShareGPT4V 7B, and Open-Qwen2VL 2B. “VIG training” denotes our VIG-guided selective training. “# Sample Tokens” represents the total number of answer tokens contained in the multimodal samples retained after sample-level selection. “# Active Tokens” refers to the effective number of tokens that contribute to the loss computation after applying token-level masking. Δ indicates the percentage reduction in token count or the performance improvement compared to the vanilla baseline. For each metric, ↑ indicates higher is better and ↓ indicates lower is better. Bold indicates entries where VIG-guided selective training outperforms the vanilla model.
Model
Vision Understanding
Hallucination
LLaVA W
MMVet
MMBench
DocVQA
POPE
CHAIR
MMHal
Score ↑
Acc. ↑
F1 ↑
Acc. ↑
CS↓
CI↓
Score ↑
Hall. ↓
LLaVA-1.5 7B
59.02
28.62
65.46
22.31
85.90
87.08
52.93
14.99
1.71
71.25
+ VCD
60.55
27.01
64.34
22.98
86.53
86.61
49.81
13.69
1.67
76.01
+ PAI
57.54
27.99
65.45
21.98
85.98
85.84
35.46
12.11
1.78
72.31
+ VAR
61.11
30.98
66.30
23.11
86.10
87.12
50.09
14.77
2.14
62.89
Table 4: Quantitative comparison with existing methods on LLaVA-1.5 7B. We compare our approach against recent training-free (VCD ( Leng et al., 2024 ) , PAI ( Liu et al., 2025a ) , VAR ( Kang et al., 2025 ) ) and training-based approaches (LACING ( Zhao et al., 2025 ) ), proposed to improve visual grounding. “VIG training” denotes VIG-guided selective training on LLaVA-1.5 7B. Values in parentheses indicate the performance improvement over the vanilla model. Bold indicates entries where VIG-guided selective training improves over the vanilla model.
Figure 5: Attention fraction allocated to visual tokens. Compared to LLaVA-1.5 7B, VIG training assigns significantly more attention to visual tokens across all layers.
Figure 6: Evaluation of text reliance under textual corruption. Base: accuracy on clean inputs. Corruption: accuracy when the same image is paired with a corrupted caption containing a conflicting description. Norm: corruption accuracy normalized by the corresponding Base (Corruption/Base).
Model
GSM8K
MMLU
HellaSwag
TruthfulQA
LLaVA 1.5 7B
15.33
51.68
76.09
45.86
+ VIG training
14.99
51.35
76.11
46.01
Table 5: Comparison on text-only benchmark performance. Performance on GSM8K, MMLU, HellaSwag, and TruthfulQA remains largely unchanged with VIG training, suggesting that improvements in visual grounding do not come at the expense of text comprehension capabilities.
Model
LLaVA W↑
MMBench ↑
CHAIR ↓
MMHal ↓
LLaVA1.5-7B
59.02
65.46
52.93
71.25
Random
56.91
55.97
54.88
74.49
SS
58.12
57.56
50.23
68.14
SS+TS
61.19
66.33
49.10
61.82
Table 6: Ablation study of selection levels on LLaVA-v1.5 7B. “Random” trains on a random 70% subset of the data, “SS” selects the top 70% samples by VIG score (sample-level selection only), and “SS+TS” additionally applies token-level VIG selection. We report a single metric per benchmark: LLaVA W score, MMBench score, CS for CHAIR, and Hall. for MMHal.
Figure 7: Ablation study of selection ratio p% on LLaVA-1.5 7B. We report a single metric per benchmark: LLaVA W score, MMBench score, CS for CHAIR, and Hall for MMHal. p=100 corresponds to the vanilla model trained on the full instruction-tuning dataset (no VIG-based selection). All scores are normalized with respect to the p=100 setting.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Table 7: Additional qualitative examples of VIG. Examples from the MS-COCO ( Lin et al., 2014 ) validation set show that VIG captures the strength of visual grounding: high positive for a strong match, near-zero for weak or partial grounding, and negative for a conflicting image. The value shown below each image is its VIG score.
Figure C.1 : Examples of with and without image setting.
Model
LLaVA-1.5 7B
LLaVA-1.5 13B
ShareGPT4V 7B
Open-Qwen2VL 2B
Threshold τ70
−0.021
0.046
−0.042
0.012
Appendix
Table D.1: Selection threshold τp at p=70 across the models.
Model
Epoch
Batch size
LR
LR schedule
LR warmup ratio
Optimizer
Training time
Pretraining
ShareGPT4V
1
256
2e-5
cosine decay
0.03
AdamW
12 hours
Instruction Tuning
LLaVA-1.5 Family, ShareGPT4V
1
128
2e-5
cosine decay
0.03
AdamW
7 hours
Open-Qwen2VL 2B
1
128
2e-5
linear-warmup + cosine decay
0.03
AdamW
10 hours
Appendix
Table D.2: Hyperparameters for training, which are identical to the original models and training time.
User
Describe this photo in detail.
LLaVA-1.5 7B
Vanilla
The image features a painting of a dog dressed up as a woman, wearing a dress and a hat. The dog is the main focus of the painting, and it appears to be the subject of the artwork. The painting is set in a room with a dining table visible in the background. There are also two people in the scene, one on the left side and another on the right side of the painting. The person on the left is partially visible, while the person on the right is more prominently featured in the scene.
+VIG Training
The image features a painting of a dog wearing a Renaissance-style dress , complete with a hood and a cape. The dog is sitting in a pose that resembles a woman , giving the impression of a human-like figure. The dog’s attire and the way it is positioned in the painting create a unique and intriguing scene.
LLaVA-1.5 13B
Vanilla
The image is a painting featuring a dog dressed up as Mona Lisa, wearing a bandana on its head and holding its hands together, posing for the portrait. The background includes mountains and a river, adding to the composition’s artistic appeal. In addition to the dog, there are three people present in the scene. One person is positioned near the left edge, another person can be seen at the bottom right corner, and the third person is located on the right side of the painting.
+VIG Training
The image features a painting of a dog dressed as the famous painting of the Mona Lisa. The dog is wearing a bandana on its head and is positioned in a way that resembles the iconic pose of the Mona Lisa . The painting is quite detailed, capturing the essence of the original artwork while incorporating the unique twist of the dog as the subject. The dog’s facial expression and posture are reminiscent of the Mona Lisa’s enigmatic smile and pose, making the painting a creative and amusing take on the classic artwork .
ShareGPT4V 7B
Vanilla
The image presents a surrealistic portrait of a dog, painted in the style of the Renaissance period. The dog, the central figure of the painting, is adorned with a red hat and a blue robe, adding a touch of whimsy to the otherwise realistic style. The background is a landscape, painted in a realistic style, with a river meandering through it, adding depth to the composition. The painting is executed in oil on canvas, a medium that allows for rich colors and intricate details. The genre of the painting is surrealism, a style that combines elements of the real and the imaginary, creating a dreamlike quality. The painting is a testament to the artist’s skill in blending realism with surrealism, creating a unique and captivating piece of art.
Appendix
Table D.3: Examples of models’ responses from LLaVA W . When applying VIG training, the model provides more visually-grounded responses for writing tasks with visual inputs than base LLaVA-1.5 and ShareGPT4V.
User
Is this airplane taking off or landing?
LLaVA-1.5 7B
Vanilla
Based on the image, it is not possible to definitively determine whether the airplane is taking off or landing. However, we can observe that the airplane is in the process of either ascending or descending, as it is flying low over the runway. The presence of smoke coming from the airplane’s wheels suggests that it could be either taking off or landing. To determine the exact phase of the flight, we would need more information or a clearer view of the airplane’s position relative to the ground.
+VIG Training
The airplane is taking off from the runway .
LLaVA-1.5 13B
Vanilla
The airplane in the image is taking off from the runway.
+VIG Training
The airplane in the image is taking off from the runway.
ShareGPT4V 7B
Vanilla
The airplane is taking off from the runway.
Appendix
Table D.4: Examples of models’ responses from MMVet. When applying VIG training, the model provides more accurate responses for writing tasks with visual inputs than base LLaVA-1.5 7B.
Figure E.1 : Attention fraction allocated to visual tokens. Compared to their respective baselines, VIG-trained models consistently assign significantly higher attention weights to visual tokens across all layers, demonstrating improved visual grounding regardless of model scale or architecture.
Figure E.2 : Evaluation of text reliance under textual corruption. Base: accuracy on clean inputs. Corruption: accuracy when the same image is paired with a corrupted caption containing a conflicting description. Norm: corruption accuracy normalized by the corresponding Base (Corruption/Base).
Model
Base
Corruption
Norm
LLaVA-1.5 7B (Baseline)
77.9
32.1
41.2
+ VCD
78.1
33.5
42.9
+ VIG training
78.2
42.3
54.1
Appendix
Table E.1: Comparison with VCD under textual corruption on VQAv2. Evaluated on LLaVA-1.5 7B. “Base” represents accuracy on clean inputs, while “Corruption” is the accuracy when misleading text is present. “Norm” is the normalized score (Corruption / Base × 100).
Model
GSM8K
MMLU
HellaSwag
TruthfulQA
LLaVA 1.5 13B
15.33
51.68
76.09
45.86
+ VIG training
14.99
51.35
76.11
46.01
ShareGPT4V 7B
18.01
47.62
54.78
46.53
+ VIG training
18.03
47.66
56.14
46.42
Appendix
Table E.2: Comparison of text-only benchmark performance before and after VIG-guided training. For both LLaVA-1.5 13B and ShareGPT4V, performance on GSM8K, MMLU, HellaSwag, and TruthfulQA remains largely unchanged after VIG training, suggesting that improvements in visual grounding do not come at the expense of text comprehension.
Figure E.3 : Data Distribution after VIG-based Filtering. Compared to the original data distribution, VIG-based filtering consistently preserves long-tail samples while selectively reshaping the task composition of the training data. This suggests that VIG-based filtering improves sample quality without sacrificing broad data coverage.
Model
Vision Understanding
Hallucination
LLaVA W
MMVet
MMBench
DocVQA
POPE
CHAIR
MMHal
Score ↑
Acc. ↑
F1 ↑
Acc. ↑
CS↓
CI↓
Score ↑
Hall. ↓
LLaVA-1.5 7B
59.02
28.62
65.46
22.31
85.90
87.08
52.93
14.99
1.71
71.25
Random
58.50
27.27
55.97
21.38
85.48
86.63
53.11
15.00
1.79
71.33
SS
58.71
32.29
57.56
22.06
85.22
86.53
52.10
14.61
1.84
63.54
SS+TS
61.22
32.71
66.33
22.51
85.93
87.47
47.00
12.80
2.23
62.78
Appendix
Table F.1 : Ablation study of selection levels on LLaVA-1.5 7B. “Random” trains on a random 70% subset of the data, “SS” selects the top 70% samples by VIG score (sample-level selection only), and “SS+TS” additionally applies token-level VIG selection. For each metric, ↑ indicates higher is better and ↓ indicates lower is better. Bold indicates the best performance.
Ratio p
30
50
70
Threshold τp
0.124
0.031
−0.021
Appendix
Table F.2: Selection threshold τp at each p on LLaVA-1.5 7B.
Ratio p
Vision Understanding
Hallucination
# Sample Tokens
# Active Tokens
LLaVA W
MMVet
MMBench
DocVQA
POPE
CHAIR
MMHal
Score ↑
Acc. ↑
F1 ↑
Acc. ↑
CS↓
CI↓
Score ↑
Hall. ↓
30%
8.58M
3.19M
60.90
34.03
56.87
22.66
82.80
84.28
48.71
13.31
1.99
63.01
50%
26.32M
10.00M
61.81
31.19
56.44
22.49
85.09
86.14
47.63
12.99
1.95
62.91
70%
51.17M
38.45M
61.22
32.71
66.33
22.51
85.93
87.47
47.00
12.80
2.23
62.78
100%
58.61M
58.61M
59.02
28.62
65.46
22.31
85.90
87.08
52.93
14.99
1.71
71.25
Appendix
Table F.3 : Ablation study of selection ratio p% on LLaVA-1.5 7B. “# Sample Tokens” represents the total number of answer tokens contained in the multimodal samples retained after sample-level selection. “# Active Tokens” refers to the effective number of tokens that contribute to the loss computation after applying token-level masking. For each metric, ↑ indicates higher is better and ↓ indicates lower is better. Bold indicates the best performance.
Large Vision-Language Models (LVLMs) extend large language models with visual understanding, but remain vulnerable to hallucination, where outputs are fluent yet inconsistent with images. Recent studies link this issue to language bias-the tendency of LVLMs to over-rely on text while neglecting visual inputs. Yet most analyses remain empirical without uncovering its underlying cause. In this paper, we provide a systematic study of language bias and identify its root in modality misalignment during training. Our analysis shows that both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) often prioritize textual improvements, which may cause LVLMs to overly lean toward language modeling rather than balanced multimodal understanding. To address this, we propose two simple yet effective methods: Language Bias Regularization (LBR) which mitigates language bias through regularization during instruction tuning, and Language Bias Penalty (LBP), which penalizes language bias in the DPO training process. Extensive experiments across diverse models and benchmarks demonstrate the effectiveness of our approach. LBR consistently improves performance on over ten general benchmarks, while LBP significantly reduces hallucination and improves trustworthiness. Together, these methods not only mitigate language bias but also advance the overall alignment of LVLMs, all without introducing any additional data or auxiliary models. Our code is publicly available at https://github.com/lab-klc/LVLM-Language-Bias.
Vision-language models (VLMs) achieve high accuracy on many visual question-answering benchmarks, yet it remains unclear whether this accuracy reflects reliable use of the visual details a question depends on. We examine this tension through the lens of visual ignorance: cases where a model can represent task-relevant visual evidence without using it to determine the answer. Across three VLMs and twelve benchmarks, we track the answer-relevant information decodable at each decoder layer and measure when it becomes coupled to the final output. The visual evidence needed for the correct answer can already be read out from intermediate layers, yet it has little effect on the output until the final layers of the decoder. Causal residual interchange confirms that shifting the answer requires the late states, not the earlier ones. We then evaluate all twelve benchmarks under progressive Gaussian blur and find that substantial fractions of predictions survive the entire degradation sequence while continuing to receive benchmark credit. These results distinguish the availability of visual information from its influence on prediction, and show that standard accuracy can coexist with limited sensitivity to the tested visual detail. They motivate training and evaluation that make answer-relevant visual distinctions necessary to answer correctly.
Uncertainty quantification (UQ) remains a critical challenge in Large Vision Language Models (LVLMs) for reliable predictions and real-world deployment. However, most existing methods are adapted from the LLM literature and primarily focus on the language modality, leaving the contribution of visual information to LVLM uncertainty largely underexplored. In this paper, we investigate how LVLMs process visual information and whether this process can be used to improve uncertainty estimation. By analyzing hidden representations after the integration of visual features during the generation process, we observe that high-confidence predictions rely more heavily on visual content than uncertain ones. Building on this insight, we propose Visual-Grounded Token UQ (VIG-TUQ), a training-free framework that explicitly incorporates visual grounding into uncertainty estimation by weighting token-level language uncertainty with visual grounding scores. We evaluate VIG-TUQ on multiple datasets and across diverse LVLM architectures, including early-fusion, late-fusion, and native-fusion models. Results indicate that our method often improves upon existing token-level uncertainty approaches. Code and data will be made available upon acceptance.
Joseph Hoche, David Brellmann, Gianni Franchi
AMIAD, Pôle Recherche, Palaiseau · Safran Tech · ENSTA Paris