Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain

Seulbi Lee; Sangheum Hwang

cs.CVFeb 19, 2026

Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain

Authors: Seulbi Lee, Sangheum Hwang

Organizations: Department of Data Science, Seoul National University of Science and Technology, Seoul, Republic of Korea. · Currently with Tomocube, Daejeon, Republic of Korea.

Abstract

Large Vision Language Models (LVLMs) have achieved remarkable progress, yet they often suffer from language bias, producing answers without relying on visual evidence. While prior work attempts to mitigate this issue through decoding strategies, architectural modifications, or curated instruction data, they typically lack a quantitative measure of how much individual training samples or tokens actually benefit from the image. In this work, we introduce Visual Information Gain (VIG), a perplexity-based metric that measures the reduction in prediction uncertainty provided by visual input. VIG enables fine-grained analysis at both sample and token levels, effectively highlighting visually grounded elements such as colors, spatial relations, and attributes. Leveraging this, we propose a VIG-guided selective training scheme that prioritizes high-VIG samples and tokens. This approach improves visual grounding and mitigates language bias, achieving superior performance with significantly reduced supervision by focusing exclusively on visually informative samples and tokens.

Figures & tables

Panel of Figure 1 — Figure 1: Examples of LLaVA-1.5 instruction tuning data. The dataset includes both samples and tokens with very different levels of visual dependency: some questions can be answered without looking at the image, whereas others need fine-grained visual details (highlighted in green).

Panel of Table 1 — Table 1: VIG’s sensitivity to the degree of the visual grounding. Examples from the MS-COCO ( Lin et al., 2014 ) validation set show that VIG quantitatively captures the strength of visual support: high positive for a perfect match, moderate positive for partial grounding, and negative for a conflicting image.

Figure 2: VIG distribution across benchmarks. Blue benchmarks (COCO, CV-Bench, POPE) show stronger multimodal interaction, while red benchmarks (GQA, SQA) exhibit weaker visual dependency.

Figure 3: Visualizing the token-level VIGs. Each point shows a token’s prediction loss with ( $x$ -axis) and without ( $y$ -axis) visual input. The color encodes the token-level loss difference ( $y-x$ ).

Red	white (3.59), black (6.08), lying (5.99), flying (5.73) sitting (4.22), standing (5.10), reading (5.974), crowd (3.30)
Blue	a (0.03), of (0.01), the (0.03), ize (0.00), which (0.00), are (-0.02), The (0.04), - (-1.10)

Table 2: Tokens and their loss differences in LLaVA-1.5 instruction-tuning data. Red regions show large positive loss differences (strong visual grounding), while blue regions exhibit near-zero or negative differences (weak visual contribution).

Figure 4: Average VIG by POS category. Visual relevant categories show higher VIG than function words, suggesting that VIG emphasizes linguistically informative tokens for visual grounding.

Model	# Sample Tokens	# Active Tokens	Vision Understanding					Hallucination
			LLaVA ${}^{\text{W}}$	MMVet	MMBench	CV-Bench	DocVQA	POPE		CHAIR		MMHal
			Score $\uparrow$			Acc. $\uparrow$		F1 $\uparrow$	Acc. $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	Score $\uparrow$	Hall. $\downarrow$
LLaVA-1.5 7B	58.61M	58.61M	59.02	28.62	65.46	59.18	22.31	85.90	87.08	52.93	14.99	1.71	71.25
+ VIG training	51.17M	38.45M	61.22	32.71	66.33	62.48	22.51	85.93	87.47	47.00	12.80	2.23	62.78
$\Delta$	-13%	-34%	+2.20	+4.09	+0.87	+3.30	+0.20	+0.03	+0.39	+5.93	+2.19	+0.52	+8.47
LLaVA-1.5 13B	58.61M	58.61M	72.01	36.19	67.52	60.16	24.08	85.72	87.05	51.96	13.22	2.05	67.09

Table 3: Quantitative comparison on vision understanding and hallucination benchmarks. We report results for four LVLMs: LLaVA-1.5 7B/13B, ShareGPT4V 7B, and Open-Qwen2VL 2B. “VIG training” denotes our VIG-guided selective training. “# Sample Tokens” represents the total number of answer tokens contained in the multimodal samples retained after sample-level selection. “# Active Tokens” refers to the effective number of tokens that contribute to the loss computation after applying token-level masking.

\Delta

indicates the percentage reduction in token count or the performance improvement compared to the vanilla baseline. For each metric,

\uparrow

indicates higher is better and

\downarrow

indicates lower is better. Bold indicates entries where VIG-guided selective training outperforms the vanilla model.

Model	Vision Understanding				Hallucination
	LLaVA ${}^{\text{W}}$	MMVet	MMBench	DocVQA	POPE		CHAIR		MMHal
	Score $\uparrow$			Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	Score $\uparrow$	Hall. $\downarrow$
LLaVA-1.5 7B	59.02	28.62	65.46	22.31	85.90	87.08	52.93	14.99	1.71	71.25
+ VCD	60.55	27.01	64.34	22.98	86.53	86.61	49.81	13.69	1.67	76.01
+ PAI	57.54	27.99	65.45	21.98	85.98	85.84	35.46	12.11	1.78	72.31
+ VAR	61.11	30.98	66.30	23.11	86.10	87.12	50.09	14.77	2.14	62.89

Table 4: Quantitative comparison with existing methods on LLaVA-1.5 7B. We compare our approach against recent training-free (VCD ( Leng et al., 2024 ) , PAI ( Liu et al., 2025a ) , VAR ( Kang et al., 2025 ) ) and training-based approaches (LACING ( Zhao et al., 2025 ) ), proposed to improve visual grounding. “VIG training” denotes VIG-guided selective training on LLaVA-1.5 7B. Values in parentheses indicate the performance improvement over the vanilla model. Bold indicates entries where VIG-guided selective training improves over the vanilla model.

Figure 5: Attention fraction allocated to visual tokens. Compared to LLaVA-1.5 7B, VIG training assigns significantly more attention to visual tokens across all layers.

Figure 6: Evaluation of text reliance under textual corruption. Base: accuracy on clean inputs. Corruption: accuracy when the same image is paired with a corrupted caption containing a conflicting description. Norm: corruption accuracy normalized by the corresponding Base (Corruption/Base).

Model	GSM8K	MMLU	HellaSwag	TruthfulQA
LLaVA 1.5 7B	15.33	51.68	76.09	45.86
+ VIG training	14.99	51.35	76.11	46.01

Table 5: Comparison on text-only benchmark performance. Performance on GSM8K, MMLU, HellaSwag, and TruthfulQA remains largely unchanged with VIG training, suggesting that improvements in visual grounding do not come at the expense of text comprehension capabilities.

Model	LLaVA ${}^{\text{W}}$ $\uparrow$	MMBench $\uparrow$	CHAIR $\downarrow$	MMHal $\downarrow$
LLaVA1.5-7B	59.02	65.46	52.93	71.25
Random	56.91	55.97	54.88	74.49
SS	58.12	57.56	50.23	68.14
SS+TS	61.19	66.33	49.10	61.82

Table 6: Ablation study of selection levels on LLaVA-v1.5 7B. “Random” trains on a random 70% subset of the data, “SS” selects the top 70% samples by VIG score (sample-level selection only), and “SS+TS” additionally applies token-level VIG selection. We report a single metric per benchmark: LLaVA

{}^{\text{W}}

score, MMBench score,

C_{S}

for CHAIR, and Hall. for MMHal.

Figure 7: Ablation study of selection ratio

p\%

on LLaVA-1.5 7B. We report a single metric per benchmark: LLaVA

{}^{\text{W}}

score, MMBench score,

C_{S}

for CHAIR, and Hall for MMHal.

p=100

corresponds to the vanilla model trained on the full instruction-tuning dataset (no VIG-based selection). All scores are normalized with respect to the

p=100

setting.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 7 — Table 7: Additional qualitative examples of VIG. Examples from the MS-COCO ( Lin et al., 2014 ) validation set show that VIG captures the strength of visual grounding: high positive for a strong match, near-zero for weak or partial grounding, and negative for a conflicting image. The value shown below each image is its VIG score.

Panel of Figure 15 — Figure C.1 : Examples of with and without image setting.

Model	LLaVA-1.5 7B	LLaVA-1.5 13B	ShareGPT4V 7B	Open-Qwen2VL 2B
Threshold $\tau_{70}$	$-0.021$	$0.046$	$-0.042$	$0.012$

Appendix

Table D.1: Selection threshold

\tau_{p}

p=70

across the models.

	Model	Epoch	Batch size	LR	LR schedule	LR warmup ratio	Optimizer	Training time
Pretraining	ShareGPT4V	1	256	2e-5	cosine decay	0.03	AdamW	12 hours
Instruction Tuning	LLaVA-1.5 Family, ShareGPT4V	1	128	2e-5	cosine decay	0.03	AdamW	7 hours
Instruction Tuning	Open-Qwen2VL 2B	1	128	2e-5	linear-warmup + cosine decay	0.03	AdamW	10 hours

Appendix

Table D.2: Hyperparameters for training, which are identical to the original models and training time.


User	Describe this photo in detail.
LLaVA-1.5 7B	Vanilla	The image features a painting of a dog dressed up as a woman, wearing a dress and a hat. The dog is the main focus of the painting, and it appears to be the subject of the artwork. The painting is set in a room with a dining table visible in the background. There are also two people in the scene, one on the left side and another on the right side of the painting. The person on the left is partially visible, while the person on the right is more prominently featured in the scene.
	+VIG Training	The image features a painting of a dog wearing a Renaissance-style dress , complete with a hood and a cape. The dog is sitting in a pose that resembles a woman , giving the impression of a human-like figure. The dog’s attire and the way it is positioned in the painting create a unique and intriguing scene.
LLaVA-1.5 13B	Vanilla	The image is a painting featuring a dog dressed up as Mona Lisa, wearing a bandana on its head and holding its hands together, posing for the portrait. The background includes mountains and a river, adding to the composition’s artistic appeal. In addition to the dog, there are three people present in the scene. One person is positioned near the left edge, another person can be seen at the bottom right corner, and the third person is located on the right side of the painting.
	+VIG Training	The image features a painting of a dog dressed as the famous painting of the Mona Lisa. The dog is wearing a bandana on its head and is positioned in a way that resembles the iconic pose of the Mona Lisa . The painting is quite detailed, capturing the essence of the original artwork while incorporating the unique twist of the dog as the subject. The dog’s facial expression and posture are reminiscent of the Mona Lisa’s enigmatic smile and pose, making the painting a creative and amusing take on the classic artwork .
ShareGPT4V 7B	Vanilla	The image presents a surrealistic portrait of a dog, painted in the style of the Renaissance period. The dog, the central figure of the painting, is adorned with a red hat and a blue robe, adding a touch of whimsy to the otherwise realistic style. The background is a landscape, painted in a realistic style, with a river meandering through it, adding depth to the composition. The painting is executed in oil on canvas, a medium that allows for rich colors and intricate details. The genre of the painting is surrealism, a style that combines elements of the real and the imaginary, creating a dreamlike quality. The painting is a testament to the artist’s skill in blending realism with surrealism, creating a unique and captivating piece of art.

Appendix

Table D.3: Examples of models’ responses from LLaVA

{}^{\text{W}}

. When applying VIG training, the model provides more visually-grounded responses for writing tasks with visual inputs than base LLaVA-1.5 and ShareGPT4V.


User	Is this airplane taking off or landing?
LLaVA-1.5 7B	Vanilla	Based on the image, it is not possible to definitively determine whether the airplane is taking off or landing. However, we can observe that the airplane is in the process of either ascending or descending, as it is flying low over the runway. The presence of smoke coming from the airplane’s wheels suggests that it could be either taking off or landing. To determine the exact phase of the flight, we would need more information or a clearer view of the airplane’s position relative to the ground.
	+VIG Training	The airplane is taking off from the runway .
LLaVA-1.5 13B	Vanilla	The airplane in the image is taking off from the runway.
	+VIG Training	The airplane in the image is taking off from the runway.
ShareGPT4V 7B	Vanilla	The airplane is taking off from the runway.

Appendix

Table D.4: Examples of models’ responses from MMVet. When applying VIG training, the model provides more accurate responses for writing tasks with visual inputs than base LLaVA-1.5 7B.

Panel of Figure 20 — Figure E.1 : Attention fraction allocated to visual tokens. Compared to their respective baselines, VIG-trained models consistently assign significantly higher attention weights to visual tokens across all layers, demonstrating improved visual grounding regardless of model scale or architecture.

Panel of Figure 21 — Figure E.2 : Evaluation of text reliance under textual corruption. Base: accuracy on clean inputs. Corruption: accuracy when the same image is paired with a corrupted caption containing a conflicting description. Norm: corruption accuracy normalized by the corresponding Base (Corruption/Base).

Model	Base	Corruption	Norm
LLaVA-1.5 7B (Baseline)	77.9	32.1	41.2
+ VCD	78.1	33.5	42.9
+ VIG training	78.2	42.3	54.1

Appendix

Table E.1: Comparison with VCD under textual corruption on VQAv2. Evaluated on LLaVA-1.5 7B. “Base” represents accuracy on clean inputs, while “Corruption” is the accuracy when misleading text is present. “Norm” is the normalized score (Corruption / Base

\times

100).

Model	GSM8K	MMLU	HellaSwag	TruthfulQA
LLaVA 1.5 13B	15.33	51.68	76.09	45.86
+ VIG training	14.99	51.35	76.11	46.01
ShareGPT4V 7B	18.01	47.62	54.78	46.53
+ VIG training	18.03	47.66	56.14	46.42

Appendix

Table E.2: Comparison of text-only benchmark performance before and after VIG-guided training. For both LLaVA-1.5 13B and ShareGPT4V, performance on GSM8K, MMLU, HellaSwag, and TruthfulQA remains largely unchanged after VIG training, suggesting that improvements in visual grounding do not come at the expense of text comprehension.

Panel of Figure 24 — Figure E.3 : Data Distribution after VIG-based Filtering. Compared to the original data distribution, VIG-based filtering consistently preserves long-tail samples while selectively reshaping the task composition of the training data. This suggests that VIG-based filtering improves sample quality without sacrificing broad data coverage.

Model	Vision Understanding				Hallucination
	LLaVA ${}^{\text{W}}$	MMVet	MMBench	DocVQA	POPE		CHAIR		MMHal
	Score $\uparrow$			Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	Score $\uparrow$	Hall. $\downarrow$
LLaVA-1.5 7B	59.02	28.62	65.46	22.31	85.90	87.08	52.93	14.99	1.71	71.25
Random	58.50	27.27	55.97	21.38	85.48	86.63	53.11	15.00	1.79	71.33
SS	58.71	32.29	57.56	22.06	85.22	86.53	52.10	14.61	1.84	63.54
SS+TS	61.22	32.71	66.33	22.51	85.93	87.47	47.00	12.80	2.23	62.78

Appendix

Table F.1 : Ablation study of selection levels on LLaVA-1.5 7B. “Random” trains on a random 70% subset of the data, “SS” selects the top 70% samples by VIG score (sample-level selection only), and “SS+TS” additionally applies token-level VIG selection. For each metric,

\uparrow

indicates higher is better and

\downarrow

indicates lower is better. Bold indicates the best performance.

Ratio $p$	30	50	70
Threshold $\tau_{p}$	$0.124$	$0.031$	$-0.021$

Appendix

Table F.2: Selection threshold

\tau_{p}

at each

p

on LLaVA-1.5 7B.

Ratio $p$			Vision Understanding				Hallucination
	# Sample Tokens	# Active Tokens	LLaVA ${}^{\text{W}}$	MMVet	MMBench	DocVQA	POPE		CHAIR		MMHal
			Score $\uparrow$			Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	Score $\uparrow$	Hall. $\downarrow$
30%	8.58M	3.19M	60.90	34.03	56.87	22.66	82.80	84.28	48.71	13.31	1.99	63.01
50%	26.32M	10.00M	61.81	31.19	56.44	22.49	85.09	86.14	47.63	12.99	1.95	62.91
70%	51.17M	38.45M	61.22	32.71	66.33	22.51	85.93	87.47	47.00	12.80	2.23	62.78
100%	58.61M	58.61M	59.02	28.62	65.46	22.31	85.90	87.08	52.93	14.99	1.71	71.25

Appendix

Table F.3 : Ablation study of selection ratio

p\%

on LLaVA-1.5 7B. “# Sample Tokens” represents the total number of answer tokens contained in the multimodal samples retained after sample-level selection. “# Active Tokens” refers to the effective number of tokens that contribute to the loss computation after applying token-level masking. For each metric,

\uparrow

indicates higher is better and

\downarrow

indicates lower is better. Bold indicates the best performance.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 724 papers

Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
May 24, 2026Yangneng Chen, Jing LiVision-Language Models VLM Adaptation
1Harbin Institute of Technology, Shenzhen, China.
Diagnosing Visual Ignorance in Vision-Language Models
Jun 5, 2026Runyu Zhou, Qi Zhang, Yisen WangVLM Evaluation VLM Robustness
Peking University
Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
May 26, 2026Joseph Hoche, David Brellmann, Gianni FranchiVision-Language Models Large Vision-Language Models
AMIAD, Pôle Recherche, Palaiseau · Safran Tech · ENSTA Paris

May 24, 2026J/K move · Enter open · S save

Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation

Yangneng Chen, Jing Li

1Harbin Institute of Technology, Shenzhen, China.

Vision-Language Models VLM Adaptation Object Hallucination in VLMs Vision-Language Alignment

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) extend large language models with visual understanding, but remain vulnerable to hallucination, where outputs are fluent yet inconsistent with images. Recent studies link this issue to language bias-the tendency of LVLMs to over-rely on text while neglecting visual inputs. Yet most analyses remain empirical without uncovering its underlying cause. In this paper, we provide a systematic study of language bias and identify its root in modality misalignment during training. Our analysis shows that both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) often prioritize textual improvements, which may cause LVLMs to overly lean toward language modeling rather than balanced multimodal understanding. To address this, we propose two simple yet effective methods: Language Bias Regularization (LBR) which mitigates language bias through regularization during instruction tuning, and Language Bias Penalty (LBP), which penalizes language bias in the DPO training process. Extensive experiments across diverse models and benchmarks demonstrate the effectiveness of our approach. LBR consistently improves performance on over ten general benchmarks, while LBP significantly reduces hallucination and improves trustworthiness. Together, these methods not only mitigate language bias but also advance the overall alignment of LVLMs, all without introducing any additional data or auxiliary models. Our code is publicly available at https://github.com/lab-klc/LVLM-Language-Bias.

Abstract

Vision-Language Models

Explore similar work

Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation

Diagnosing Visual Ignorance in Vision-Language Models

Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation