A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

Lixin Xiu; Xufang Luo; Hideki Nakayama

cs.LGMar 31, 2026

A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

Authors: Lixin Xiu, Xufang Luo, Hideki Nakayama

Organizations: The University of Tokyo · Microsoft Research

Abstract

Large vision-language models (LVLMs) achieve impressive performance, yet their internal decision-making processes remain opaque, making it difficult to determine if the success stems from true multimodal fusion or from reliance on unimodal priors. To address this attribution gap, we introduce a novel framework using partial information decomposition (PID) to quantitatively measure the "information spectrum" of LVLMs -- decomposing a model's decision-relevant information into redundant, unique, and synergistic components. By adapting a scalable estimator to modern LVLM outputs, our model-agnostic pipeline profiles 26 LVLMs on four datasets across three dimensions -- breadth (cross-model & cross-task), depth (layer-wise information dynamics), and time (learning dynamics across training). Our analysis reveals two key results: (i) two task regimes (synergy-driven vs. knowledge-driven) and (ii) two stable, contrasting family-level strategies (fusion-centric vs. language-centric). We also uncover a consistent three-phase pattern in layer-wise processing and identify visual instruction tuning as the key stage where fusion is learned. Together, these contributions provide a quantitative lens beyond accuracy-only evaluation and offer insights for analyzing and designing the next generation of LVLMs. Code and data are available at https://github.com/RiiShin/pid-lvlm-analysis .

Figures & tables

Figure 1: Overview of this research. The first part is the framework of PID estimation for LVLMs. Given an image-text pair, we extract image and text embeddings as two features, run a standard multimodal forward pass and collect two unimodal predictions by masking the other modality. PID values are estimated with BATCH estimator. The second part reveals three analysis dimensions: (1) cross-model and cross-task comparison, (2) layer-wise information dynamics, and (3) learning dynamics over training. To our knowledge, this is the first comprehensive LVLM analysis through the lens of information decomposition.

Dataset	Task	# of options	# of training samples	# of test samples
MMBench	General visual reasoning	2–4	3246	1083
POPE	Hallucination evaluation	2	2250	750
Reefknot	Hallucination evaluation	4	1612	538
PMC-VQA	Domain-specific knowledge	4	1500	500

Table 1: Details of the datasets used for evaluation. The listed training and test splits are not for LVLM fine-tuning; they are created by randomly partitioning each dataset (3:1 ratio) for the PID estimation, as the BATCH estimator requires separate sets to train networks and estimate PID values.

Figure 2: Share of synergy

S

and language uniqueness

U_{2}

across four datasets.

Dataset	$S$		$U_{2}$		$I(X_{1},X_{2};Y)$		$I(X_{1};X_{2};Y)$
	$\rho$	$p$ -val	$\rho$	$p$ -val	$\rho$	$p$ -val	$\rho$	$p$ -val
MMBench	0.750	$<0.001$	0.194	0.343	0.632	$<0.001$	-0.757	$<0.001$
POPE	0.742	$<0.001$	-0.009	0.964	0.157	0.445	-0.701	$<0.001$
Reefknot	0.357	0.073	0.313	0.119	0.266	0.196	-0.348	0.081
PMC-VQA	0.432	0.027	0.406	0.040	0.559	0.003	-0.587	0.002

Table 2: Spearman correlations (

\rho

) and

p

-values across datasets.

Figure 3: Family-level strategies: median

S

versus median

U_{2}

per family, computed across model sizes within each task regime. Points show the family medians for each regime. Outliers (InstructBLIP, Fuyu) are omitted for clarity.

Family	Sizes (B)	S $\to$ M (%)			M $\to$ VL (%)
		$\Delta$ Acc	$\Delta S$	$\Delta U_{2}$	$\Delta$ Acc	$\Delta S$	$\Delta U_{2}$
LLaVA-OneVision	0.5 $\to$ 7 $\to$ 72	11.9	11.9	-6.5	3.1	14.8	-9.7
Qwen2-VL	2 $\to$ 7 $\to$ 72	5.7	0.6	8.4	-3.9	0.4	-0.6
Qwen2.5-VL	3 $\to$ 7 $\to$ 72	1.3	-6.3	1.0	2.5	5.5	2.5
InternVL2.5	2 $\to$ 8 $\to$ 78	7.3	36.8	-55.6	3.6	10.6	3.8
InternVL3	2 $\to$ 8 $\to$ 78	2.7	2.5	-6.2	6.4	4.6	-10.3

Table 3: Scaling on synergy-driven tasks: changes in accuracy (

\Delta

Acc) and PID shares (

\Delta S

\Delta U_{2}

) for S

\to

M and M

\to

VL within representative families.

Figure 4: Layer-wise PID dynamics for representative models on synergy-driven (MMBench, top) and knowledge-driven (PMC-VQA, bottom) tasks. A consistent three-phase pattern appears across models and datasets.

Figure 5: Evolution of

S

and

U_{2}

during two-stage training of LLaVA-1.5 (7B, 13B).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Feature summarization			Confidence threshold $\tau$
	Mean (ours)	Last-hidden	Max-pool	0.3 (ours)	0.2	0.4
Qwen2.5-VL-7B	1.112	1.112	1.112	1.112	1.112	1.112
Qwen2.5-VL-72B	1.088	1.088	1.088	1.088	1.088	1.088
Gemma3-4B	0.167	0.172	0.173	0.167	0.167	0.167
Cambrian-34B	0.630	0.637	0.630	0.630	0.606	0.630

Appendix

Table 4:

S

on MMBench for four chosen models under two ablations (feature summarization and confidence threshold).

Model	Feature summarization			Confidence threshold $\tau$
	Mean (ours)	Last-hidden	Max-pool	0.3 (ours)	0.2	0.4
Qwen2.5-VL-7B	0.665	0.665	0.665	0.665	0.665	0.665
Qwen2.5-VL-72B	0.893	0.893	0.893	0.893	0.893	0.893
Gemma3-4B	1.864	1.864	1.864	1.864	1.864	1.864
Cambrian-34B	0.698	0.698	0.698	0.698	0.698	0.698

Appendix

Table 5:

U_{2}

on PMC-VQA for four chosen models under two ablations (feature summarization and confidence threshold).

Hyperparameter	Value
Learning rate	1e-3
Optimizer	Adam
Number of epochs	8
Network architecture	3-layer MLP
Hidden dimension	32
Activation function	ReLU

Appendix

Table 6: Hyperparameters for the BATCH Estimator.

Figure 6: PID analysis for a synergy-driven task. All models answer correctly, but the PID results reveal two distinct solution strategies: generating high synergy (Llama-3.2-vision, Qwen2.5-VL) versus correcting a strong, incorrect language prior (Gemma3).

Figure 7: PID analysis for a knowledge-driven task. While all models rely heavily on language uniqueness $U_{2}$ , fusion-centric models like Llama-3.2-vision and Qwen2.5-VL also generate non-trivial synergy, unlike the language-centric Gemma.

Figure 8: Full information spectrum and accuracy on the MMBench dataset. This figure provides detailed evidence for the synergy-driven regime. It visually confirms the existence of two primary model strategies: fusion-centric families (e.g., LLaVA-ov, Qwen2.5-VL, InternVL3) show a large share of synergy

S

, while language-centric families (e.g., Gemma3) are dominated by language uniqueness

U_{2}

. Furthermore, the plot illustrates that scaling models tends to increase synergy

S

within many families.

Figure 9: Full information spectrum and accuracy on the POPE dataset. As a synergy-driven task, POPE shows that synergy

S

is a key component for many high-performing models. Uniquely, this dataset also elicits significant redundancy

R

(green bars). This is likely because the task uses simple binary questions about common objects (from COCO ( Lin et al., 2014 ) ). In this context, both the visual modality (seeing the object) and the language modality (understanding the object’s name) can independently confirm the object’s presence, leading to high informational overlap.

Figure 10: Full information spectrum and accuracy on Reefknot. This plot exemplifies the knowledge-driven regime. Language uniqueness

U_{2}

is the overwhelmingly dominant information component for nearly all models, demonstrating that performance is constrained by language-side priors. Even in this environment, the fundamental strategies of model families persist: fusion-centric models (e.g., Llama-3.2-v) still generate noticeably more synergy

S

than their language-centric counterparts (e.g., Gemma3 and Cambrian).

Figure 11: Full information spectrum and accuracy on PMC-VQA. As a specialized, domain-specific task, this dataset clearly exemplifies the knowledge-driven regime. Similar to Reefknot, performance is heavily dominated by language uniqueness

U_{2}

, confirming that models must rely on internal, language-based knowledge for specialized topics. This result strongly supports our finding that for domain-specific tasks, performance is fundamentally limited by a model’s internal, language-based knowledge. The negligible synergy

S

in many top models highlights the challenge of multimodal fusion when highly specific domain knowledge is required.

Figure 12: Layer-wise PID dynamics for InternVL3-2B and Qwen2.5-VL-3B. The plot for Qwen2.5-VL-3B clearly illustrates the standard three-phase reasoning process: after information emerges, the later layers show a phase of representation building (rising

U_{2}

), which culminates in a decisive fusion event at the final layer (a spike in

S

and a drop in

U_{2}

). In contrast, InternVL3-2B lacks the final fusion event; its language uniqueness

U_{2}

continues to rise into the final layer without the characteristic drop. This may be due to its relatively smaller and shallower LLM, which might not have the capacity for the distinct final-layer fusion seen in other models.

Figure 13: Layer-wise PID dynamics for LLaVA-1.5-7B and LLaVA-1.5-13B. The LLaVA-1.5 family also exhibits the three-phase reasoning process, though with slightly different characteristics. Information emerges in the middle layers of the network. The representation building phase is distinct, with language uniqueness

U_{2}

rising to a high plateau while synergy

S

forms a noticeable “hump,” suggesting an ongoing fusion process prior to the final layer. The process concludes with the characteristic fusion event, marked by a drop in

U_{2}

and a final spike in

S

at the output layer. Unlike the other models, LLaVA-1.5 shows some minor, non-zero activity for redundancy

R

and vision uniqueness

U_{1}

in its later layers.

Figure 14: Learning dynamics of LLaVA-1.5-7B. This figure provides the evidence for the first part of Finding 6, showing how smaller models develop fusion. During Stage 1 (Pre-training), all PID components are negligible. Upon commencing Stage 2 (Visual Instruction Tuning), there is a dramatic and sustained increase in synergy

S

, which becomes the dominant information component by the end of training. Language uniqueness

U_{2}

also increases but to a much lesser extent, confirming that the 7B model primarily prioritizes developing synergistic inference.

Figure 15: Learning dynamics of LLaVA-1.5-13B. In contrast to the 7B model, this figure illustrates the second part of Finding 6. While PID values are also flat during Stage 1, the larger 13B model exhibits a massive and continuous increase in language uniqueness

U_{2}

during Stage 2, becoming by far the dominant information component. Although synergy

S

also grows, its increase is less pronounced than that of

U_{2}

, demonstrating that larger models prioritize enhancing their language-side priors during visual instruction tuning.

Explore similar work

Large Vision-Language Models Get Lost in Attention
May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNs Vision-Language Models
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3
Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
Mar 30, 2026Surendra Pathak, Bo HanEfficient VLM Inference Visual Token Compression
Computer Science Department, George Mason University, Fairfax, VA - 22030, USA
How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning
Feb 17, 2026Hongxuan Wu, Yukun Zhang, Xueqing ZhouVision-Language Models VLM Interpretability
Duke kunshan University, Duke University · The Chinese University of Hong Kong, Hong Kong, China · Fudan University, Shanghai, China

May 7, 2026J/K move · Enter open · S save

Large Vision-Language Models Get Lost in Attention

Gongli Xi, Ye Tian, Mengyu Yang, Huahui Yi+4

School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China+3

Transformer FFNs Vision-Language Models Transformer Self-Attention Large Vision-Language Models VLM Interpretability+1

Open paper AskPDF arXiv

Despite the rapid evolution of training paradigms, the decoder backbone of large vision--language models (LVLMs) remains fundamentally rooted in the residual-connection Transformer architecture. Therefore, deciphering the distinct roles of internal modules is critical for understanding model mechanics and guiding architectural optimization. While prior statistical approaches have provided valuable attribution-based insights, they often lack a unified theoretical basis. To bridge this gap, we propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates. Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-preserving operator focused on reconfiguration, whereas FFNs serve as subspace-expanding operators driving semantic innovation. Strikingly, further experiments demonstrate that replacing learned attention weights with predefined values (e.g., Gaussian noise) yields comparable or even superior performance across a majority of datasets relative to vanilla models. These results expose severe misallocation and redundancy in current mechanisms, suggesting that state-of-the-art LVLMs effectively ``get lost in attention'' rather than efficiently leveraging visual context.

Abstract

Explore similar work

Large Vision-Language Models Get Lost in Attention

Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies

How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning