A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

Lixin Xiu; Xufang Luo; Hideki Nakayama

cs.LGMar 31, 2026

A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

Authors: Lixin Xiu, Xufang Luo, Hideki Nakayama

Organizations: The University of Tokyo · Microsoft Research

Abstract

Large vision-language models (LVLMs) achieve impressive performance, yet their internal decision-making processes remain opaque, making it difficult to determine if the success stems from true multimodal fusion or from reliance on unimodal priors. To address this attribution gap, we introduce a novel framework using partial information decomposition (PID) to quantitatively measure the "information spectrum" of LVLMs -- decomposing a model's decision-relevant information into redundant, unique, and synergistic components. By adapting a scalable estimator to modern LVLM outputs, our model-agnostic pipeline profiles 26 LVLMs on four datasets across three dimensions -- breadth (cross-model & cross-task), depth (layer-wise information dynamics), and time (learning dynamics across training). Our analysis reveals two key results: (i) two task regimes (synergy-driven vs. knowledge-driven) and (ii) two stable, contrasting family-level strategies (fusion-centric vs. language-centric). We also uncover a consistent three-phase pattern in layer-wise processing and identify visual instruction tuning as the key stage where fusion is learned. Together, these contributions provide a quantitative lens beyond accuracy-only evaluation and offer insights for analyzing and designing the next generation of LVLMs. Code and data are available at https://github.com/RiiShin/pid-lvlm-analysis .

Figures & tables

Figure 1: Overview of this research. The first part is the framework of PID estimation for LVLMs. Given an image-text pair, we extract image and text embeddings as two features, run a standard multimodal forward pass and collect two unimodal predictions by masking the other modality. PID values are estimated with BATCH estimator. The second part reveals three analysis dimensions: (1) cross-model and cross-task comparison, (2) layer-wise information dynamics, and (3) learning dynamics over training. To our knowledge, this is the first comprehensive LVLM analysis through the lens of information decomposition.

Dataset	Task	# of options	# of training samples	# of test samples
MMBench	General visual reasoning	2–4	3246	1083
POPE	Hallucination evaluation	2	2250	750
Reefknot	Hallucination evaluation	4	1612	538
PMC-VQA	Domain-specific knowledge	4	1500	500

Table 1: Details of the datasets used for evaluation. The listed training and test splits are not for LVLM fine-tuning; they are created by randomly partitioning each dataset (3:1 ratio) for the PID estimation, as the BATCH estimator requires separate sets to train networks and estimate PID values.

Figure 2: Share of synergy

S

and language uniqueness

U_{2}

across four datasets.

Dataset	$S$		$U_{2}$		$I(X_{1},X_{2};Y)$		$I(X_{1};X_{2};Y)$
	$\rho$	$p$ -val	$\rho$	$p$ -val	$\rho$	$p$ -val	$\rho$	$p$ -val
MMBench	0.750	$<0.001$	0.194	0.343	0.632	$<0.001$	-0.757	$<0.001$
POPE	0.742	$<0.001$	-0.009	0.964	0.157	0.445	-0.701	$<0.001$
Reefknot	0.357	0.073	0.313	0.119	0.266	0.196	-0.348	0.081
PMC-VQA	0.432	0.027	0.406	0.040	0.559	0.003	-0.587	0.002

Table 2: Spearman correlations (

\rho

) and

p

-values across datasets.

Figure 3: Family-level strategies: median

S

versus median

U_{2}

per family, computed across model sizes within each task regime. Points show the family medians for each regime. Outliers (InstructBLIP, Fuyu) are omitted for clarity.

Family	Sizes (B)	S $\to$ M (%)			M $\to$ VL (%)
		$\Delta$ Acc	$\Delta S$	$\Delta U_{2}$	$\Delta$ Acc	$\Delta S$	$\Delta U_{2}$
LLaVA-OneVision	0.5 $\to$ 7 $\to$ 72	11.9	11.9	-6.5	3.1	14.8	-9.7
Qwen2-VL	2 $\to$ 7 $\to$ 72	5.7	0.6	8.4	-3.9	0.4	-0.6
Qwen2.5-VL	3 $\to$ 7 $\to$ 72	1.3	-6.3	1.0	2.5	5.5	2.5
InternVL2.5	2 $\to$ 8 $\to$ 78	7.3	36.8	-55.6	3.6	10.6	3.8
InternVL3	2 $\to$ 8 $\to$ 78	2.7	2.5	-6.2	6.4	4.6	-10.3

Table 3: Scaling on synergy-driven tasks: changes in accuracy (

\Delta

Acc) and PID shares (

\Delta S

\Delta U_{2}

) for S

\to

M and M

\to

VL within representative families.

Figure 4: Layer-wise PID dynamics for representative models on synergy-driven (MMBench, top) and knowledge-driven (PMC-VQA, bottom) tasks. A consistent three-phase pattern appears across models and datasets.

Figure 5: Evolution of

S

and

U_{2}

during two-stage training of LLaVA-1.5 (7B, 13B).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Feature summarization			Confidence threshold $\tau$
	Mean (ours)	Last-hidden	Max-pool	0.3 (ours)	0.2	0.4
Qwen2.5-VL-7B	1.112	1.112	1.112	1.112	1.112	1.112
Qwen2.5-VL-72B	1.088	1.088	1.088	1.088	1.088	1.088
Gemma3-4B	0.167	0.172	0.173	0.167	0.167	0.167
Cambrian-34B	0.630	0.637	0.630	0.630	0.606	0.630

Appendix

Table 4:

S

on MMBench for four chosen models under two ablations (feature summarization and confidence threshold).

Model	Feature summarization			Confidence threshold $\tau$
	Mean (ours)	Last-hidden	Max-pool	0.3 (ours)	0.2	0.4
Qwen2.5-VL-7B	0.665	0.665	0.665	0.665	0.665	0.665
Qwen2.5-VL-72B	0.893	0.893	0.893	0.893	0.893	0.893
Gemma3-4B	1.864	1.864	1.864	1.864	1.864	1.864
Cambrian-34B	0.698	0.698	0.698	0.698	0.698	0.698

Appendix

Table 5:

U_{2}

on PMC-VQA for four chosen models under two ablations (feature summarization and confidence threshold).

Hyperparameter	Value
Learning rate	1e-3
Optimizer	Adam
Number of epochs	8
Network architecture	3-layer MLP
Hidden dimension	32
Activation function	ReLU

Appendix

Table 6: Hyperparameters for the BATCH Estimator.

Figure 6: PID analysis for a synergy-driven task. All models answer correctly, but the PID results reveal two distinct solution strategies: generating high synergy (Llama-3.2-vision, Qwen2.5-VL) versus correcting a strong, incorrect language prior (Gemma3).

Figure 7: PID analysis for a knowledge-driven task. While all models rely heavily on language uniqueness $U_{2}$ , fusion-centric models like Llama-3.2-vision and Qwen2.5-VL also generate non-trivial synergy, unlike the language-centric Gemma.

Figure 8: Full information spectrum and accuracy on the MMBench dataset. This figure provides detailed evidence for the synergy-driven regime. It visually confirms the existence of two primary model strategies: fusion-centric families (e.g., LLaVA-ov, Qwen2.5-VL, InternVL3) show a large share of synergy

S

, while language-centric families (e.g., Gemma3) are dominated by language uniqueness

U_{2}

. Furthermore, the plot illustrates that scaling models tends to increase synergy

S

within many families.

Figure 9: Full information spectrum and accuracy on the POPE dataset. As a synergy-driven task, POPE shows that synergy

S

is a key component for many high-performing models. Uniquely, this dataset also elicits significant redundancy

R

(green bars). This is likely because the task uses simple binary questions about common objects (from COCO ( Lin et al., 2014 ) ). In this context, both the visual modality (seeing the object) and the language modality (understanding the object’s name) can independently confirm the object’s presence, leading to high informational overlap.

Figure 10: Full information spectrum and accuracy on Reefknot. This plot exemplifies the knowledge-driven regime. Language uniqueness

U_{2}

is the overwhelmingly dominant information component for nearly all models, demonstrating that performance is constrained by language-side priors. Even in this environment, the fundamental strategies of model families persist: fusion-centric models (e.g., Llama-3.2-v) still generate noticeably more synergy

S

than their language-centric counterparts (e.g., Gemma3 and Cambrian).

Figure 11: Full information spectrum and accuracy on PMC-VQA. As a specialized, domain-specific task, this dataset clearly exemplifies the knowledge-driven regime. Similar to Reefknot, performance is heavily dominated by language uniqueness

U_{2}

, confirming that models must rely on internal, language-based knowledge for specialized topics. This result strongly supports our finding that for domain-specific tasks, performance is fundamentally limited by a model’s internal, language-based knowledge. The negligible synergy

S

in many top models highlights the challenge of multimodal fusion when highly specific domain knowledge is required.

Figure 12: Layer-wise PID dynamics for InternVL3-2B and Qwen2.5-VL-3B. The plot for Qwen2.5-VL-3B clearly illustrates the standard three-phase reasoning process: after information emerges, the later layers show a phase of representation building (rising

U_{2}

), which culminates in a decisive fusion event at the final layer (a spike in

S

and a drop in

U_{2}

). In contrast, InternVL3-2B lacks the final fusion event; its language uniqueness

U_{2}

continues to rise into the final layer without the characteristic drop. This may be due to its relatively smaller and shallower LLM, which might not have the capacity for the distinct final-layer fusion seen in other models.

Figure 13: Layer-wise PID dynamics for LLaVA-1.5-7B and LLaVA-1.5-13B. The LLaVA-1.5 family also exhibits the three-phase reasoning process, though with slightly different characteristics. Information emerges in the middle layers of the network. The representation building phase is distinct, with language uniqueness

U_{2}

rising to a high plateau while synergy

S

forms a noticeable “hump,” suggesting an ongoing fusion process prior to the final layer. The process concludes with the characteristic fusion event, marked by a drop in

U_{2}

and a final spike in

S

at the output layer. Unlike the other models, LLaVA-1.5 shows some minor, non-zero activity for redundancy

R

and vision uniqueness

U_{1}

in its later layers.

Figure 14: Learning dynamics of LLaVA-1.5-7B. This figure provides the evidence for the first part of Finding 6, showing how smaller models develop fusion. During Stage 1 (Pre-training), all PID components are negligible. Upon commencing Stage 2 (Visual Instruction Tuning), there is a dramatic and sustained increase in synergy

S

, which becomes the dominant information component by the end of training. Language uniqueness

U_{2}

also increases but to a much lesser extent, confirming that the 7B model primarily prioritizes developing synergistic inference.

Figure 15: Learning dynamics of LLaVA-1.5-13B. In contrast to the 7B model, this figure illustrates the second part of Finding 6. While PID values are also flat during Stage 1, the larger 13B model exhibits a massive and continuous increase in language uniqueness

U_{2}

during Stage 2, becoming by far the dominant information component. Although synergy

S

also grows, its increase is less pronounced than that of

U_{2}

, demonstrating that larger models prioritize enhancing their language-side priors during visual instruction tuning.

Explore similar work

May 7, 2026

Large Vision-Language Models Get Lost in Attention

Despite the rapid evolution of training paradigms, the decoder backbone of large vision--language models (LVLMs) remains fundamentally rooted in the residual-connection Transformer architecture. Therefore, deciphering the distinct roles of internal modules is critical for understanding model mechanics and guiding architectural optimization. While prior statistical approaches have provided valuable attribution-based insights, they often lack a unified theoretical basis. To bridge this gap, we propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates. Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-preserving operator focused on reconfiguration, whereas FFNs serve as subspace-expanding operators driving semantic innovation. Strikingly, further experiments demonstrate that replacing learned attention weights with predefined values (e.g., Gaussian noise) yields comparable or even superior performance across a majority of datasets relative to vanilla models. These results expose severe misallocation and redundancy in current mechanisms, suggesting that state-of-the-art LVLMs effectively ``get lost in attention'' rather than efficiently leveraging visual context.

Gongli Xi, Ye Tian, Mengyu Yang +5

School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3

Transformer FFNs Vision-Language Models Transformer Self-Attention Large Vision-Language Models VLM Interpretability+1

Ask PDF

Mar 30, 2026

Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies

Although Large Vision Language Models (LVLMs) have demonstrated impressive multimodal reasoning capabilities, their scalability and deployment are constrained by massive computational requirements. In particular, the massive amount of visual tokens from high-resolution input data aggravates the situation due to the quadratic complexity of attention mechanisms. To address these issues, the research community has developed several optimization frameworks. This paper presents a comprehensive survey of the current state-of-the-art techniques for accelerating LVLM inference. We introduce a systematic taxonomy that categorizes existing optimization frameworks into four primary dimensions: visual token compression, memory management and serving, efficient architectural design, and advanced decoding strategies. Furthermore, we critically examine the limitations of these current methodologies and identify critical open problems to inspire future research directions in efficient multimodal systems.

Surendra Pathak, Bo Han

Computer Science Department, George Mason University, Fairfax, VA - 22030, USA

Efficient VLM Inference Visual Token Compression

Ask PDF

Feb 17, 2026

How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning

When a multimodal Transformer answers a visual question, is the prediction driven by visual evidence, linguistic reasoning, or genuinely fused cross-modal computation -- and how does this structure evolve across layers? We address this question with a layer-wise framework based on Partial Information Decomposition (PID) that decomposes the predictive information at each Transformer layer into redundant, vision-unique, language-unique, and synergistic components. To make PID tractable for high-dimensional neural representations, we introduce PID Flow, a pipeline combining dimensionality reduction, normalizing-flow Gaussianization, and closed-form Gaussian PID estimation. Applying this framework to LLaVA-1.5-7B and LLaVA-1.6-7B across six GQA reasoning tasks, we uncover a consistent modal transduction pattern: visual-unique information peaks early and decays with depth, language-unique information surges in late layers to account for roughly 82% of the final prediction, and cross-modal synergy remains below 2%. This trajectory is highly stable across model variants (layer-wise correlations

>

0.96) yet strongly task-dependent, with semantic redundancy governing the detailed information fingerprint. To establish causality, we perform targeted Image

\rightarrow

Question attention knockouts and show that disrupting the primary transduction pathway induces predictable increases in trapped visual-unique information, compensatory synergy, and total information cost -- effects that are strongest in vision-dependent tasks and weakest in high-redundancy tasks. Together, these results provide an information-theoretic, causal account of how vision becomes language in multimodal Transformers, and offer quantitative guidance for identifying architectural bottlenecks where modality-specific information is lost.

Hongxuan Wu, Yukun Zhang, Xueqing Zhou

Duke kunshan University, Duke University · The Chinese University of Hong Kong, Hong Kong, China · Fudan University, Shanghai, China

Vision-Language Models VLM Interpretability Multimodal Reasoning

Ask PDF