How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning

Hongxuan Wu; Yukun Zhang; Xueqing Zhou

cs.AIFeb 17, 2026

How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning

Authors: Hongxuan Wu, Yukun Zhang, Xueqing Zhou

Organizations: Duke kunshan University, Duke University · The Chinese University of Hong Kong, Hong Kong, China · Fudan University, Shanghai, China

Abstract

When a multimodal Transformer answers a visual question, is the prediction driven by visual evidence, linguistic reasoning, or genuinely fused cross-modal computation -- and how does this structure evolve across layers? We address this question with a layer-wise framework based on Partial Information Decomposition (PID) that decomposes the predictive information at each Transformer layer into redundant, vision-unique, language-unique, and synergistic components. To make PID tractable for high-dimensional neural representations, we introduce PID Flow, a pipeline combining dimensionality reduction, normalizing-flow Gaussianization, and closed-form Gaussian PID estimation. Applying this framework to LLaVA-1.5-7B and LLaVA-1.6-7B across six GQA reasoning tasks, we uncover a consistent modal transduction pattern: visual-unique information peaks early and decays with depth, language-unique information surges in late layers to account for roughly 82% of the final prediction, and cross-modal synergy remains below 2%. This trajectory is highly stable across model variants (layer-wise correlations $>$ 0.96) yet strongly task-dependent, with semantic redundancy governing the detailed information fingerprint. To establish causality, we perform targeted Image $\rightarrow$ Question attention knockouts and show that disrupting the primary transduction pathway induces predictable increases in trapped visual-unique information, compensatory synergy, and total information cost -- effects that are strongest in vision-dependent tasks and weakest in high-redundancy tasks. Together, these results provide an information-theoretic, causal account of how vision becomes language in multimodal Transformers, and offer quantitative guidance for identifying architectural bottlenecks where modality-specific information is lost.

Figures & tables

Figure 1: Core research framework. (A) Multimodal input: image and question tokens are processed by a 32-layer LLaVA Transformer. (B) PID Flow estimation: layer-wise representations are compressed via mean pooling and PCA, Gaussianized by a normalizing flow, and decomposed into four PID components ( $R,U_{V},U_{L},S$ ). (C) Information trajectories reveal a three-stage modal transduction pattern: visual injection (Stage I), consolidation (Stage II), and language-dominant decision formation (Stage III), with $U_{L}\approx 82\%$ and $S<2\%$ at the final layer. Cross-model trajectory correlations exceed $r>0.96$ .

Component	Information-theoretic meaning	Multimodal interpretation
$R$	shared predictive information	aligned vision–language concepts
$U_{V}$	information unique to vision	visual attributes (color, position, texture)
$U_{L}$	information unique to language	linguistic or logical structure
$S$	information available only jointly	cross-modal binding (e.g., object–attribute relations)

Table 1: Semantic interpretation of PID components in multimodal models.

Component	Symbol	Interpretation (MLLMs)
Redundancy	$R^{(\ell)}$	predictive information jointly encoded by both modalities
Vision-unique	$U_{V}^{(\ell)}$	evidence available only from the visual representation
Language-unique	$U_{L}^{(\ell)}$	evidence available only from the language representation
Synergy	$S^{(\ell)}$	evidence available only from the joint pair (PID sense)

Table 2: PID components at layer

\ell

and their interpretation in MLLMs.

Panel of Figure 2 — Figure 2: Layer-wise PID trajectories for LLaVA-1.5-7B across six tasks. Each panel plots redundancy ( $R^{\ell}$ ), vision-unique ( $U_{V}^{\ell}$ ), language-unique ( $U_{L}^{\ell}$ ), and synergy ( $S^{\ell}$ ) as functions of Transformer depth. All tasks share a consistent pattern: $U_{V}$ peaks early and decays, $U_{L}$ surges in late layers, and $S$ remains bounded throughout—the signature of modal transduction .

Task	$R$	$U_{L}$	$U_{V}$	$S$	$U_{L}/\text{Total}$
ChooseAttr	1.13	14.18	2.01	1.71	74.5%
ChooseCat	7.74	15.40	5.31	3.54	48.1%
ChooseRel	0.13	18.83	0.50	0.65	93.6%
CompareAttr	0.22	24.54	0.01	0.83	95.9%
LogicalObj	0.07	12.62	0.58	0.35	92.7%
QueryAttr	0.03	22.70	0.00	0.59	97.3%

Table 3: Final-layer (Layer 31) PID decomposition for LLaVA-1.5-7B .

Task	Peak of $U_{V}$	Trough of $U_{L}$	Onset of $U_{L}$ surge
ChooseAttr	0	13	20
ChooseCat	1	14	19
ChooseRel	0	11	19
CompareAttr	1	–	17
LogicalObj	0	13	19
QueryAttr	1	13	20

Table 4: Cross-task consistency of modal-transduction turning points.

Figure 3: Layer-wise PID trajectories for LLaVA-1.5-7B across six tasks. All tasks exhibit decay of $U_{V}$ , a late-stage surge of $U_{L}$ , and bounded synergy $S$ , supporting modal transduction as the dominant mechanism.

Figure 4: Turning layers and final-layer composition. Transition points ( $U_{V}$ peak, $U_{L}$ trough, $U_{L}$ surge onset) cluster across tasks, and final-layer predictive information is dominated by $U_{L}$ .

Panel of Figure 5 — Figure 5: Layer-wise PID trajectories for LLaVA-1.6-7B across six tasks. The depth-wise pattern mirrors LLaVA-1.5-7B (Figure 2 ): $U_{V}$ decays, $U_{L}$ surges late, and synergy remains bounded, confirming modal transduction in an architecturally distinct model.

Task	Model	$R$	$U_{L}$	$U_{V}$	$S$	Total	$U_{L}$ Share	$U_{L}/U_{V}$
ChooseAttr	1.5-7b	1.13	14.18	2.01	1.71	19.03	74.5%	$7.05\times$
	1.6-7b	1.27	13.73	2.33	1.80	19.13	71.8%	$5.89\times$
	Diff.	+12%	-3.2%	+16%	+5.3%	+0.5%	-2.7%	-16%
ChooseCat	1.5-7b	7.74	15.40	5.31	3.54	31.99	48.1%	$2.90\times$
	1.6-7b	7.69	14.17	5.20	3.30	30.36	46.7%	$2.73\times$
	Diff.	-0.6%	-8.0%	-2.1%	-6.8%	-5.1%	-1.4%	-5.9%

Table 5: LLaVA-1.5-7B vs. LLaVA-1.6-7B : final-layer (Layer 31) PID comparison.

Figure 6: Final-layer PID composition across models. $U_{L}$ dominates at Layer 31 in both LLaVA-1.5-7B and LLaVA-1.6-7B , while synergy remains bounded.

Figure 7: Cross-model trajectory alignment. Solid: LLaVA-1.5-7B ; dashed: LLaVA-1.6-7B . Depth-wise dynamics of all four PID components are closely synchronised.

Task	$r(R)$	$r(U_{L})$	$r(U_{V})$	$r(S)$	Mean
ChooseAttr	0.968	0.993	0.978	0.941	0.970
ChooseCat	0.989	0.981	0.953	0.968	0.973
ChooseRel	0.972	0.996	0.945	0.892	0.951
CompareAttr	0.963	0.987	0.968	0.913	0.958
LogicalObj	0.981	0.974	0.972	0.925	0.963
QueryAttr	0.984	0.959	0.948	0.923	0.954

Table 6: Cross-model PID trajectory correlation (Pearson

r

across 32 layers).

Task	Turning point	1.5-7B	1.6-7B	Diff.
ChooseAttr	$U_{V}$ peak	1	0	$-1$
	$U_{L}$ trough	14	15	$+1$
	$U_{L}$ surge onset	20	19	$-1$
ChooseCat	$U_{V}$ peak	0	1	$+1$
	$U_{L}$ trough	13	15	$+2$
	$U_{L}$ surge onset	19	18	$-1$

Table 7: Cross-model comparison of key turning points.

Task	Condition	$R$	$U_{L}$	$U_{V}$	$S$	Total	$U_{L}/U_{V}$	$\Delta\,\text{Total}$
ChooseRel	Normal	0.15	21.48	0.72	0.85	23.20	$29.8\times$	–
(High vision dep.)	Knockout	0.15	23.14	0.85	1.07	25.20	$27.2\times$	+8.6%
	$\Delta$	0%	+7.7%	+18.1%	+25.9%	–	-8.7%	–
LogicalObj	Normal	0.07	14.09	0.84	0.45	15.45	$16.8\times$	–
(High vision dep.)	Knockout	0.06	14.63	0.90	0.49	16.09	$16.3\times$	+4.1%
	$\Delta$	-14%	+3.8%	+7.7%	+9.8%	–	-3.0%	–

Table 8: PID effects of Image

\rightarrow

Question attention knockout at Layer 31 on LLaVA-1.6-7B . Values are in bits;

\Delta:=(\text{knockout}-\text{normal})/\text{normal}\times 100\%

. Bold entries mark changes exceeding

5\%

or mechanism-critical shifts.

Figure 8: Final-layer effects of Image $\rightarrow$ Question attention knockout. Vision-dependent tasks ( ChooseRel , LogicalObj ) show the strongest PID shifts; high-redundancy tasks ( ChooseAttr , ChooseCat ) are largely robust.

Figure 9: Layer-wise knockout effects for ChooseRel . Solid: normal; dashed: knockout. Blocking Image $\rightarrow$ Question attention traps visual evidence (elevated $U_{V}$ ), delays $U_{L}$ growth, and forces late-layer compensation at higher total-information cost.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Description
$V$	Vision modality	Visual input stream (image tokens)
$L$	Language modality	Linguistic input stream (question tokens)
$Y$	Target variable	Prediction target (logit or probability of correct answer)
$\ell$	Layer index	Transformer layer, $\ell\in\{0,\dots,L\}$
$L$	Model depth	Total number of Transformer layers (32 in LLaVA)
$\mathbf{x}_{V}^{(\ell)}$	Vision representation	Aggregated visual representation at layer $\ell$

Appendix

Table 9: Symbol table for key notation.

Explore similar work

May 31, 2026

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment. We introduce Partial Information Decomposition (PID) as a decision-level framework that separates unique, redundant, and synergistic contributions of sensory and linguistic inputs, beyond representation alignment and outcome-based evaluation. Across vision--language benchmarks, PID reveals recurring modality-use profiles: reasoning and grounding-oriented tasks tend to exhibit high synergy, whereas expert and knowledge-oriented tasks show stronger language-unique reliance. These profiles generalize across model families and predict sensitivity to modality-level interventions. We further extend PID to tri-modal systems with Sensory PID, treating language as a control variable to decompose video--audio information gain. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck dominated by visual information even on audio--visual fusion tasks. Finally, PID-guided reweighting provides initial evidence for improving multimodal reasoning and grounding performance.

Wanlong Fang, Tianle Zhang, Wen Tao +1

College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore

Multimodal Grounding Multimodal Large Language Models Multimodal Reasoning Multimodal Foundation Models Multimodal Model Interpretability Multimodal Fusion+1

Ask PDF

Mar 31, 2026

A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

Large vision-language models (LVLMs) achieve impressive performance, yet their internal decision-making processes remain opaque, making it difficult to determine if the success stems from true multimodal fusion or from reliance on unimodal priors. To address this attribution gap, we introduce a novel framework using partial information decomposition (PID) to quantitatively measure the "information spectrum" of LVLMs -- decomposing a model's decision-relevant information into redundant, unique, and synergistic components. By adapting a scalable estimator to modern LVLM outputs, our model-agnostic pipeline profiles 26 LVLMs on four datasets across three dimensions -- breadth (cross-model & cross-task), depth (layer-wise information dynamics), and time (learning dynamics across training). Our analysis reveals two key results: (i) two task regimes (synergy-driven vs. knowledge-driven) and (ii) two stable, contrasting family-level strategies (fusion-centric vs. language-centric). We also uncover a consistent three-phase pattern in layer-wise processing and identify visual instruction tuning as the key stage where fusion is learned. Together, these contributions provide a quantitative lens beyond accuracy-only evaluation and offer insights for analyzing and designing the next generation of LVLMs. Code and data are available at https://github.com/RiiShin/pid-lvlm-analysis .

Lixin Xiu, Xufang Luo, Hideki Nakayama

The University of Tokyo · Microsoft Research

Large Vision-Language Models VLM Interpretability Multimodal Fusion Multimodal Instruction Tuning

Ask PDF

Sep 4, 2026

From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making

Vision-Language Models are commonly evaluated through their final predictions, but understanding whether these decisions are grounded in visual evidence requires tracing how visual information contributes to language-based decisions. With this purpose in mind, we investigate cross-modal information flow in a video-based generative multiple-choice-like setting by applying a layer-wise causal intervention on video-text attention pathways. We target spatial, causal, and temporal visual reasoning. Our results show that visual information is mainly integrated while the model processes the candidate answer options, which serve as the primary textual grounding sites for the final decision. We further show that nouns play an important role as semantic anchors during multimodal enrichment, while verbs are more relevant when temporal relations are processed. Finally, we identify a distinct pattern in temporal reasoning, suggesting that VLMs struggle to reconstruct sequential information across video frames, but we remark that such fragility may also reflect linguistic biases associated with specific temporal expressions used for defining the relation between events within a scene.

Davide Testa, Hugh Mee Wong, Alessandro Lenci +2

Fondazione Bruno Kessler (FBK) - Trento, Italy · Università di Roma La Sapienza - Rome, Italy · Utrecht University - Utrecht, The Netherlands +1

Vision-Language Models VLM Reasoning VLM Interpretability

Ask PDF