How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning

Hongxuan Wu; Yukun Zhang; Xueqing Zhou

cs.AIFeb 17, 2026

How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning

Authors: Hongxuan Wu, Yukun Zhang, Xueqing Zhou

Organizations: Duke kunshan University, Duke University · The Chinese University of Hong Kong, Hong Kong, China · Fudan University, Shanghai, China

Abstract

When a multimodal Transformer answers a visual question, is the prediction driven by visual evidence, linguistic reasoning, or genuinely fused cross-modal computation -- and how does this structure evolve across layers? We address this question with a layer-wise framework based on Partial Information Decomposition (PID) that decomposes the predictive information at each Transformer layer into redundant, vision-unique, language-unique, and synergistic components. To make PID tractable for high-dimensional neural representations, we introduce PID Flow, a pipeline combining dimensionality reduction, normalizing-flow Gaussianization, and closed-form Gaussian PID estimation. Applying this framework to LLaVA-1.5-7B and LLaVA-1.6-7B across six GQA reasoning tasks, we uncover a consistent modal transduction pattern: visual-unique information peaks early and decays with depth, language-unique information surges in late layers to account for roughly 82% of the final prediction, and cross-modal synergy remains below 2%. This trajectory is highly stable across model variants (layer-wise correlations $>$ 0.96) yet strongly task-dependent, with semantic redundancy governing the detailed information fingerprint. To establish causality, we perform targeted Image $\rightarrow$ Question attention knockouts and show that disrupting the primary transduction pathway induces predictable increases in trapped visual-unique information, compensatory synergy, and total information cost -- effects that are strongest in vision-dependent tasks and weakest in high-redundancy tasks. Together, these results provide an information-theoretic, causal account of how vision becomes language in multimodal Transformers, and offer quantitative guidance for identifying architectural bottlenecks where modality-specific information is lost.

Figures & tables

Figure 1: Core research framework. (A) Multimodal input: image and question tokens are processed by a 32-layer LLaVA Transformer. (B) PID Flow estimation: layer-wise representations are compressed via mean pooling and PCA, Gaussianized by a normalizing flow, and decomposed into four PID components ( $R,U_{V},U_{L},S$ ). (C) Information trajectories reveal a three-stage modal transduction pattern: visual injection (Stage I), consolidation (Stage II), and language-dominant decision formation (Stage III), with $U_{L}\approx 82\%$ and $S<2\%$ at the final layer. Cross-model trajectory correlations exceed $r>0.96$ .

Component	Information-theoretic meaning	Multimodal interpretation
$R$	shared predictive information	aligned vision–language concepts
$U_{V}$	information unique to vision	visual attributes (color, position, texture)
$U_{L}$	information unique to language	linguistic or logical structure
$S$	information available only jointly	cross-modal binding (e.g., object–attribute relations)

Table 1: Semantic interpretation of PID components in multimodal models.

Component	Symbol	Interpretation (MLLMs)
Redundancy	$R^{(\ell)}$	predictive information jointly encoded by both modalities
Vision-unique	$U_{V}^{(\ell)}$	evidence available only from the visual representation
Language-unique	$U_{L}^{(\ell)}$	evidence available only from the language representation
Synergy	$S^{(\ell)}$	evidence available only from the joint pair (PID sense)

Table 2: PID components at layer

\ell

and their interpretation in MLLMs.

Panel of Figure 2 — Figure 2: Layer-wise PID trajectories for LLaVA-1.5-7B across six tasks. Each panel plots redundancy ( $R^{\ell}$ ), vision-unique ( $U_{V}^{\ell}$ ), language-unique ( $U_{L}^{\ell}$ ), and synergy ( $S^{\ell}$ ) as functions of Transformer depth. All tasks share a consistent pattern: $U_{V}$ peaks early and decays, $U_{L}$ surges in late layers, and $S$ remains bounded throughout—the signature of modal transduction .

Task	$R$	$U_{L}$	$U_{V}$	$S$	$U_{L}/\text{Total}$
ChooseAttr	1.13	14.18	2.01	1.71	74.5%
ChooseCat	7.74	15.40	5.31	3.54	48.1%
ChooseRel	0.13	18.83	0.50	0.65	93.6%
CompareAttr	0.22	24.54	0.01	0.83	95.9%
LogicalObj	0.07	12.62	0.58	0.35	92.7%
QueryAttr	0.03	22.70	0.00	0.59	97.3%

Table 3: Final-layer (Layer 31) PID decomposition for LLaVA-1.5-7B .

Task	Peak of $U_{V}$	Trough of $U_{L}$	Onset of $U_{L}$ surge
ChooseAttr	0	13	20
ChooseCat	1	14	19
ChooseRel	0	11	19
CompareAttr	1	–	17
LogicalObj	0	13	19
QueryAttr	1	13	20

Table 4: Cross-task consistency of modal-transduction turning points.

Figure 3: Layer-wise PID trajectories for LLaVA-1.5-7B across six tasks. All tasks exhibit decay of $U_{V}$ , a late-stage surge of $U_{L}$ , and bounded synergy $S$ , supporting modal transduction as the dominant mechanism.

Figure 4: Turning layers and final-layer composition. Transition points ( $U_{V}$ peak, $U_{L}$ trough, $U_{L}$ surge onset) cluster across tasks, and final-layer predictive information is dominated by $U_{L}$ .

Panel of Figure 5 — Figure 5: Layer-wise PID trajectories for LLaVA-1.6-7B across six tasks. The depth-wise pattern mirrors LLaVA-1.5-7B (Figure 2 ): $U_{V}$ decays, $U_{L}$ surges late, and synergy remains bounded, confirming modal transduction in an architecturally distinct model.

Task	Model	$R$	$U_{L}$	$U_{V}$	$S$	Total	$U_{L}$ Share	$U_{L}/U_{V}$
ChooseAttr	1.5-7b	1.13	14.18	2.01	1.71	19.03	74.5%	$7.05\times$
	1.6-7b	1.27	13.73	2.33	1.80	19.13	71.8%	$5.89\times$
	Diff.	+12%	-3.2%	+16%	+5.3%	+0.5%	-2.7%	-16%
ChooseCat	1.5-7b	7.74	15.40	5.31	3.54	31.99	48.1%	$2.90\times$
	1.6-7b	7.69	14.17	5.20	3.30	30.36	46.7%	$2.73\times$
	Diff.	-0.6%	-8.0%	-2.1%	-6.8%	-5.1%	-1.4%	-5.9%

Table 5: LLaVA-1.5-7B vs. LLaVA-1.6-7B : final-layer (Layer 31) PID comparison.

Figure 6: Final-layer PID composition across models. $U_{L}$ dominates at Layer 31 in both LLaVA-1.5-7B and LLaVA-1.6-7B , while synergy remains bounded.

Figure 7: Cross-model trajectory alignment. Solid: LLaVA-1.5-7B ; dashed: LLaVA-1.6-7B . Depth-wise dynamics of all four PID components are closely synchronised.

Task	$r(R)$	$r(U_{L})$	$r(U_{V})$	$r(S)$	Mean
ChooseAttr	0.968	0.993	0.978	0.941	0.970
ChooseCat	0.989	0.981	0.953	0.968	0.973
ChooseRel	0.972	0.996	0.945	0.892	0.951
CompareAttr	0.963	0.987	0.968	0.913	0.958
LogicalObj	0.981	0.974	0.972	0.925	0.963
QueryAttr	0.984	0.959	0.948	0.923	0.954

Table 6: Cross-model PID trajectory correlation (Pearson

r

across 32 layers).

Task	Turning point	1.5-7B	1.6-7B	Diff.
ChooseAttr	$U_{V}$ peak	1	0	$-1$
	$U_{L}$ trough	14	15	$+1$
	$U_{L}$ surge onset	20	19	$-1$
ChooseCat	$U_{V}$ peak	0	1	$+1$
	$U_{L}$ trough	13	15	$+2$
	$U_{L}$ surge onset	19	18	$-1$

Table 7: Cross-model comparison of key turning points.

Task	Condition	$R$	$U_{L}$	$U_{V}$	$S$	Total	$U_{L}/U_{V}$	$\Delta\,\text{Total}$
ChooseRel	Normal	0.15	21.48	0.72	0.85	23.20	$29.8\times$	–
(High vision dep.)	Knockout	0.15	23.14	0.85	1.07	25.20	$27.2\times$	+8.6%
	$\Delta$	0%	+7.7%	+18.1%	+25.9%	–	-8.7%	–
LogicalObj	Normal	0.07	14.09	0.84	0.45	15.45	$16.8\times$	–
(High vision dep.)	Knockout	0.06	14.63	0.90	0.49	16.09	$16.3\times$	+4.1%
	$\Delta$	-14%	+3.8%	+7.7%	+9.8%	–	-3.0%	–

Table 8: PID effects of Image

\rightarrow

Question attention knockout at Layer 31 on LLaVA-1.6-7B . Values are in bits;

\Delta:=(\text{knockout}-\text{normal})/\text{normal}\times 100\%

. Bold entries mark changes exceeding

5\%

or mechanism-critical shifts.

Figure 8: Final-layer effects of Image $\rightarrow$ Question attention knockout. Vision-dependent tasks ( ChooseRel , LogicalObj ) show the strongest PID shifts; high-redundancy tasks ( ChooseAttr , ChooseCat ) are largely robust.

Figure 9: Layer-wise knockout effects for ChooseRel . Solid: normal; dashed: knockout. Blocking Image $\rightarrow$ Question attention traps visual evidence (elevated $U_{V}$ ), delays $U_{L}$ growth, and forces late-layer compensation at higher total-information cost.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Description
$V$	Vision modality	Visual input stream (image tokens)
$L$	Language modality	Linguistic input stream (question tokens)
$Y$	Target variable	Prediction target (logit or probability of correct answer)
$\ell$	Layer index	Transformer layer, $\ell\in\{0,\dots,L\}$
$L$	Model depth	Total number of Transformer layers (32 in LLaVA)
$\mathbf{x}_{V}^{(\ell)}$	Vision representation	Aggregated visual representation at layer $\ell$

Appendix

Table 9: Symbol table for key notation.

Explore similar work

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Grounding Multimodal Large Language Models
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models
Mar 31, 2026Lixin Xiu, Xufang Luo, Hideki NakayamaLarge Vision-Language Models VLM Interpretability
The University of Tokyo · Microsoft Research
From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making
Sep 4, 2026Davide Testa, Hugh Mee Wong, Alessandro Lenci +2Vision-Language Models VLM Reasoning
Fondazione Bruno Kessler (FBK) - Trento, Italy · Università di Roma La Sapienza - Rome, Italy · Utrecht University - Utrecht, The Netherlands +1

May 31, 2026J/K move · Enter open · S save

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

Wanlong Fang, Tianle Zhang, Wen Tao, Alvin Chan

College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore

Multimodal Grounding Multimodal Large Language Models Multimodal Reasoning Multimodal Foundation Models Multimodal Model Interpretability Multimodal Fusion+1

Open paper AskPDF arXiv

Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment. We introduce Partial Information Decomposition (PID) as a decision-level framework that separates unique, redundant, and synergistic contributions of sensory and linguistic inputs, beyond representation alignment and outcome-based evaluation. Across vision--language benchmarks, PID reveals recurring modality-use profiles: reasoning and grounding-oriented tasks tend to exhibit high synergy, whereas expert and knowledge-oriented tasks show stronger language-unique reliance. These profiles generalize across model families and predict sensitivity to modality-level interventions. We further extend PID to tri-modal systems with Sensory PID, treating language as a control variable to decompose video--audio information gain. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck dominated by visual information even on audio--visual fusion tasks. Finally, PID-guided reweighting provides initial evidence for improving multimodal reasoning and grounding performance.

Abstract

Explore similar work

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making