Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs

Jayadev Billa

cs.CLFeb 26, 2026

Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs

Authors: Jayadev Billa

Abstract

Numerous studies have shown that multimodal LLMs process speech and images well but fail in non-intuitive ways rendering trivial tasks such as object counting unreliable. We investigate this behavior from an information-theoretic perspective by framing multimodal LLM inference as a mismatched decoder problem: a decoder trained primarily on text can only extract information along text-aligned directions (removing up to 98% of the variation in modality-specific (non-text) directions improves decoder loss) and the amount of accessible information is bounded by the Generalized Mutual Information (GMI). We show that information loss is bounded as the distributional mismatch between the source data and the text data increases, and as the sensitivity of the decoder increases. This bound is a function of the model's scoring rule not its architecture. We validate the predictions across five models spanning speech and vision. A controlled study (two Prismatic VLMs differing only in encoder text-alignment) shows that the bottleneck lies in the scoring rule of the decoder rather than the text-alignment of the encoder or the learned projection. A LoRA intervention demonstrates that simply training with an emotion-related objective improves emotion detection from 17.3% to 61.8% task accuracy without affecting other attributes, confirming that the training objective determines what becomes accessible.

Figures & tables

Model	Encoder	Text-aligned?	LLM
Ultravox	Whisper	No	Llama-3.1-8B
Qwen2-Audio	Whisper	No	Qwen2-7B
LLaVA	CLIP-ViT	Yes	Vicuna-7B
Prismatic-D	DINOv2	No	Vicuna-7B
Prismatic-S	SigLIP	Yes	Vicuna-7B

Table 1: Models studied. Prismatic-D and Prismatic-S share the same architecture, adapter, training recipe, and LLM backbone; only the vision encoder differs.

Model	Condition	Modes	Var. %	$\Delta$ Loss (%)
Non-aligned encoders
Ultravox	MS (all)	11	63.6	$-$ 1.4
	TA-matched	11	29.4	$-$ 0.3
	Random	11	2.8	$+$ 0.3
Qwen2-Audio	MS (all)	5	97.6	$-$ 0.1
	TA-matched	5	1.8	$+$ 1.0

Table 2: Causal ablation at the adapter output. Eigenmodes are classified as modality-specific (MS;

\rho_{k}<0.5

) or text-aligned (TA), and matched subsets are projected out. Random controls ablate the same number of modes chosen uniformly at random (seed-averaged). For non-aligned encoders, removing MS modes improves loss while removing equal-sized TA subsets has negligible effect; the contrast is

160\times

for Prismatic-D (

t=-25.2

vs.

t=-0.4

). Random ablation gives intermediate effects proportional to variance removed. For SigLIP, few modes are MS and all effects are

{<}1

Condition	Emotion (probe)	Speaker (probe)	Lexical (probe)	Task acc.
Base Ultravox	.557	.137	.999	17.3%
+ Emotion LoRA	.632	.135	.994	61.8%
$\Delta$	$+$ 7.5%	$-$ 0.2%	$-$ 0.5%	$+$ 44.5%

Table 3: LoRA intervention on Ultravox (CREMA-D, llm-final probe accuracy). The emotion LoRA selectively improves emotion accessibility without meaningfully affecting speaker or lexical probes. Task accuracy measures forced-choice emotion detection on 1,002 held-out samples. The encoder and adapter outputs are identical across conditions (LoRA modifies only the LLM).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Encoder	Adapter	LLM-16	LLM-final
Ultravox	audio_tower.layer_norm	multi_modal_projector.ln_post	language_model.model.layers.16	language_model.model.norm
Qwen2-Audio	audio_tower.layer_norm	multi_modal_projector	language_model.model.layers.16	language_model.model.norm
LLaVA	...vision_tower. ...post_layernorm	model.mm_projector	...language_model. layers.16	...language_model. norm

Appendix

Table 4: Hook points for representation extraction.

Model	Info type (Dataset)	Encoder	Adapter	LLM-16	LLM-Final
Ultravox	Lexical (LS)	$.422\pm.014$	$.279\pm.018$	$.570\pm.023$	$.535\pm.027$
	Speaker (LS)	$.721\pm.029$	$.605\pm.031$	$.628\pm.019$	$.556\pm.019$
	Emotion (CD)	$.679\pm.012$	$.621\pm.007$	$.603\pm.011$	$.557\pm.006$
	Speaker (CD)	$.508\pm.005$	$.196\pm.010$	$.204\pm.011$	$.137\pm.008$
	Acoustic (E50)	$.760\pm.033$	$.647\pm.041$	$.645\pm.035$	$.581\pm.032$
Q2-Audio	Lexical (LS)	$.270\pm.006$	$.242\pm.015$	$.506\pm.018$	$.471\pm.021$

Appendix

Table 5: Full probe accuracy (mean

\pm

std over 5 seeds) for all models, datasets, hook points, and information types.

Model	Encoder	Lexical	Best non-text	Worst non-text
Encoder not text-aligned
Ultravox	Whisper	192%	92% (speaker, LS)	70% (speaker, CD)
Qwen2-Audio	Whisper	195%	102% (acoustic)	61% (speaker, LS)
Prismatic-D	DINOv2	104%	101% (super cat.)	100% (object cat.)
Encoder text-aligned
LLaVA	CLIP	108%	106% (super cat.)	105% (object cat.)

Appendix

Table 6: Information retention (adapter

\to

LLM-final, %). Values

{>}100

% indicate the LLM amplifies information;

{<}100

% indicates degradation. Text-aligned encoders retain all types; non-aligned encoders amplify lexical content while speaker identity collapses by up to 39%. LS = LibriSpeech, CD = CREMA-D, CO = COCO.

Model	Info type (Dataset)	Adapter	LLM-Final	Retention (%)
Encoder not text-aligned
Ultravox	Lexical (LS)	.279	.535	192
	Speaker (LS)	.605	.556	92
	Emotion (CD)	.621	.557	90
	Speaker (CD)	.196	.137	70
	Acoustic (E50)	.647	.581	90

Appendix

Table 7: Information retention from adapter to LLM-final layer (probe accuracy ratio

\times

100). Retention

{>}

100% means the LLM amplifies the signal;

{<}

100% means degradation. Non-aligned encoders show selective amplification (lexical recovers, speaker degrades); text-aligned encoders retain or slightly improve all types. LS = LibriSpeech, CD = CREMA-D, E50 = ESC-50, CO = COCO.

Speech
Model	Dataset	$L_{\log}$	$L_{\log}$ (p95)	$D$	$W_{\!1}$	$L_{\log}\cdot W_{\!1}$
Ultravox	LibriSpeech	9.08	14.58	10.3	17.8	162
	CREMA-D	5.82	9.70	17.0	–	–
	ESC-50	3.66	5.92	20.7	–	–
Q2-Audio	LibriSpeech	0.49	0.67	122.2	38.9	19.1
	CREMA-D	0.59	0.75	229.9	–	–

Appendix

Table 8: Lipschitz constants (

L_{\log}

, mean and p95 of gradient norm distribution), representation diameter (

D

), Wasserstein-1 distance (

W_{\!1}

at LLM layer 16), and their product.

L_{\log}

is content-dependent: it decreases as audio moves away from text (LibriSpeech

>

CREMA-D

>

ESC-50).

L_{\log}\cdot W_{\!1}

is largest for Ultravox (162, strongest degradation) and smallest for LLaVA (13.4, no degradation). Prismatic-S has

2.8\times

higher

L_{\log}

than Prismatic-D, consistent with the LLM being more responsive to text-aligned representations.

LLM hidden layer 16
Dataset	Label	All-token	Audio-only	$\Delta$
LibriSpeech	Speaker	.637 $\pm$ .026	.639 $\pm$ .023	$+$ .002
LibriSpeech	Lexical	.591 $\pm$ .009	.581 $\pm$ .019	$-$ .011
CREMA-D	Speaker	.210 $\pm$ .006	.213 $\pm$ .009	$+$ .003
CREMA-D	Lexical	.999 $\pm$ .001	.999 $\pm$ .001	$+$ .000
CREMA-D	Emotion	.593 $\pm$ .004	.595 $\pm$ .006	$+$ .002

Appendix

Table 9: Probe accuracy with all-token vs. audio-only pooling at LLM layers (Ultravox). All deltas are within

\pm

2.1%, most under 1%. Audio-only pooling does not improve probe accuracy.

Figure 1: Linear probe accuracy at four extraction points (encoder, adapter, LLM layer 16, LLM final) for all five models. For speech models (rows 1–2), lexical accuracy increases through the LLM while speaker identity degrades, illustrating the information accessibility gap. For vision models (row 3), LLaVA and Prismatic-S (text-aligned encoders) show improvement across all information types, while Prismatic-D (non-aligned) shows flat or declining non-textual accuracy. Datasets: LibriSpeech, CREMA-D, ESC-50 (speech); COCO (vision).

Figure 2: Mode alignment profiles for all five models. Each panel shows the top modes (eigenvectors of the adapter-output covariance

\Sigma_{M}

, sorted by decreasing eigenvalue; 100 for speech models, 200 for vision). Blue (left axis): alignment score

\rho_{k}

, the ratio of text variance to modal variance along each mode (§ 5.3 ). Orange (right axis): eigenvalue

\lambda_{k}

(log scale), the amount of adapter variance carried by each mode. Annotations show the percentage of total variance in Mode 0 and the count of modality-specific modes (

\rho_{k}<0.5

). For non-aligned encoders, the highest-variance modes have

\rho_{k}\approx 0

: the adapter’s dominant variation is along directions the decoder has never seen. For Prismatic-S (SigLIP, text-aligned), even Mode 0 is text-aligned (

\rho_{0}=0.83

Text-describable attributes
		Probe accuracy
Encoder	Info type	Adapter	LLM-final	$\Delta_{\text{LLM}}$
DINOv2	Lexical	.637	.664	+4.2%
	Object cat.	.792	.792	$-$ 0.0%
	Super cat.	.832	.839	+0.8%
SigLIP	Lexical	.657	.663	+0.9%

Appendix

Table 10: Prismatic controlled comparison (full results).

\Delta_{\text{LLM}}

= change from adapter to LLM-final. Top: text-describable attributes (named in captions). Bottom: non-textual attributes (object count, average object size, spatial spread; derived from annotations, not captions). Text baseline accuracy is substantially lower (.415–.495) for non-textual attributes, confirming their visual nature. Same architecture, same LLM, only encoder differs.

Model	Mode 0 $\rho_{k}$	Mean $\rho_{k}$ (top 10)	MS modes	MS var.
Ultravox	0.001	1.76	11/100	63.6%
Qwen2-Audio	0.010	7.21	5/100	97.6%
LLaVA	0.013	3.94	9/100	71.9%
Prismatic-D	0.034	0.087	53/100	71.0%
Prismatic-S	0.827	1.95	14/100	24.6%

Appendix

Table 11: Mode alignment summary. MS modes = modes with

\rho_{k}<0.5

(modality-specific). MS var. = fraction of top-100 eigenmode variance in MS modes. Mode 0 is modality-specific (

\rho_{k}\ll 1

) for all non-text-aligned encoders; for SigLIP (text-aligned), even Mode 0 is text-aligned (

\rho_{k}=0.83

). Extending to all 4096 modes strengthens the pattern: tail modes are overwhelmingly text-aligned with negligible individual variance, so the MS/TA split is driven by the high-variance modes reported here.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Encoder-Free Multimodal LLMs Learn Implicit Encoders
Oct 16, 2026David Harwath, Desh Raj, Xavier Velez +5Multimodal Large Language Models Multimodal Model Interpretability
The University of Texas at Austin · NVIDIA · Johns Hopkins University +2
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Mar 10, 2026Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4Multimodal Large Language Models Modality Gap in VLMs
Johns Hopkins University · Amazon · New York University +1
Bottleneck Tokens for Unified Multimodal Retrieval
Apr 13, 2026Siyu Sun, Jing Ren, Zhaohe Liao +8Multimodal IR Multimodal Embedding
Shanghai Jiao Tong University · ByteDance

Oct 16, 2026J/K move · Enter open · S save

Encoder-Free Multimodal LLMs Learn Implicit Encoders

David Harwath, Desh Raj, Xavier Velez, Sandra Arcos-Holzinger+4

The University of Texas at Austin · NVIDIA · Johns Hopkins University+2

Multimodal Large Language Models Multimodal Model Interpretability

Open paper AskPDF arXiv

We empirically analyze two recently proposed encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small. Unlike previous multimodal LLMs, these models do not use modality-specific encoders to extract tokenized embeddings representing audio and image input; instead, they directly feed windows of samples, pixels, or spectral features to the Transformer language model. We analyze whether the early layers of these models learn to function similarly to dedicated audio and visual encoders. We find that these early layers do in fact act as ``implicit encoders,'' encoding audio and visual inputs in a way that is largely invariant to their cross-modal context. Furthermore, when performing multimodal reasoning tasks, we show that independently encoding the modalities throughout these early model layers achieves the same accuracy as jointly encoding them from the first layer onwards. Finally, we use layer-wise probing to analyze the salient information captured by the implicit encoders and find their behavior to be similar to several popular encoder models.

Abstract

Multimodal Large Language Models

Explore similar work

Encoder-Free Multimodal LLMs Learn Implicit Encoders

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

Bottleneck Tokens for Unified Multimodal Retrieval