What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models

Yingqi Fan; Junlong Tong; Anhao Zhao; Xiaoyu Shen

cs.CVFeb 28, 2026

What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models

Authors: Yingqi Fan, Junlong Tong, Anhao Zhao, Xiaoyu Shen

Organizations: Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University · The Hong Kong Polytechnic University

Abstract

Multimodal large language models (MLLMs) project visual tokens into the embedding space of language models, yet the internal structuring and processing of visual semantics remain poorly understood. In this work, we introduce a two-fold analytical framework featuring a novel probing tool, $\textbf{EmbedLens}$ , to conduct a fine-grained analysis. We uncover a pronounced semantic sparsity at the input level: visual tokens consistently partition into sink, dead, and alive categories. Remarkably, only the alive tokens, comprising $\approx60\%$ of the total input, carry image-specific meaning. Furthermore, using a targeted patch-compression benchmark, we demonstrate that these alive tokens already encode rich, fine-grained cues (e.g., objects, colors, and OCR) prior to entering the LLM. Internal visual computations (such as visual attention and feed-forward networks) are redundant for most standard tasks. For the small subset of highly vision-centric tasks that actually benefit from internal processing, we reveal that alive tokens naturally align with intermediate LLM layers rather than the initial embedding space, indicating that shallow-layer processing is unnecessary and that direct mid-layer injection is both sufficient. Ultimately, our findings provide a unified mechanistic view of visual token processing, paving the way for more efficient and interpretable MLLM architectures through selective token pruning, minimized visual computation, and mid-layer injection. The code is released at: https://github.com/EIT-NLP/EmbedLens.

Figures & tables

Figure 1: Cluster dominance and Cross-image stability in post-projection visual embeddings . (Left) Cluster similarity cross images. (Right) Number of tokens in the top-5 clusters. The cluster index is ranked by the number of tokens.

Figure 2: ( Left ) The t-SNE visualization of visual and textual clusters in word embedding space. ( Right ) An example illustrating the multi-semantic properties of a token within the green-marked text-proximity cluster shown in the left figure.

Figure 3: Comparison between EmbedLens and LogitLens in label–token matching accuracy. EmbedLens achieves higher semantic recall at shallow and middle layers across multiple MLLMs, suggesting that visual embeddings already encode object-level semantics prior to LLM fusion .

Figure 4: Formation (Main) and persistence (Insets) of LLM sink tokens. (Left) Cosine similarity between $\langle\mathrm{bos}\rangle$ , ViT sinks, and LLM sinks across sublayers. (Right) Layer-wise tracking of the top-5 most similar tokens to $\langle\mathrm{bos}\rangle$ .

Sinks Pruning
Method	General	OCR	CV Centric	Hallu.	Avg.
LLaVA-v1.5 7B	58.7	36.9	54.0	61.1	52.7
$-\mathcal{I}_{S_{\text{LLM}}}$	58.6	37.0	54.4	61.3	52.7
$-\mathcal{I}_{S_{\text{ViT}}}$	58.7	37.2	53.8	61.1	52.8
$-\mathcal{I}_{S}$	58.6	37.2	54.1	61.3	52.8
Sublayer Skipping

Table 1: Impact of sink pruning and sublayer skipping on downstream performance. 2 2 2 The General VQA score averages results from GQA [ 17 ] , MME [ 12 ] , MMBench

{}^{\text{en}}_{\text{dev}}

[ 32 ] , and MMStar [ 6 ] . OCR performance averages TextVQA

{}_{\text{val}}

[ 44 ] , OCRBench [ 33 ] , and DocVQA [ 36 ] . The CV-centric ability is measured by RefCOCO+ REC val, and RefCOCO REC testA/B [ 22 ] . Hallucination is evaluated using POPE [ 24 ] and HallusionBench [ 14 ] . See the appendix for detailed setups. “

\notin\mathcal{I}_{S_{\text{LLM}}}

–FF 2 ” denotes skipping MLP-2 for all non-sink visual tokens.

Method	General	OCR	CV centric	Hallu.	Avg.
LLaVA-v1.5 7B	58.7	36.9	54.0	61.1	52.7
-Dead	58.7	37.1	57.7	61.2	53.7
-Remaining	57.2	34.3	48.5	60.2	50.1

Table 2: Effect of pruning dead-token clusters. -Dead removes all dead tokens. -Remaining randomly removes the same number of dead tokens sampled from the remaining tokens.

Figure 5: Dead tokens exhibit minimal representation change and receive limited cross-modal attention. Left: Layer-wise cosine similarity within each cluster, showing that dead tokens remain highly self-consistent while other tokens evolve substantially. Right: Text-to-visual attention distribution across clusters. Inset: Token-averaged attention comparison.

Figure 6: Emergence and persistence of dead tokens. ( Left ) Average L1 norm across ViT layers. ( Right ) Layer-to-layer cosine similarity in CLIP. ( Bottom ) Average entropy of cluster semantics at the last layer of LLMs.

Figure 7: Semantic sparsity of visual tokens across layers.

Figure 8: Layer-wise alignment dynamics and effect of norm scaling.

Figure 9: Results on our multi-trajectory benchmark.

Layer Decoupling
Method	General	OCR	CV Centric	Hallu.	Avg.
LLaVA-v1.5 7B	58.7	36.9	54.0	61.1	52.7
$-\texttt{vMHA}^{\dagger}$	58.7	35.4	37.2	62.0	48.3
$-\texttt{vFFN}^{\dagger}$	57.8	35.4	50.6	61.7	51.4
Shallow Layers Skipping
$\texttt{L}_{\texttt{in}}\texttt{=4}^{\dagger}$	58.7	36.6	56.4	61.2	53.2

Table 3: Effect of layer decoupling and shallow-layer skipping on multimodal reasoning.

\dagger

denotes training-based method.

Method	General				OCR			CV Centric			Hallucination		Avg.
Method	GQA	MME P	MMStar	MMBench	VQA text	OCRbench	VQA Doc	RefCOCO+ val	RefCOCO testA	RefCOCO testB	POPE	Hallusion	Avg.
LLaVA-v1.5 7B	61.9	1507	33.5	64.1	58.2	31.2	21.5	50.0	64.5	47.4	85.9	36.2	52.7
Sinks Pruning
$-\texttt{ViT Sinks}$	61.9	1502	33.2	64.5	58.2	31.5	21.9	50.4	65.1	47.6	85.9	36.3	52.8
$-\texttt{LLM Sinks}$	62.0	1488	33.5	63.8	58.2	31.3	21.5	50.1	64.2	47.0	85.9	36.6	52.7
$-\texttt{All Sinks}$	62.0	1499	33.1	64.3	58.2	31.4	21.9	50.5	64.5	47.4	85.9	36.6	52.8

Table a : Effect of sink pruning and sublayer skipping. Performance remains stable under pruning or skipping operations, indicating weak reliance on visual sink tokens.

\notin\mathcal{I}_{S_{\text{LLM}}}

–MLP2 skips MLP-2 only for non-sink visual tokens.

Method	General				OCR			CV Centric			Hallucination		Avg.
Method	GQA	MME P	MMStar	MMBench	VQA text	OCRbench	VQA Doc	RefCOCO+ val	RefCOCO testA	RefCOCO testB	POPE	Hallusion	Avg.
LLaVA-v1.5 7B	61.9	1507	33.5	64.1	58.2	31.2	21.5	50.0	64.5	47.4	85.9	36.2	52.7
-Dead Tokens	61.8	1495	34	64.1	58.2	31.6	21.6	53.4	68.4	51.4	86	36.3	53.7
-Alive Tokens	60.5	1475	32.2	62.5	55.4	29	18.6	44.4	58.3	42.9	84.2	36.2	50.1

Table b : Effect of pruning dead-token clusters. Pruning dead-token clusters yields a small boost in accuracy, whereas removing the same number of remaining visual tokens causes substantial drops across all benchmarks. This validates that dead tokens are semantically void and redundant, while non-dead tokens carry meaningful information.

Method	General				OCR			CV Centric			Hallucination		Avg.
Method	GQA	MME P	MMStar	MMBench	VQA text	OCRbench	VQA Doc	RefCOCO+ val	RefCOCO testA	RefCOCO testB	POPE	Hallusion	Avg.
LLaVA-v1.5 7B	61.9	1507	33.5	64.1	58.2	31.2	21.5	50.0	64.5	47.4	85.9	36.2	52.7
Layer Decoupling
- vMHA †	61.9	1472	32.1	63.5	56.9	30.3	19	45.1	62.3	44.5	86.6	36.7	51.4
- vFFN †	62.2	1507	33.7	63.5	56.3	31.2	18.7	33.5	46	32.1	86.6	37.3	48.3
Shallow Layers Skipping

Table c : Effect of layer decoupling and shallow-layer skipping on multimodal reasoning. vMHA is crucial for spatially grounded reasoning in CV-centric tasks, while vFFN contributes to knowledge refinement. In contrast, shallow-layer skipping shows minimal degradation, confirming that early visual processing is largely redundant.

\dagger

denotes training-based method.

Figure a : Clusters Distribution when

\tau=0.95

. (Left) Number of tokens in the top-5 clusters. (Right) Cluster similarity cross images. The cluster index is ranked by the number of tokens.

Figure b : Clusters Distribution when

\tau=0.8

. (Left) Number of tokens in the top-5 clusters. (Right) Cluster similarity cross images. The cluster index is ranked by the number of tokens.

Figure c : Clusters Distribution of LLaVA-v1.5 13B . (Left) Number of tokens in the top-5 clusters. (Right) Cluster similarity cross images. The cluster index is ranked by the number of tokens.

Figure d : Effect of LLM sink manipulation. (Left) Attention redistribution after pruning visual sinks at the input: attention shifts from visual sinks to textual sinks in the system prompt. (Right) Skipping MLP-2 weakens alignment between visual sinks and BOS, confirming its role in sink formation, while overall attention patterns remain stable.

Figure e : Examples from the multi-trajectory benchmark. (Top) Object recognition: a bus image with associated object, color, and counting questions. (Middle) OCR: a single character on white padding for text–color grounding. (Bottom) OCR with background: same setup with a colored background, testing contextual bias in color reasoning.

Figure f : Bias Toward Surrounding Color Context

Figure g : Semantic grounding in vision late-entry models.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,069 papers

Representation Dynamics Reveal Semantic Saliency and Similarity for Visual Token Pruning in MLLMs
Sep 29, 2026Weixuan Li, Zikun Zhou, Xinyi Zhuang +4Efficient VLM Inference Visual Token Pruning
Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute
Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
Sep 28, 2026Jingdi lei, Junxian Li, Di Zhang +3Efficient Multimodal Inference Multimodal Large Language Models
Nanyang Technological University · Shanghai Jiao Tong University · Fudan University +2
TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
Jun 25, 2026Tinghao Wang, Yichen Guo, Rui Huang +9Efficient VLM Inference Multimodal Large Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · University of Electronic Science and Technology of China · Nanyang Technological University +1

Sep 29, 2026J/K move · Enter open · S save

Representation Dynamics Reveal Semantic Saliency and Similarity for Visual Token Pruning in MLLMs

Weixuan Li, Zikun Zhou, Xinyi Zhuang, Xinyan Guo+3

Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute

Efficient VLM Inference Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) incur high inference latency from long visual token sequences. Existing pruning methods commonly use attention maps or output features to estimate token importance or redundancy. Several recent approaches also exploit representation changes, but when and how these changes reflect foreground saliency and semantic consistency remain insufficiently understood. We analyze visual token representation dynamics across encoder depth and uncover two findings. First, the relationship between token update magnitudes and foreground saliency is layer-dependent: large token updates concentrate on foreground regions in two depth intervals, separated by several sink-dominated layers at intermediate depths. Second, similarities between token update directions better distinguish same-class from different-class tokens than those between encoder output features. Building on these findings, we propose MSDG-Prune, a training-free method that uses update magnitudes and directions to preserve salient and diverse visual information. Specifically, we group tokens by update-direction similarity and use query-weighted saliency derived from update magnitudes across a chosen depth window for group-wise token pruning. Extensive experiments across four MLLMs demonstrate the effectiveness and generalizability of MSDG-Prune. On LLaVA-NeXT, it retains 91.9% of uncompressed performance on average with only 5.6% of visual tokens, while achieving a 7.8x prefilling speedup. Code is available at https://github.com/liweixuan-hitsz/MSDG-Prune.

Abstract

Multimodal Large Language Models

Explore similar work

Representation Dynamics Reveal Semantic Saliency and Similarity for Visual Token Pruning in MLLMs

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference