cs.CVApr 10, 2026

Do Vision Language Models Need to Process Image Tokens?

Authors: Sambit Ghosh, R. Venkatesh Babu, Chirag Agarwal

Organizations: IBM · Indian Institute of Science · University of Virginia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision Language Models (VLMs) have achieved remarkable success by integrating visual encoders with large language models (LLMs). While VLMs process dense image tokens across deep transformer stacks (incurring substantial computational overhead), it remains fundamentally unclear whether sustained image-token processing is necessary for their performance or visual representations meaningfully evolve from early to later layers. In this work, we systematically investigate the functional role of image tokens in VLMs and show that visual representations rapidly converge to a bounded-complexity regime, \ie their entropy stabilizes, intrinsic dimensionality compresses, and trajectory curvature approaches a near-constant profile. In contrast, textual representations continue to undergo substantial restructuring across depth. Once stabilized, visual representations become largely interchangeable between layers, indicating limited additional transformation in deeper stages. Further, depth-wise visual truncation reveals that the necessity of visual processing is task-dependent, where single-token predictions remain comparatively robust to truncated visual depth, but multi-token generation require sustained access to visual representations. Under deterministic decoding, reducing visual depth perturbs intermediate reasoning trajectories more strongly than final outputs, suggesting that image tokens influence the structure of reasoning more than the ultimate conclusions. Collectively, these findings question the assumption that deeper visual processing is uniformly essential in VLMs, challenging the current paradigm of multimodal LLM architectures.

Figures & tables

Figure 1: When is visual depth necessary in vision language models? We investigate a) stabilization of image tokens, b) layer-wise functional interchangeability, c) task-dependent utility, d) recoverability after truncation, e) reasoning-chain sensitivity. Our results reveal that image representations stabilize early, but their necessity depends on task and reasoning demands.
Figure 2: Layer-wise evolution of matrix entropy, intrinsic dimensionality, and trajectory curvature for image and textual tokens. Image tokens exhibit early stabilization across all three metrics, whereas textual tokens remain dynamically evolving across depth.
Figure 3: Layer-substitution analysis for captioning with Qwen2.5-VL-7B-Instruct. For each row, we show the image-token similarity heatmap (left), text-token similarity heatmap (center), and the semantic similarity as a function of the layer gap (right). Additional results are reported in Appendix A.2
Figure 4: Depth-dependent performance under image-token removal. We evaluate three output regimes on the same visual inputs: MCQ (bounded single-token generation), VQA (index+answer match, bounded multi-token generation), and caption generation.
Figure 5: Functional recovery on Flickr8K, ChartQA after distillation-based fine-tuning. Additional details of the result in Appendix- A.4
Figure 6: Performance across truncation depths under multiple decoding strategies for base and fine-tuned models. Fine-tuned models exhibit reduced dispersion across decoding configurations.
Figure 7: Coefficient of variation (CV) across decoding strategies. Fine-tuned models consistently show lower variability than truncated base models.
Figure 8: Compute–performance trade-off under visual-depth truncation. Blue: total GFLOPs per sample. Red: truncated base model performance. Green: fine-tuned model performance. Rightmost panel: prefill vs. decode compute decomposition. Fine-tuned models achieve high functional performance at substantially lower retained depth and compute budgets. Additional result on Appendix- A.4
Figure 9: Retention of reasoning-chain components under visual-token truncation. Semantic similarity is computed for summary, caption, and reasoning relative to the base model output (see Appendix- A.5 for additional results). Caption exhibits the strongest degradation, followed by reasoning, while summary remains comparatively stable.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Layer-wise evolution of representation metrics (section A.1 ), for the LLaVA family. Image tokens rapidly stabilize in early layers across all three metrics, while textual tokens continue to evolve across depth.
Figure 11: Representation metrics across layer-wise depth for Qwen2.5-VL models showing the same stabilization behaviour observed in Fig. 10 .
Figure 12: Layer-wise representation dynamics for the Qwen3-VL family with similar early stabilization of visual tokens and continued evolution of textual representations.
Figure 13: Representation metrics for InternVL models showing consistent visual-token stabilization across architectures.
Figure 14: Cross-layer substitution experiments for VQA-style tasks for LLaVA, Qwen2.5-VL and InternVL. Replacing image tokens with earlier-layer representations causes minimal degradation in exact match (Appendix A.6 ) accuracy, whereas substituting textual tokens results in larger performance drops.
Figure 15: Layer substitution results for caption generation in LLaVA and InternVL. Image-token substitutions preserve semantic similarity (Appendix A.6 ) across layers, while textual substitutions lead to progressively larger degradation as layer gaps increase.
Figure 16: Caption generation performance using BLUE and ROGUE scores on Flickr8K under visual processing depth truncation. Fine-tuned models significantly improve semantic similarity with the base model outputs compared to truncated models.
Figure 17: ChartQA VQA performance vs compute flops using exact match (Appendix A.6 ) under image-token truncation. Exact match accuracy improves after fine-tuning with fewer flops. Also, prefill phase benefits more compare to decode phase.
Figure 18: BLEU and ROGUE similarity of reasoning-chain components under visual-token truncation. The caption shows the strongest degradation, followed by reasoning, while the summary remains comparatively stable.
More in this topic

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Explore similar work

CardsList

  1. What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models

    Feb 28, 2026Yingqi Fan, Junlong Tong, Anhao Zhao +1Multimodal Large Language ModelsMultimodal Model Interpretability

    Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +1

  2. Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?

    May 20, 2026Tianyi Zhang, Mahtab Bigverdi, Ranjay KrishnaVLM EvaluationVLM Interpretability

    University of Washington

  3. Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

    Jun 8, 2026Siyuan Liu, Jinyang WuEfficient Multimodal InferenceMultimodal Large Language Models

    aSchool of Mechanics and Engineering Science, Peking University, China · bDepartment of Automation, Tsinghua University, China