Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

Zhaoyang Luo; Runmin Dong; Miao Yang; Fan Wei; Yushan Lai; Bin Luo; Haohuan Fu

cs.CVJun 30, 2026

Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

Authors: Zhaoyang Luo, Runmin Dong, Miao Yang, Fan Wei, Yushan Lai, Bin Luo, Haohuan Fu

Organizations: Tsinghua Shenzhen International Graduate School, Shenzhen, China · Sun Yat-sen University, Zhuhai, China · Tsinghua University, Beijing, China · National Supercomputing Center in Shenzhen, Shenzhen, China

Abstract

Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acceleration methods usually remove visual tokens or skip visual-token updates in entire layers, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. In this paper, we study visual-token computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations. Motivated by this answer-silent redundancy, we decompose each Transformer layer into attention and FFN operators and show that useful visual computation is often operator-dominant and layer-dependent. We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while selectively bypassing redundant attention, FFN, or both. Experiments across three MLLM architectures and 10 VQA benchmarks show that our method achieves strong efficiency-accuracy trade-offs, reducing \textbf{33.7%} TFLOPs on Qwen3-VL while retaining \textbf{99.5%} of the vanilla model performance.

Explore similar work

AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference
Sep 14, 2026Yuyao Sun, Tao Deng, Shuang Li +1Visual Tokens Multi-Modal Large Language Models
Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs
Jun 7, 2026Jie Ma, Zhike Qiu, Jiayi Ji +2Cross-Modal Attention Multi-Modal Large Language Models
Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning
Aug 4, 2026Yuyao Sun, Tao Deng, Shuang Li +3Visual Token Pruning Cross-Modal Attention

Sep 14, 2026 · cs.CVJ/K move · Enter open · S save

AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference

Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang

Beihang University

Multimodal large language models (MLLMs) require substantial computation to process numerous visual tokens across all transformer layers. Most methods for efficient MLLM inference exploit horizontal redundancy by compressing visual tokens. Beyond token reduction, recent studies exploit vertical redundancy through early exit or fixed-layer skipping. However, we find that the extent and distribution of this redundancy vary across inputs and differ between self-attention and MLP modules. Motivated by these observations, we propose AdaVSkip, which equips each layer with two lightweight routers that independently determine whether visual tokens pass through by or skip the self-attention and MLP modules. These decisions collectively define an input-specific visual-computation path, but their discrete and non-differentiable nature makes learning effective paths challenging. To address this challenge, we develop a progressive two-stage training framework that updates only the routers while keeping the backbone frozen. Stage I establishes an initial routing policy through supervised training with input-specific targets derived from module-wise necessity scores. To further align the routing policy with task performance, Stage II uses reinforcement learning to optimize routing decisions with direct feedback from generated answers. It combines an answer correctness reward with a skip-consistency reward that discourages excessive retention of visual-token computation. Across three MLLM backbones, AdaVSkip maintains strong task performance with substantially less computation. On LLaVA-NeXT-7B, AdaVSkip reduces FLOPs by 53.2% while preserving the original model's average performance. Combining it with visual token compression increases this reduction to 91.2%, while retaining 97.2% of the original performance on average.

Visual Tokens Multi-Modal Large Language Models Cross-Modal Attention LLM Inference Efficiency Ad

Open paper Ask about it PDF

Abstract

Explore similar work

AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference

Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning