Efficient VLM Inference

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

38 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 322

All topics

RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference
May 1, 2026Ben Wan, Yan Feng, Zihan Tang +4Efficient VLM Inference Optical Character Recognition
Shanghai Jiao Tong University · University of Science and Technology of China · Tsinghua University
EdgeFM: Efficient Edge Inference for Vision-Language Models
Apr 30, 2026Mengling Deng, Yuanpeng Chen, Sheng Yang +12Efficient VLM Inference Edge Inference
Go Further. AI · School of Data Science, Fudan University · RUYi Dynamics Co., Ltd +1
Progressive Semantic Communication for Efficient Edge-Cloud Vision-Language Models
Apr 29, 2026Cyril Shih-Huan Hsu, Wig Yuan-Cheng Cheng, Chrysa PapagianniEfficient VLM Inference Edge Computing
Informatics Institute, University of Amsterdam, The Netherlands. · Open-EP Community, Taoyuan City 330049, Taiwan (R.O.C.).
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU Acceleration Efficient VLM Inference
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India
FreqCache: Accelerating Embodied VLN Models with Adaptive Frequency-Guided Token Caching
Apr 27, 2026Zihao Zheng, Xingyue Zhou, Zhihao Mao +7Efficient VLM Inference Vision-Language Navigation
School of Computer Science, Peking University · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computer Science, China University of Geosciences (Wuhan) +3
Don't Pause! Every prediction matters in a streaming video
Apr 27, 2026Dibyadip Chatterjee, Zhanzhong Pang, Fadime Sener +2Streaming Video Understanding Efficient VLM Inference
National University of Singapore · Google Inc.
LiteVLA-H: Dual-Rate Vision-Language-Action Inference for Onboard Aerial Guidance and Semantic Perception
Apr 27, 2026Justin williams, Kishor Datta Gupta, Roy George +1Aerial Robotics Efficient VLM Inference
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vision-Language Models Efficient VLM Inference
Li Auto Inc.
LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
Apr 27, 2026Rinyoichi Takezoe, Yaqian Li, Zihao Bo +3Efficient VLM Inference Attention Sinks
Li Auto Inc.
GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
Apr 27, 2026Hongxin Li, Yuntao Chen, Zhaoxiang ZhangEfficient VLM Inference Training Data Curation
University of Chinese Academy of Sciences, Beijing, China. · New Laboratory of Pattern Recognition, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China.
Zoom In, Reason Out: Efficient Far-field Anomaly Detection in Expressway Surveillance Videos via Focused VLM Reasoning Guided by Bayesian Inference
Apr 26, 2026Xiaowei Mao, Bowen Sui, Weijie Zhang +7Efficient VLM Inference Anomaly Localization
School of Computer Science and Technology, Beijing Jiaotong University · 2Key Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education · 3Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruning
Apr 24, 2026Joonmyung Choi, Sanghyeok Lee, Jongha Kim +4Efficient VLM Inference Document QA
Korea University · KAIST · Adobe Research
Prototype-Based Test-Time Adaptation of Vision-Language Models
Apr 23, 2026Zhaohong Huang, Yuxin Zhang, Wenjing Liu +2Test-Time Adaptation VLM Adaptation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Sub-Token Routing for KV Cache Compression
Apr 23, 2026Wei Jiang, Wei WangLLM Inference Efficiency KV Caching
Futurewei Technologies San Jose, CA 95131, USA
Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models
Apr 22, 2026Juhong Min, Lazar Valkov, Vitali Petsiuk +2Visual Attention Efficient VLM Inference
AI Center – Mountain View, Samsung Electronics
Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
Apr 22, 2026Kibum Kim, Jiwan Kim, Kyle Min +4Efficient VLM Inference Attention Sinks
Korea Advanced Institute of Science and Technology (KAIST) · Oracle · University of California, San Diego +1
FOCAL: Filtered On-device Continuous Activity Logging for Efficient Personal Desktop Summarization
Apr 21, 2026Haoran Yin, Zhiyuan Wen, Jiannong Cao +2Efficient VLM Inference Text Summarization
The Hong Kong Polytechnic University Hong Kong, China · China Mobile Communications Company Limited Research Institute Beijing, China
ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving
Apr 21, 2026Lin Sha, Haiyun Guo, Tao Wang +4VLMs for Autonomous Driving Efficient VLM Inference
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Carizon, Beijing 100094, China
Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions
Apr 20, 2026Kecheng Zhang, Zongxin Yang, Mingfei Han +6Streaming Video Understanding Efficient VLM Inference
University of Science and Technology of China · Harvard University · Department of CV, MBZUAI +3
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Self-Attention Long-Context Language Model Inference
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University
Towards Joint Quantization and Token Pruning of Vision-Language Models
Apr 19, 2026Xinqing Li, Xin He, Xindong Zhang +3Vision-Language Models VLM Quantization
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute +3
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models
Apr 16, 2026Huawei Ji, Yuanhao Sun, Yuan Jin +4Vision-Language Models Efficient VLM Inference
Shanghai Jiao Tong University, Shanghai, China · University of Edinburgh, Edinburgh, UK
Learning Adaptive Reasoning Paths for Efficient Visual Reasoning
Apr 16, 2026Yixu Huang, Tinghui Zhu, Muhao ChenEfficient VLM Inference Visual Reasoning
♠Fudan University · ♣University of California, Davis
BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation
Apr 15, 2026Baoyou Chen, Hanchen Xia, Peng Tu +5Vision-Language Models VLM Adaptation
Shanghai Academy of AI for Science · Fudan University · Shanghai Innovation Institute +1
Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs
Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM Evaluation Confidence Estimation in Language Models
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models
Apr 13, 2026Yvon Apedo, Martyna Poreba, Michal Szczepanski +1Vision-Language Models Efficient VLM Inference
Universit´e Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France · Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France
SMART: When is it Actually Worth Expanding a Speculative Tree?
Apr 9, 2026Lifu Wang, Pan ZhouInference-Time Optimization GPU Acceleration
Singapore Management University, Singapore
CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
Apr 7, 2026Yulin Zou, Wenyan Chen, Yan Chen +6Efficient VLM Inference VLM Inference
NTU Singapore Singapore · Beihang University China · A*STAR IAIC Singapore +1
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
Apr 1, 2026Dong-Jae Lee, Sunghyun Baek, Junmo KimEfficient VLM Inference Large Vision-Language Models
KAIST, South Korea

May 1, 2026, cs.CVJ/K move · Enter open · S save

RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference

Ben Wan, Yan Feng, Zihan Tang, Weizhe Huang+3

Shanghai Jiao Tong University · University of Science and Technology of China · Tsinghua University

Efficient VLM Inference Optical Character Recognition Visual Token Pruning Document Image Understanding Visual Token Compression

Open paper AskPDF arXiv

DeepSeek-OCR leverages visual-text compression to reduce long-text processing costs and accelerate inference, yet visual tokens remain prone to redundant textual and structural information. Moreover, current token pruning methods for conventional vision-language models (VLMs) fail to preserve textual fidelity due to improper compression mechanisms. By analyzing the decoding process of DeepSeek-OCR, we find that a distinct two-stage reading trajectory: the model initially prioritizes the majority of high-norm tokens, then subsequently redistributes its attention to the remaining ones. Motivated by this insight, we propose RTPrune, a two-stage token pruning method tailored for DeepSeek-OCR. In the first stage, we prioritize high-norm visual tokens that capture salient textual and structural information. In the second stage, the remaining tokens are paired and merged based on optimal transport theory to achieve efficient feature aggregation. We further introduce a dynamic pruning ratio that adapts to token similarity and textual density for OCR tasks, enabling a better efficiency-accuracy trade-off. Extensive experiments demonstrate state-of-the-art performance, as evidenced by 99.47% accuracy and 1.23

\times

faster prefill on OmniDocBench, achieved with 84.25% token retention when applied to DeepSeek-OCR-Large.

Momentum

Related topics

Latest papers 322