Efficient VLM Inference

VLM: Vision-Language Model

The field studies how to make vision-language models answer faster and use less memory and energy, especially for high-resolution images, long videos, and edge devices. Common approaches prune or merge visual tokens and frames, adapt resolution or retrieve regions on demand, reuse or compress KV caches, and use quantization, efficient kernels, routing, or speculative decoding. Methods are evaluated on tasks such as VQAv2, TextVQA, OCRBench, VideoMME, and LongVideoBench, measuring accuracy alongside time-to-first-token, throughput, and memory use. The central tension is preserving fine-grained evidence and reliable grounding while reducing computation: fewer tokens do not always mean lower end-to-end latency, and generating long answers can dominate cost on some devices.

Latest papers

487 papers66 papers in the last four weeks, up 50% on the four weeks before. 0.4% of all new papers.

Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding
Mar 24, 2026Mincheol Kwon, Minseung Lee, Seonga Choi +7Efficient VLM Inference Large Vision-Language Models
Korea University · KT Corporation · Soongsil University +1
CATNAV: Cached Vision-Language Traversability for Efficient Zero-Shot Robot Navigation
Mar 24, 2026Aditya Potnis, Francisco Affonso, Shreya Gummadi +2Robot Traversability Estimation Robot Navigation
Field Robotics Engineering and Science Hub (FRESH), Illinois Autonomous Farm, University of Illinois at Urbana-Champaign (UIUC), IL
Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels
Mar 23, 2026Alexandra Zelenin, Alexandra ZhuravlyovaLow-Rank Adaptation Memory-Efficient Fine-Tuning
Rethinking Token Reduction for Large Vision-Language Models
Mar 23, 2026Yi Wang, Haofei Zhang, Qihan Huang +7Efficient VLM Inference Visual Token Compression
College of Computer Science and Technology, Zhejiang University · State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security +3
StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding
Mar 23, 2026Guowei Tang, Tianwen Qian, Huanran Zheng +2Streaming Video Understanding Video-Language Model Evaluation
East China Normal University, Shanghai, China
DyGeoVLN: Infusing Dynamic Geometry Foundation Model into Vision-Language Navigation
Mar 22, 2026Xiangchen Liu, Hanghan Zheng, Jeil Jeong +5Vision-Language Navigation Efficient VLM Inference
KAIST · HKUST(GZ) · JD Explore Academy
QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression
Mar 22, 2026Zhongyang Li, Yaqian Li, Faming Fang +6Multimodal Token Compression Visual Token Compression
East China Normal University · Li Auto Inc.
ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models
Mar 22, 2026Xu Li, Yi Zheng, Yuxuan Liang +5Visual Token Pruning Efficient VLM Inference
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China
ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding
Mar 20, 2026Quan Kong, Yuhao Shen, Yicheng Ji +2Efficient VLM Inference Video-Language Models
Zhejiang University, Hangzhou, China
Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding
Mar 19, 2026Yikai Zheng, Xin Ding, Yifan Yang +6Streaming Video Understanding Efficient VLM Inference
Institute for AI Industry Research (AIR), Tsinghua University · University of Science and Technology of China · 4Microsoft Research Asia +1
MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interaction
Mar 19, 2026Joerg Deigmoeller, Nakul Agarwal, Stephan Hasler +8Vision-Language Grounding Human-Robot Interaction
Honda Research Institute Europe, 63073 Offenbach, Germany · Honda Research Institute USA, San Jose, CA 95134, USA.
HORNet: Task-Guided Frame Selection for Video Question Answering with Vision-Language Models
Mar 19, 2026Xiangyu Bai, Bishoy Galoaa, Sarah OstadabbasVideo QA Query-Aware Video Frame Selection
Northeastern University, Boston, MA, USA
Unified Spatio-Temporal Token Scoring for Efficient Video VLMs
Mar 18, 2026Jianrui Zhang, Yue Yang, Rohun Tripathi +5Visual Token Pruning Efficient VLM Inference
University of Wisconsin-Madison · Allen Institute for AI
Empirical Recipes for Efficient and Compact Vision-Language Models
Mar 17, 2026Jiabo Huang, Zhizhong Li, Sina Sajadmanesh +2Vision-Language Models Efficient VLM Inference
Sony AI.
Parallel In-context Learning for Large Vision Language Models
Mar 17, 2026Shin'ya Yamaguchi, Daiki Chijiwa, Tamao Sakao +1Product of Experts In-Context Learning
NTT
Efficient Document Parsing via Parallel Token Prediction
Mar 16, 2026Lei Li, Ze Zhao, Meng Li +6Efficient VLM Inference Document Parsing
Platform and Content Group, Tencent · Renmin University of China
MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
Mar 16, 2026Hui Shen, Xin Wang, Ping Zhang +11Speculative Decoding Efficient VLM Inference
University of Michigan · The Ohio State University · Independent +5
Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs
Mar 16, 2026Jaehoon Lee, Mingi Jung, Soohyuk Jang +3Efficient VLM Inference Visual Token Pruning
Interdisciplinary Program in AI, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University · Department of Artificial Intelligence, Chung-Ang University
LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models
Mar 16, 2026Soumyaratna Debnath, Bui Duc Manh, Zinan Liu +1Foveated Vision Efficient VLM Inference
Nanyang Technological University, Singapore
ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference
Mar 15, 2026Surendra Pathak, Bo HanEfficient VLM Inference Visual Token Pruning
George Mason University, Fairfax, VA, USA
A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoning
Mar 14, 2026Yichang Xu, Gaowen Liu, Ramana Rao Kompella +6Video Reasoning Long-Video QA
Georgia Institute of Technology, Atlanta, GA · Cisco Systems, USA
Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs
Mar 14, 2026Nimrod Shabtay, Moshe Kimhi, Artem Spector +5Tool Use in VLMs Efficient VLM Inference
IBM Research · Tel-Aviv University · Technion +1
NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval
Mar 13, 2026Zhuchenyang Liu, Yao Zhang, Yu XiaoVisual Document Retrieval Vision-Language Retrieval
Aalto University, Espoo, Finland
Adaptive Vision-Language Model Routing for Computer Use Agents
Mar 13, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM Agent Routing Computer-Use Agents
vLLM Semantic Router Project · MBZUAI · McGill University +2
Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
Mar 12, 2026Baifeng Shi, Stephanie Fu, Long Lian +10Efficient VLM Inference Visual Token Pruning
UC Berkeley · NVIDIA · Clarifai +1
ForensicZip: More Tokens are Better but Not Necessary in Forensic Vision-Language Models
Mar 12, 2026Yingxin Lai, Zitong Yu, Jun Wang +3Visual Token Pruning Visual Token Compression
Great Bay University · Shenzhen University · Harbin Institute of Technology +1
MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models
Mar 12, 2026Shengyuan Liu, Zanting Ye, Yunrui Lin +6Efficient VLM Inference 3D Medical Imaging
Chinese University of Hong Kong · Westlake University · Southern Medical University +4
Detect Anything in Real Time: From Single-Prompt Segmentation to Multi-Class Detection
Mar 12, 2026Mehmet Kerem TurkcanObject Detection Efficient Object Detection
Columbia University
UniCompress: Token Compression for Unified Vision-Language Understanding and Generation
Mar 11, 2026Ziyao Wang, Chen Chen, Jingtao Li +4Multimodal Token Compression Visual Token Compression
Sony AI · University of Maryland, College Park
Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models
Mar 11, 2026Qingtao Pan, Zhihao Dou, Shuo LiEfficient VLM Inference Multimodal Large Language Models
Case Western Reserve University

Mar 24, 2026J/K move · Enter open · S save

Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding

Mincheol Kwon, Minseung Lee, Seonga Choi, Miso Choi+6

Korea University · KT Corporation · Soongsil University+1

Efficient VLM Inference Large Vision-Language Models Visual Question Answering Visual Grounding

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown strong performance across various multimodal tasks by leveraging the reasoning capabilities of Large Language Models (LLMs). However, processing visually complex and information-rich images, such as infographics or document layouts, requires these models to generate a large number of visual tokens, leading to significant computational overhead. To address this, we propose PinPoint, a novel two-stage framework that first identifies instruction-relevant image regions and then refines them to extract fine-grained visual features for improved reasoning and efficiency. Central to our approach is the Instruction-Region Alignment, which localizes relevant regions using both visual input and textual instructions. We further introduce new annotations that provide richer ground-truth supervision for instruction-relevant regions across challenging VQA benchmarks: InfographicVQA, MultiPageDocVQA, and SinglePageDocVQA. Experimental results show that PinPoint not only achieves superior accuracy compared to existing methods but also reduces computational overhead by minimizing irrelevant visual tokens.