Efficient VLM Inference

VLM: Vision-Language Model

Ask about this topic

Overview

The field studies how to make vision-language models answer faster and use less memory and energy, especially for high-resolution images, long videos, and edge devices. Common approaches prune or merge visual tokens and frames, adapt resolution or retrieve regions on demand, reuse or compress KV caches, and use quantization, efficient kernels, routing, or speculative decoding. Methods are evaluated on tasks such as VQAv2, TextVQA, OCRBench, VideoMME, and LongVideoBench, measuring accuracy alongside time-to-first-token, throughput, and memory use. The central tension is preserving fine-grained evidence and reliable grounding while reducing computation: fewer tokens do not always mean lower end-to-end latency, and generating long answers can dominate cost on some devices.

Latest papers

CardsList

487 papers66 papers in the last four weeks, up 50% on the four weeks before. 0.4% of all new papers.

  1. Don't Ask the LLM: Visual Token Pruning with Spectral Structure and Semantic Relevance

    Oct 28, 2026Intaek Shin, Geuntaek Lim, Sungjune Park +1Visual Token PruningEfficient VLM Inference

    Sejong University, South Korea

  2. REEL: Same Frames, Different Answers and Different Costs

    Oct 28, 2026Xin Liang, Jiangpeng HeVideo-Language Model EvaluationMemory-Augmented Video Understanding

    Indiana University Bloomington

  3. V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

    Oct 27, 2026Miso Choi, Daekeun Kim, Eunji Kim +1Memory-Augmented VLMsEfficient VLM Inference

    Dept. of Computer Science and Engineering, Korea University · Amazon

  4. Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

    Oct 27, 2026Donghyeon Cho, Joowon Kim, Seungho Shin +1Efficient VLM InferenceDiffusion VLMs

    KAIST · AITRICS

  5. Shared Context, Separate Engines: Stateful Streaming Inference for Multimodal LLMs on Edge SoCs

    Oct 26, 2026Yilong Li, Chengpo Yan, Yijing Zeng +3Efficient VLM InferenceVLM Inference

    University of Wisconsin–Madison · Meta · Carnegie Mellon University

  6. LOCUS: Locate Once, Check Until Settled. Draft-and-Verify Visual Grounding with a Box Head and Coordinate-Token Verification

    Oct 26, 2026Muhammad TalhaVision-Language GroundingVisual Grounding

    Independent Researcher

  7. GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

    Oct 26, 2026Ye Li, Haoran Ding, Huanan Liu +10Efficient InferenceGUI Agents

    Tsinghua University · The Chinese University of Hong Kong · Joy Future Academy, JD Group, Beijing, China

  8. Seeing the Options: Typed Decisions over Visual Candidates with Reusable Option Memory

    Oct 24, 2026Fengxin LiEfficient VLM InferenceMultimodal Model Evaluation

    Harbin Institute of Technology, Shenzhen

  9. Rescuing Your Pruning: Rethinking Visual Token Pruning and Post-Training Recovery in MLLMs

    Oct 24, 2026Peiran Yin, Hao Wu, Zihao Yang +3Visual Token PruningEfficient VLM Inference

    EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Uppsala University

  10. Vision Token Pruning via Query-Vision Interaction Decomposition

    Oct 23, 2026Harshithanjani Athi, Sravan Kumar Ankireddy, Jianzhong Charlie Zhang +1Efficient VLM InferenceVisual Token Pruning

    1The University of Texas at Austin · 2Samsung Research America

  11. The Answer Knows Where to Look: Zooming before MLLMs Answer under Visual-Token Budgets

    Oct 23, 2026Zhensheng Hu, Guanglong Xu, Guanqiang HuLarge Vision-Language ModelsVisual Evidence Sensitivity in VLMs

    Integrated Hospital of Traditional Chinese Medicine, Southern Medical University, Guangzhou, China · Institute of Medical Engineering Interdisciplinary Research, Xinjiang Medical University, Urumqi, China · School of Mathematics and Statistics, Zhaoqing University, Guangdong 526061, China +1

  12. MoG-VLM: Efficient Video Anomaly Detection through Motion-Guided Vision-Language Models

    Oct 22, 2026Khaled Waleed Dawoud, Muhammad Zaigham Zaheer, Khan Muhammad +1Video Anomaly DetectionEfficient VLM Inference

    Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · Sungkyunkwan University, Seoul 03063, South Korea

  13. DisasterRoute: Evidence-Guided Dual-System Routing for Visual Disaster Question Answering

    Oct 21, 2026Tan Zhang, Xing Deng, Lu Zhang +2Visual Question AnsweringSelective Prediction for VLMs

    UESTC · DLUT · Hainan University

  14. Resuming Vision-Language Inference across Onboard Interruptions

    Oct 21, 2026Xiaolong Huang, Yinyuan Zhang, Daliang Xu +6Efficient VLM Inference

    Peking University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China

  15. GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction

    Oct 20, 2026Trong-Thang Pham, Anh Nguyen, Hien Van Nguyen +1Egocentric Video QAEfficient VLM Inference

  16. ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

    Oct 18, 2026Guangzong Si, Dong Wang, Yifan Yu +3Efficient VLM InferenceVisual Token Compression

    University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Tsinghua University +2

  17. Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count

    Oct 17, 2026Geng Li, Guohao Chen, Shilin Shan +1Visual Token PruningVLM Evaluation

    Nanyang Technological University

  18. Hierarchical and Query-Aware Visual Token Pruning for High-Resolution LVLMs

    Oct 16, 2026Luther Wei, Ling Zou, Kang-Yang Huang +2Visual Token PruningEfficient VLM Inference

    Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan · MediaTek Inc., Hsinchu, Taiwan

  19. HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models

    Oct 16, 2026Oscar Kemme, Janek Haberer, Ali Hojjat +1Elastic Neural NetworksEfficient VLM Inference

    Hamburg University of Technology (TUHH), Germany · Kiel University, Germany · UNU-INWEH, Germany

  20. CatchUp: Depth-Aware Re-Admission of Visual Tokens for Efficient Large Vision-Language Models

    Oct 16, 2026Eunju Park, Uiwon HwangEfficient VLM InferenceLarge Vision-Language Models

    Division of Artificial Intelligence & Software, Ewha Womans University

  21. Query-adaptive Recurrent Tokenization

    Oct 16, 2026Bavesh Balaji, Pranav Gopinath, Pouya Sadeghi +1Visual TokenizationEfficient VLM Inference

    Critical ML Lab, University of Waterloo

  22. EgoP2^{2}C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video

    Oct 15, 2026Chaoyu Li, Arka Sadhu, Yuliang Li +3Visual Spatial ReasoningMultimodal Spatial Reasoning

    Arizona State University · Meta

  23. MJev-Spatial: System One Spatial Intelligence via Native Attention Matching

    Oct 15, 2026Luozheng Qin, Yuhan Zhu, Changhai Zhou +10Spatial Reasoning in Language ModelsVisual Spatial Reasoning

  24. GeoThinker-FF: Endogenous Predictive World Representations for Fast Spatial Intelligence

    Oct 14, 2026Haoyuan Li, Qihang Cao, Huanyuan Zhou +103D VLMs3D Spatial Reasoning

    Shenzhen campus of Sun Yat-sen University · Nanyang Technological University · XPeng Motors +1

  25. STP-GUI: Spatially-Aware Token Pruning for Efficient Web-GUI Agents

    Oct 14, 2026Shuo Wang, Sifan Zhou, Xiaobo Wang +6Visual Token PruningEfficient VLM Inference

    Mininglamp · Southeast University · Shenzhen University of Advanced Technology

  26. Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms

    Oct 14, 2026Junlin Liao, Harshit Kondle, Di Liu +1Efficient VLM InferenceVLM Inference

    Hofstra University, Hempstead, NY, USA · Bridgewater-Raritan High School, Bridgewater, NJ, USA · Norwegian University of Science and Technology, Trondheim, Norway

  27. SP2^2ec: Adaptive Self-Speculative Decoding for Vision-Language Models

    Oct 13, 2026Yuqi Huang, Xingyao Li, Yunlong Hou +3Speculative DecodingEfficient VLM Inference

    Department of Mathematics, National University of Singapore, Singapore · Yale University, Center for Algorithms, Data, and Market Design, USA · Department of Electrical and Computer Engineering, National University of Singapore, Singapore

  28. Just Respect the Semantics: Image-Token Compression via Pre-LLM Semantic Variable-Resolution Tokenization and In-LLM Relevance Pruning

    Oct 13, 2026Tianhao Fu, Austin Wang, Charles Chen +5Visual Token CompressionEfficient VLM Inference

    University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · McGill University, Montreal, QC, Canada +2

  29. VideoJev: System One Model for Video Understanding and Interaction

    Oct 13, 2026Yuhan Zhu, Xiangyu Zeng, Qingyi Si +15Video QATemporal Video Grounding

    NJU · UCAS · THU +4

  30. Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs

    Oct 13, 2026Peiran Yin, Hao Wu, Zihao Yang +6Visual Token PruningEfficient VLM Inference

    EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Uppsala University +1