Efficient VLM Inference

VLM: Vision-Language Model

The field studies how to make vision-language models answer faster and use less memory and energy, especially for high-resolution images, long videos, and edge devices. Common approaches prune or merge visual tokens and frames, adapt resolution or retrieve regions on demand, reuse or compress KV caches, and use quantization, efficient kernels, routing, or speculative decoding. Methods are evaluated on tasks such as VQAv2, TextVQA, OCRBench, VideoMME, and LongVideoBench, measuring accuracy alongside time-to-first-token, throughput, and memory use. The central tension is preserving fine-grained evidence and reliable grounding while reducing computation: fewer tokens do not always mean lower end-to-end latency, and generating long answers can dominate cost on some devices.

Latest papers

487 papers66 papers in the last four weeks, up 50% on the four weeks before. 0.4% of all new papers.

LiteVLA-H: Dual-Rate Vision-Language-Action Inference for Onboard Aerial Guidance and Semantic Perception
Apr 27, 2026Justin williams, Kishor Datta Gupta, Roy George +1Aerial Robotics Efficient VLM Inference
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vision-Language Models Efficient VLM Inference
Li Auto Inc.
LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
Apr 27, 2026Rinyoichi Takezoe, Yaqian Li, Zihao Bo +3Efficient VLM Inference Attention Sinks
Li Auto Inc.
GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
Apr 27, 2026Hongxin Li, Yuntao Chen, Zhaoxiang ZhangEfficient VLM Inference Training Data Curation
University of Chinese Academy of Sciences, Beijing, China. · New Laboratory of Pattern Recognition, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China.
Zoom In, Reason Out: Efficient Far-field Anomaly Detection in Expressway Surveillance Videos via Focused VLM Reasoning Guided by Bayesian Inference
Apr 26, 2026Xiaowei Mao, Bowen Sui, Weijie Zhang +7Efficient VLM Inference Anomaly Localization
School of Computer Science and Technology, Beijing Jiaotong University · 2Key Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education · 3Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruning
Apr 24, 2026Joonmyung Choi, Sanghyeok Lee, Jongha Kim +4Efficient VLM Inference Document QA
Korea University · KAIST · Adobe Research
Prototype-Based Test-Time Adaptation of Vision-Language Models
Apr 23, 2026Zhaohong Huang, Yuxin Zhang, Wenjing Liu +2Test-Time Adaptation VLM Adaptation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Sub-Token Routing for KV Cache Compression
Apr 23, 2026Wei Jiang, Wei WangLLM Inference Efficiency KV Caching
Futurewei Technologies San Jose, CA 95131, USA
Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models
Apr 22, 2026Juhong Min, Lazar Valkov, Vitali Petsiuk +2Visual Reasoning Visual Attention
AI Center – Mountain View, Samsung Electronics
Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
Apr 22, 2026Kibum Kim, Jiwan Kim, Kyle Min +4Efficient VLM Inference Attention Sinks
Korea Advanced Institute of Science and Technology (KAIST) · Oracle · University of California, San Diego +1
FOCAL: Filtered On-device Continuous Activity Logging for Efficient Personal Desktop Summarization
Apr 21, 2026Haoran Yin, Zhiyuan Wen, Jiannong Cao +2Efficient VLM Inference Text Summarization
The Hong Kong Polytechnic University Hong Kong, China · China Mobile Communications Company Limited Research Institute Beijing, China
ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving
Apr 21, 2026Lin Sha, Haiyun Guo, Tao Wang +4VLMs for Autonomous Driving Efficient VLM Inference
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Carizon, Beijing 100094, China
Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions
Apr 20, 2026Kecheng Zhang, Zongxin Yang, Mingfei Han +6Streaming Video Understanding Efficient VLM Inference
University of Science and Technology of China · Harvard University · Department of CV, MBZUAI +3
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Self-Attention Long-Context Language Model Inference
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University
Towards Joint Quantization and Token Pruning of Vision-Language Models
Apr 19, 2026Xinqing Li, Xin He, Xindong Zhang +3Vision-Language Models VLM Quantization
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute +3
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models
Apr 16, 2026Huawei Ji, Yuanhao Sun, Yuan Jin +4Vision-Language Models Efficient VLM Inference
Shanghai Jiao Tong University, Shanghai, China · University of Edinburgh, Edinburgh, UK
Learning Adaptive Reasoning Paths for Efficient Visual Reasoning
Apr 16, 2026Yixu Huang, Tinghui Zhu, Muhao ChenEfficient VLM Inference Visual Reasoning
♠Fudan University · ♣University of California, Davis
BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation
Apr 15, 2026Baoyou Chen, Hanchen Xia, Peng Tu +5Vision-Language Models Diffusion VLMs
Shanghai Academy of AI for Science · Fudan University · Shanghai Innovation Institute +1
Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs
Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM Evaluation Confidence Estimation for VLMs
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs
Apr 13, 2026Haicheng Wang, Yuan Liu, Yikun Liu +9Multimodal Large Language Models Long-Context VLMs
SAI, Shanghai Jiao Tong University, China · WeChat AI, Tencent, China
Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models
Apr 13, 2026Yvon Apedo, Martyna Poreba, Michal Szczepanski +1Vision-Language Models Efficient VLM Inference
Universit´e Paris-Saclay, Univ Evry, IBISC, 91020, Evry-Courcouronnes, France · Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France
Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models
Apr 13, 2026Kexin Ma, Jing Xiao, Chaofeng Chen +4Visual Token Pruning Efficient VLM Inference
School of Artificial Intelligence, Wuhan University, Wuhan, China · Department of Computer Science, University of Exeter, Exeter, United Kingdom · Chinese Academy of Sciences, Beijing, China; Wuhan AI Research, Wuhan, China +1
Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understanding
Apr 13, 2026Yueying Li, Fengxiang Wang, Yan Li +3Visual Token Compression Efficient VLM Inference
College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
UIPress: Bringing Optical Token Compression to UI-to-Code Generation
Apr 10, 2026Dasen Dai, Shuoqi Li, Ronghao Chen +3Token Compression Visual Token Compression
The Chinese University of Hong Kong China · Peking University China · University of Chinese Academy of Sciences China +2
How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms
Apr 10, 2026Shengji Jin, Yuanhao Zou, Victor Zhu +2Temporal Video Grounding Efficient VLM Inference
The University of Central Florida · Axon
SMART: When is it Actually Worth Expanding a Speculative Tree?
Apr 9, 2026Lifu Wang, Pan ZhouInference-Time Optimization GPU Acceleration
Singapore Management University, Singapore
AdaSpark: Adaptive Sparsity for Efficient Long-Video Understanding
Apr 9, 2026Handong Li, Zikang Liu, Longteng Guo +10Long-Video Understanding Efficient VLM Inference
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Alibaba Group Holding Limited +1
HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models
Apr 9, 2026Qihui Zhu, Tao Zhang, Yuchen Wang +9Visual Token Pruning Efficient VLM Inference
University of Science and Technology of China · ChangXin Memory Technologies, Inc · Huawei Noah’s Ark Lab
Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
Apr 8, 2026Yuheng Shi, Xiaohuan Pei, Linfeng Wen +2Efficient VLM Inference Large Vision-Language Models
University of Sydney, Australia · Sun Yat-sen University, China · City University of Hong Kong, China

Apr 27, 2026J/K move · Enter open · S save

LiteVLA-H: Dual-Rate Vision-Language-Action Inference for Onboard Aerial Guidance and Semantic Perception

Justin williams, Kishor Datta Gupta, Roy George, Mrinmoy Sarkar

Aerial Robotics Efficient VLM Inference Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models have shown strong semantic grounding and task generalization in manipulation, but aerial deployment remains difficult because drones require low-latency closed-loop guidance under strict onboard compute and communication constraints. We present LiteVLA-H, a compact 256M-parameter VLA system designed for dual-rate operation on an NVIDIA Jetson AGX Orin: a fast outer-loop guidance mode for short action-token outputs and a slower semantic mode for scene understanding, hazard description, and operator-facing narration. The central empirical observation is that, in this compact edge regime, end-to-end latency is dominated by multimodal pre-fill rather than by the marginal cost of decoding a few extra tokens. This motivates a scheduler that issues reactive action tokens at 50.65,ms (19.74,Hz) while still supporting sentence-level semantic outputs at 149.90--164.57\ms (6.08--6.67,Hz) on the same embedded platform. To specialize the model without collapsing its descriptive competence, we use a knowledge-preserving fine-tuning recipe that mixes reactive flight data, aerial semantic data, and generic caption/VQA supervision. Beyond reporting current latency measurements, we position the system against recent state-of-the-art architectures, including AnywhereVLA, FutureVLA, and ReMem-VLA, showing that the measured action branch reaches a higher edge inference rate under our deployment conditions while retaining periodic semantic awareness.