Efficient VLM Inference

VLM: Vision-Language Model

The field studies how to make vision-language models answer faster and use less memory and energy, especially for high-resolution images, long videos, and edge devices. Common approaches prune or merge visual tokens and frames, adapt resolution or retrieve regions on demand, reuse or compress KV caches, and use quantization, efficient kernels, routing, or speculative decoding. Methods are evaluated on tasks such as VQAv2, TextVQA, OCRBench, VideoMME, and LongVideoBench, measuring accuracy alongside time-to-first-token, throughput, and memory use. The central tension is preserving fine-grained evidence and reliable grounding while reducing computation: fewer tokens do not always mean lower end-to-end latency, and generating long answers can dominate cost on some devices.

Latest papers

487 papers66 papers in the last four weeks, up 50% on the four weeks before. 0.4% of all new papers.

LoopVLA: Learning Sufficiency in Recurrent Refinement for Vision-Language-Action Models
May 11, 2026Boyang Shen, Kaixiang Yang, Hao Wang +4Efficient VLM Inference Efficient VLA Model Inference
1Huazhong University of Science and Technology · 2Wuhan United Imaging Surgical Co.,Ltd. (UIS)
Distilling 3D Spatial Reasoning into a Lightweight Vision-Language Model with CoT
May 10, 2026Alaa Asfour, Christopher Indris, Leihan Chen +2Cross-Modal Knowledge Distillation 3D Spatial Reasoning
Department of Computer Science, Toronto Metropolitan University, Toronto, ON, Canada
Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models
May 10, 2026Jie Ma, Yihang Liu, Zhike Qiu +2Efficient VLM Inference Vision-Language Grounding
Xiamen University
Fre-Res: Frequency-Residual Video Token Compression for Efficient Video MLLMs
May 10, 2026Yigui Feng, Qinglin Wang, Yang Liu +1Efficient VLM Inference Visual Tokenization
The College of Computer Science, National University of Defense Technology, Changsha, Hunan, China · College of Computer Science, National University of Defense Technology, Deya Road 109, Changsha, Hunan 410073, China · The Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou, Guangdong, China +1
LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?
May 9, 2026Kechen Fang, Yihua Qin, Chongyi Wang +3Efficient VLM Inference Multimodal Large Language Models
Tsinghua University · ModelBest
How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
May 9, 2026YiJie Huang, Yiqun Zhang, Zhuoyue Jia +7Efficient VLM Inference Large Vision-Language Models
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · Shanghai Artificial Intelligence Laboratory · School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China
DAPE: Dynamic Non-uniform Alignment and Progressive Detail Enhancement Techniques for Improving the Performance of Efficient Visual Language Models
May 9, 2026Mengyuan Tian, Qiyan Zhao, Yanan Wang +1VLM Adaptation Efficient VLM Inference
The Wang Yanan Institute for Studies in Economics, Xiamen University, Xiamen 361005, China · Fujian Key Laboratory of Pattern Recognition and Image Understanding, School of Computer and Information Engineering, Xiamen University of Technology, Xiamen 361024, China
Zero-Shot Chinese Character Recognition via Global-Local Dual-Branch Alignment and Hierarchical Inference
May 9, 2026Wei Cao, Hao Xu, Xiaolei DiaoZero-Shot Learning Efficient VLM Inference
1Jilin University, China · University College London, UK
Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding
May 8, 2026Hang Wu, Sherin Mary Mathews, Yujun Cai +2Streaming Video Understanding Efficient VLM Inference
University of California, Merced · US Bank · University of Queensland
TTF: Temporal Token Fusion for Efficient Video-Language Model
May 8, 2026Simin Huo, Ning LIToken Merging Efficient VLM Inference
Shanghai Jiao Tong University
GeoStack: A Framework for Quasi-Abelian Knowledge Composition in VLMs
May 7, 2026Pranav Mantini, Shishir K. ShahVision-Language Models VLM Adaptation
Department of Computer Science University of Houston · The University of Oklahoma School of Computer Science
VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
May 7, 2026Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1Expert Offloading Vision-Language Models
Shanghai Jiao Tong University Minhang Qu, Shanghai, China
VideoRouter: Query-Adaptive Dual Routing for Efficient Long-Video Understanding
May 7, 2026Kuanwei Lin, Wenhao Zhang, Ge LiEfficient VLM Inference Long-Video Understanding
School of Electronic and Computer Engineering, Peking University
Visual Text Compression as Measure Transport
May 6, 2026Lv Tang, Tianyi Zheng, Yang Liu +2Efficient VLM Inference Visual Tokenization
University of Alberta · vivo Mobile Communication Co., Ltd · Tsinghua University
ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation
May 6, 2026Wei Li, Jizhihui Liu, Li Yixing +3Multi-View Consistency Spatiotemporal Reasoning
Harbin Institute of Technology, Shenzhen · State Key Laboratory of Mobile Network and Mobile Multimedia Technology, ZTE, China · Shenzhen Loop Area Institute
VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Models
May 5, 2026JF Bastien, Sam D'AmicoEfficient VLM Inference Video-Language Models
Look, they said attention was all you needed, well we got your attention with this title.
Understanding Asynchronous Inference Methods for Vision-Language-Action Models
May 4, 2026Ayoub AgouzoulRobotic Control Efficient VLM Inference
École polytechnique · Systems Group, ETH Zürich
Latent Bridge: Feature Delta Prediction for Efficient Dual-System Vision-Language-Action Model Inference
May 4, 2026Yudong Liu, Yuan Li, Zijia Tang +12Efficient VLM Inference Efficient VLA Model Inference
1Duke University · 2Qualcomm AI Research · University of Florida
LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment
May 4, 2026Junyi Hu, Qiji Zhou, Lei Zhang +1VLM Robustness Efficient VLM Inference
Beijing Jiaotong University · Westlake University · Rochester Institute of Technology
WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimization
May 4, 2026Wei Tao, Xiaoyang Qu, Peiqiang Wang +4Mixed-Precision Quantization VLM Quantization
Huazhong University of Science and Technology, Wuhan, Hubei, China · Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, Guangdong, China · Tsinghua University, Shenzhen International Graduate School, Shenzhen, Guangdong, China
CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding
May 4, 2026Yuanyuan Jia, Shunpu Tang, Qianqian YangEfficient VLM Inference Edge Computing
College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China
Decouple and Cache: KV Cache Construction for Streaming Video Understanding
May 3, 2026Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener +1Streaming Video Understanding KV Caching
National University of Singapore.
Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models
May 3, 2026Martin Q. Ma, Willis Guo, Aditya Agrawal +4Active Perception Efficient VLM Inference
Carnegie Mellon University · MIT
Make Your LVLM KV Cache More Lightweight
May 1, 2026Xihao Chen, Yangyang Guo, Roger ZimmermannKV Caching Efficient VLM Inference
Integrative Sciences and Engineering Programme, National University of Singapore · School of Computing, National University of Singapore
RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference
May 1, 2026Ben Wan, Yan Feng, Zihan Tang +4Efficient VLM Inference Optical Character Recognition
Shanghai Jiao Tong University · University of Science and Technology of China · Tsinghua University
EdgeFM: Efficient Edge Inference for Vision-Language Models
Apr 30, 2026Mengling Deng, Yuanpeng Chen, Sheng Yang +12Efficient VLM Inference Edge Inference
Go Further. AI · School of Data Science, Fudan University · RUYi Dynamics Co., Ltd +1
Progressive Semantic Communication for Efficient Edge-Cloud Vision-Language Models
Apr 29, 2026Cyril Shih-Huan Hsu, Wig Yuan-Cheng Cheng, Chrysa PapagianniEdge-Cloud Collaborative Inference Efficient VLM Inference
Informatics Institute, University of Amsterdam, The Netherlands. · Open-EP Community, Taoyuan City 330049, Taiwan (R.O.C.).
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU Acceleration Efficient VLM Inference
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India
FreqCache: Accelerating Embodied VLN Models with Adaptive Frequency-Guided Token Caching
Apr 27, 2026Zihao Zheng, Xingyue Zhou, Zhihao Mao +7Efficient VLM Inference Vision-Language Navigation
School of Computer Science, Peking University · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computer Science, China University of Geosciences (Wuhan) +3
Don't Pause! Every prediction matters in a streaming video
Apr 27, 2026Dibyadip Chatterjee, Zhanzhong Pang, Fadime Sener +2Streaming Video Understanding Efficient VLM Inference
National University of Singapore · Google Inc.

May 11, 2026J/K move · Enter open · S save

LoopVLA: Learning Sufficiency in Recurrent Refinement for Vision-Language-Action Models

Boyang Shen, Kaixiang Yang, Hao Wang, Qiuyu Yu+3

1Huazhong University of Science and Technology · 2Wuhan United Imaging Surgical Co.,Ltd. (UIS)

Efficient VLM Inference Efficient VLA Model Inference Visuomotor Policy Learning Efficient VLA Models Vision-Language-Action Models Test-Time Adaptation of VLA Models+1

Open paper AskPDF arXiv

Current Vision-Language-Action (VLA) models typically treat the deepest representation of a vision-language backbone as universally optimal for action prediction. However, robotic manipulation is composed of many frequent closed-loop spatial adjustments, for which excessive abstraction may waste computation and weaken low-level geometric cues essential for precise control. Existing early-exit strategies attempt to reduce computation by stopping at predefined layers or applying heuristic rules such as action consistency, but they do not directly answer when a representation is actually sufficient for action. In this paper, we present LoopVLA, a recurrent VLA architecture that jointly learns representation refinement, action prediction, and sufficiency estimation. LoopVLA iteratively applies a shared Transformer block to refine multimodal tokens, and at each iteration produces both a candidate action and a sufficiency score that estimates whether further refinement is necessary. By sharing parameters across iterations, LoopVLA decouples refinement from absolute layer indices and grounds sufficiency estimation in the evolving representation itself. Since sufficiency has no direct supervision, we introduce a self-supervised distribution alignment objective, where intermediate confidence scores are trained to match the relative action quality across refinement steps, thereby linking sufficiency learning to policy optimization signals. Experiments on LIBERO, LIBERO-Plus, and VLA-Arena show that LoopVLA pushes the efficiency-performance frontier of VLA policies, reducing parameters by 45% and improving inference throughput by up to 1.7 times while matching or outperforming strong baselines in task success.