Efficient VLM Inference

VLM: Vision-Language Model

The field studies how to make vision-language models answer faster and use less memory and energy, especially for high-resolution images, long videos, and edge devices. Common approaches prune or merge visual tokens and frames, adapt resolution or retrieve regions on demand, reuse or compress KV caches, and use quantization, efficient kernels, routing, or speculative decoding. Methods are evaluated on tasks such as VQAv2, TextVQA, OCRBench, VideoMME, and LongVideoBench, measuring accuracy alongside time-to-first-token, throughput, and memory use. The central tension is preserving fine-grained evidence and reliable grounding while reducing computation: fewer tokens do not always mean lower end-to-end latency, and generating long answers can dominate cost on some devices.

Latest papers

487 papers66 papers in the last four weeks, up 50% on the four weeks before. 0.4% of all new papers.

Efficient-VLN: A Simple yet Strong Baseline for Efficient Vision-Language Navigation
Dec 11, 2025Duo Zheng, Shijia Huang, Yanyang Li +1KV Caching Efficient VLM Inference
The Chinese University of Hong Kong · Weitu AI
MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
Aug 28, 2025Junpeng Ma, Qizhe Zhang, Ming Lu +4Efficient VLM Inference Video Token Compression
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Fudan University · Taobao & Tmall Group of Alibaba
AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference via Dynamical Text Guidance
Aug 8, 2025Weichen Zhang, Zhui Zhu, Ningbo Li +5Efficient VLM Inference Visual Token Pruning
Global Innovation Exchange, Tsinghua University, Beijing, China · Department of Automation, Tsinghua University, Beijing, China · School of Computer Science, Peking University, Beijing, China
SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
Aug 5, 2025Yufei Xue, Yushi Huang, Lunjie Zhu +2Vision-Language Models VLM Quantization
Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology
StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
Jul 7, 2025Meng Wei, Chenyang Wan, Xiqian Yu +9Efficient VLM Inference Efficient VLA Models
1Shanghai AI Lab · 2The University of Hong Kong · 3Zhejiang University +1
ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
May 24, 2025Duo Li, Zuhao Yang, Xiaoqin Zhang +2Efficient VLM Inference Large Vision-Language Models
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Multimodal Understanding Vision-Language Models
1UCLA · 2Panasonic AI Research · 4Salesforce Research +1

Dec 11, 2025J/K move · Enter open · S save

Efficient-VLN: A Simple yet Strong Baseline for Efficient Vision-Language Navigation

Duo Zheng, Shijia Huang, Yanyang Li, Liwei Wang

The Chinese University of Hong Kong · Weitu AI

KV Caching Efficient VLM Inference Large Vision-Language Models Vision-Language Navigation Dataset Aggregation

Open paper AskPDF arXiv

While Multimodal Large Language Models (MLLMs) have demonstrated significant promise in Vision-Language Navigation (VLN), existing agents remain heavily constrained by systemic bottlenecks across inference, training, and data collection. Specifically, they suffer from prohibitive latency due to visual history reprocessing, action leakage during sequence-packed training, and suboptimal exploration in self-correction data collection. To overcome these intertwined challenges, we present Efficient-VLN, a highly efficient and robust baseline that systematically resolves these issues through three simple-yet-effective mechanisms. (1) Inference: We introduce KV-cache reuse with contiguous RoPE, enabling the model to process only the newly observed frame at each step for real-time inference. (2) Training: We propose packed training with an action-isolating mask to accelerate throughput while effectively bridging the training-inference gap by preventing action leakage. (3) Data Collection: We employ an Adaptive DAgger to dynamically balance autonomous exploration and oracle guidance, enhancing error-recovery capability without escalating computational costs. Extensive evaluations show that Efficient-VLN significantly advances the state-of-the-art across the R2R-CE (73.2% SR) and RxR-CE (75.6% SR) benchmarks. Meanwhile, it yields a 28% latency reduction compared to the previous state-of-the-art StreamVLN, establishing a new paradigm for streaming MLLM-based navigation.