Efficient VLM Inference

VLM: Vision-Language Model

The field studies how to make vision-language models answer faster and use less memory and energy, especially for high-resolution images, long videos, and edge devices. Common approaches prune or merge visual tokens and frames, adapt resolution or retrieve regions on demand, reuse or compress KV caches, and use quantization, efficient kernels, routing, or speculative decoding. Methods are evaluated on tasks such as VQAv2, TextVQA, OCRBench, VideoMME, and LongVideoBench, measuring accuracy alongside time-to-first-token, throughput, and memory use. The central tension is preserving fine-grained evidence and reliable grounding while reducing computation: fewer tokens do not always mean lower end-to-end latency, and generating long answers can dominate cost on some devices.

Latest papers

487 papers66 papers in the last four weeks, up 50% on the four weeks before. 0.4% of all new papers.

Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM
Apr 8, 2026Chengyue Wu, Shiyi Lan, Yonggan Fu +9Vision-Language Models Diffusion VLMs
The University of Hong Kong · NVIDIA · MBZUAI +1
From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration
Apr 8, 2026Jiaqi Shi, Yuechan Li, Xulong Zhang +2Efficient VLM Inference Large Vision-Language Models
University of Science and Technology of China · Wuhan University · Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China
CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
Apr 7, 2026Yulin Zou, Wenyan Chen, Yan Chen +6Efficient VLM Inference VLM Inference
NTU Singapore Singapore · Beihang University China · A*STAR IAIC Singapore +1
ID-Selection: Importance-Diversity Based Visual Token Selection for Efficient LVLM Inference
Apr 7, 2026Zhaohong Huang, Wenjing Liu, Yuxin Zhang +2Efficient VLM Inference Visual Token Pruning
Xiamen University, 361005, P.R. China.
Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
Apr 7, 2026Jun Zhang, Yicheng Ji, Feiyang Ren +7Efficient VLM Inference Large Vision-Language Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Integration of Object Detection and Small VLMs for Construction Safety Hazard Identification
Apr 6, 2026Muhammad Adil, Mehmood Ahmed, Muhammad Aqib +3Construction Safety Vision-Language Models
Infrastructure Human Tech (IHT) Lab, Department of Civil and Environmental Engineering, University of Alberta, Edmonton, Alberta, Canada
Rethinking Model Efficiency: Multi-Agent Inference with Large Models
Apr 6, 2026Sixun Dong, Juhua Hu, Steven Li +2Efficient VLM Inference LLM Inference Acceleration
Independent Researcher · University of Washington · Meta Reality Labs
Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation
Apr 6, 2026Quoc-Huy Trinh, Mustapha Abdullahi, Bo Zhao +1Efficient VLM Inference Vision-Language Grounding
Aalto University · University of South Dakota
RCP: Representation Consistency Pruner for Mitigating Distribution Shift in Large Vision-Language Models
Apr 4, 2026Jianwei Zhang, Chaoning Zhang, Sihan Cao +7Efficient VLM Inference Large Vision-Language Models
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Qingshuihe Campus, No. 2006 Xiyuan Avenue, High-Tech Zone (West), Chengdu 611731, China · Machine Learning Department, Mohamed bin Zayed University of Artificial Intelligence, Building 1B, Masdar City, Abu Dhabi, United Arab Emirates · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, University Town of Shenzhen, Nanshan District, Shenzhen 518055, China +1
KiToke: Distribution-Aware Token Compression for Video Large Language Models
Apr 3, 2026Haifeng Huang, Biyin Xu, Yang LiVideo Token Compression Visual Token Compression
Department of Computer Science, Iowa State University.
QVAD: A Question-Centric Agentic Framework for Efficient and Training-Free Video Anomaly Detection
Apr 3, 2026Lokman Bekit, Hamza Karim, Nghia T Nguyen +1Video Anomaly Detection Zero-Shot Anomaly Detection
Department of Electrical Engineering, University of South Florida, Tampa, Florida, USA.
Collaborative Multi-Mode Pruning for Vision-Language Models
Apr 3, 2026Zimeng Wu, Yunhong Wang, Donghao Wang +1LVLM Pruning Efficient VLM Inference
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China
Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMs
Apr 3, 2026Yuhui Lin, Siyue Yu, Yuxing Yang +2Efficient VLM Inference Visual Token Pruning
Xi’an Jiaotong-Liverpool University · University of Liverpool
WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models
Apr 2, 2026Haiyu Wang, Yutong Wang, Jack Jiang +1Efficient VLM Inference Low-Rank Compression
Tandon School of Engineering, New York University · Courant Institute of Mathematical Sciences, New York University
Efficient Reasoning via Thought Compression for Language Segmentation
Apr 2, 2026Qing Zhou, Shiyu Zhang, Yuyu Jia +4Reasoning Segmentation CoT Compression
Northwestern Polytechnical University, China · Northwest Institute of Nuclear Technology, China.
PixelPrune: Pixel-Level Adaptive Visual Token Reduction via Predictive Coding
Apr 1, 2026Nan Wang, Zhiwei Jin, Chen Chen +1Efficient VLM Inference Visual Token Pruning
OPPO AI Center
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
Apr 1, 2026Dong-Jae Lee, Sunghyun Baek, Junmo KimEfficient VLM Inference Large Vision-Language Models
KAIST, South Korea
A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection
Mar 30, 2026Kunal Runwal, Swaraj Gajare, Daniel Adejumo +3Anomaly Detection VLMs for Autonomous Driving
Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA (ASAS-Labs). · Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University, 6 MetroTech Center, Brooklyn, NY 11201, USA (ASAS-Labs). · The City College of New York, Grove School of Engineering, New York, NY 10031, USA.
Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model
Mar 30, 2026Athos GeorgiouVision-Language Models Visual Document Retrieval
Independent Researcher
Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
Mar 30, 2026Surendra Pathak, Bo HanEfficient VLM Inference Visual Token Compression
Computer Science Department, George Mason University, Fairfax, VA - 22030, USA
Rényi Entropy: A New Token Pruning Metric for Vision Transformers
Mar 29, 2026Wei-Yuan Su, Ruijie Zhang, Zheng ZhangToken Pruning Visual Token Pruning
University of California, Santa Barbara
V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models
Mar 29, 2026Xinying Lin, Xuyang Liu, Yiyu Wang +2Efficient VLM Inference Video Token Compression
Shenzhen Campus of Sun Yat-sen University · Shenzhen Loop Area Institute · Sichuan University +1
RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation
Mar 28, 2026Sen Zhang, Runmei Li, Shizhuang Deng +7Visual Question Answering VLM Interpretability
School of Automation and Intelligence, Beijing Jiaotong University, Beijing 100091, China · School of Software, Northwestern Polytechnical University, Xi’an 710072, China · School of Artificial Intelligence, Beihang University, Beijing 100191, China
Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation
Mar 27, 2026Daiqiang Li, Zihao Pan, Zeyu Zhang +8Visual Token Pruning Efficient VLM Inference
Shanghai Jiao Tong University · Sichuan University · The University of Hong Kong +5
VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning
Mar 26, 2026Zhe Gao, Shiyu Shen, Taifeng Chai +7Tool Use in VLMs Efficient VLM Inference
Nanjing University, Nanjing, China · Nankai University, Tianjin, China · East China Normal University, Shanghai, China +3
ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs
Mar 25, 2026An Yu, Ting Yu Tsai, Zhenfei Zhang +3Efficient VLM Inference Token Pruning
University at Albany, SUNY · Peking University
Boosting Document Parsing Efficiency and Performance with Coarse-to-Fine Visual Processing
Mar 25, 2026Cheng Cui, Ting Sun, Suyin Liang +15Document Parsing Document Image Understanding
PaddlePaddle Team, Baidu Inc. · Xi’an Jiaotong University
RVLM: Recursive Vision-Language Models with Adaptive Depth
Mar 25, 2026Nicanor Mayumu, Zeenath Khan, Melodena Stephens +2VLM Reasoning Medical VLMs
Department of Computer Science, University of Wollongong in Dubai, Dubai Knowledge Park, Dubai, UAE · Department of Academic Affairs, Mohammed Bin Rashid School of Government, City Walk, Building B02, Dubai, UAE
Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding
Mar 25, 2026Fatih Ilhan, Gaowen Liu, Ramana Rao Kompella +5Efficient VLM Inference Large Vision-Language Models
Georgia Institute of Technology · Cisco Research
VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions
Mar 24, 2026Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas +2Efficient VLM Inference Vision-Language Models
Samsung AI Cambridge · Technical University of Iasi · Queen Mary University of London

Apr 8, 2026J/K move · Enter open · S save

Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM

Chengyue Wu, Shiyi Lan, Yonggan Fu, Sensen Gao+8

The University of Hong Kong · NVIDIA · MBZUAI+1

Vision-Language Models Diffusion VLMs Efficient VLM Inference Diffusion Language Model Decoding Blockwise Diffusion Language Models Speculative Decoding for Diffusion Language Models

Open paper AskPDF arXiv

Vision-language models (VLMs) predominantly rely on autoregressive decoding, which generates tokens one at a time and fundamentally limits inference throughput. This limitation is especially acute in physical AI scenarios such as robotics and autonomous driving, where VLMs are deployed on edge devices at batch size one, making AR decoding memory-bandwidth-bound and leaving hardware parallelism underutilized. While block-wise discrete diffusion has shown promise for parallel text generation, extending it to VLMs remains challenging due to the need to jointly handle continuous visual representations and discrete text tokens while preserving pretrained multimodal capabilities. We present Fast-dVLM, a block-diffusion-based VLM that enables KV-cache-compatible parallel decoding and speculative block decoding for inference acceleration. We systematically compare two AR-to-diffusion conversion strategies: a two-stage approach that first adapts the LLM backbone with text-only diffusion fine-tuning before multimodal training, and a direct approach that converts the full AR VLM in one stage. Under comparable training budgets, direct conversion proves substantially more efficient by leveraging the already multimodally aligned VLM; we therefore adopt it as our recommended recipe. We introduce a suite of multimodal diffusion adaptations, block size annealing, causal context attention, auto-truncation masking, and vision efficient concatenation, that collectively enable effective block diffusion in the VLM setting. Extensive experiments across 11 multimodal benchmarks show Fast-dVLM matches its autoregressive counterpart in generation quality. With SGLang integration and FP8 quantization, Fast-dVLM achieves over 6x end-to-end inference speedup over the AR baseline.