Efficient VLM Inference

VLM: Vision-Language Model

The field studies how to make vision-language models answer faster and use less memory and energy, especially for high-resolution images, long videos, and edge devices. Common approaches prune or merge visual tokens and frames, adapt resolution or retrieve regions on demand, reuse or compress KV caches, and use quantization, efficient kernels, routing, or speculative decoding. Methods are evaluated on tasks such as VQAv2, TextVQA, OCRBench, VideoMME, and LongVideoBench, measuring accuracy alongside time-to-first-token, throughput, and memory use. The central tension is preserving fine-grained evidence and reliable grounding while reducing computation: fewer tokens do not always mean lower end-to-end latency, and generating long answers can dominate cost on some devices.

Latest papers

487 papers66 papers in the last four weeks, up 50% on the four weeks before. 0.4% of all new papers.

Taking Shortcuts for Categorical VQA Using Super Neurons
Mar 11, 2026Pierre Musacchio, Jaeyi Jeong, Dahun Kim +1Efficient VLM Inference Visual Question Answering
Seoul National University · EPFL · Google Deepmind
Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models
Mar 11, 2026Sihan Cao, Jianwei Zhang, Pengcheng Zheng +7Efficient VLM Inference Visual Token Pruning
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China
Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity
Mar 10, 2026Zhengyao Fang, Pengyuan Lyu, Chengquan Zhang +3Visual Token Pruning Visual Token Compression
Harbin Institute of Technology, Shenzhen · Independent Researcher · Peng Cheng Laboratory
The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating
Mar 7, 2026Landi He, Xiaoyu Yang, Lijian XuEfficient VLM Inference Visual Token Pruning
Shenzhen University of Advanced Technology
VLN-Cache: Enabling Token Caching for VLN Models with Visual/Semantic Dynamics Awareness
Mar 7, 2026Zihao Zheng, Zhihao Mao, Xingyue Zhou +9Efficient VLM Inference Vision-Language Navigation
School of Computer Science, Peking University, Beijing, China. · School of Computer Science, China University of Geosciences (Wuhan), Wuhan, China. · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. +1
History-Conditioned Spatio-Temporal Visual Token Pruning for Efficient Vision-Language Navigation
Mar 6, 2026Qitong Wang, Yijun Liang, Ming Li +2Efficient VLM Inference Token Pruning
The authors are with the Department of Computer and Information Sciences at the University of Delaware, Newark, DE 19711, USA. · The authors are with the University of Maryland’s Department of Computer Science, College Park, MD 20742, USA. · The author is with the Mohamed bin Zayed University of Artificial Intelligence, Masdar City, Abu Dhabi 7909, UAE.
HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models
Mar 6, 2026Lincen Bai, Hedi Tabia, Raul Santos-RodriguezLVLM Pruning Neural Network Pruning
Université Paris-Saclay · University of Bristol
Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models
Mar 6, 2026Jialuo He, Huangxun ChenEfficient VLM Inference Visual Token Pruning
The Hong Kong University of Science and Technology (Guangzhou)
FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering
Mar 4, 2026Tatiana Zemskova, Solomon Andryushenko, Ilya Obrubov +4Efficient VLM Inference Egocentric Video QA
AXXX · MIRAI · Yandex +1
EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs
Mar 4, 2026Yuhao Chen, Bin Shan, Xin Ye +1Efficient VLM Inference Visual Token Pruning
ByteDance, Beijing, China
Benchmarking Compact VLMs for Clip-Level Surveillance Anomaly Detection Under Weak Supervision
Mar 3, 2026Kirill Borodin, Kirill Kondrashov, Nikita Vasiliev +4Vision-Language Models Weakly Supervised Video Anomaly Detection
Faculty of Information Technology, Moscow Technical University of Communication and Informatics, Moscow 111024, Russia
Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models
Mar 3, 2026Jialiang Zhang, Junlong Tong, Junyan Lin +4VLM Reasoning Streaming Inference
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ocean University of China · Shanghai Jiao Tong University +3
Stateful Token Reduction for Long-Video Hybrid VLMs
Feb 27, 2026Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko +7Hybrid Mamba-Transformer Architectures Efficient VLM Inference
NVIDIA
HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
Feb 27, 2026Hao Wu, Yingqi Fan, Jinyang Dai +3Efficient VLM Inference Large Vision-Language Models
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · University of Science and Technology of China +2
RT-NeuS: Towards Real-Time Neuro-Symbolic Video Understanding via Adaptive Temporal Verification
Feb 26, 2026Shawn Liang, Sahil Shah, Chengwei Zhou +5Long-Video QA Temporal Video Understanding
Case Western Reserve University · The University of Texas at Austin
Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents
Feb 26, 2026Zhou Xu, Bowen Zhou, Qi Wang +2GUI Agents Visual Token Pruning
Tsinghua University, Shenzhen, China · Xidian University, Xi’an, China · The Chinese University of Hong Kong, Hong Kong, China
LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
Feb 26, 2026Yuxiang Shen, Hailong Huang, Zhenkun Gao +6VLM Reasoning Large Vision-Language Models
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddings
Feb 24, 2026Bibin WilsonObject Detection Open-Vocabulary Object Detection
Independent Researcher
ApET: Approximation-Error Guided Token Compression for Efficient VLMs
Feb 23, 2026Qiankun Ma, Ziyao Zhang, Haofei Wang +3Visual Token Compression Efficient VLM Inference
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +2
QuickGrasp: Responsive Video-Language Querying Service via Accelerated Tokenization and Edge-Augmented Inference
Feb 23, 2026Miao Zhang, Ruixiao Zhang, Jianxin Shi +3Efficient VLM Inference VLM Inference
School of Computing Science, Simon Fraser University, Canada · Computer Science Department, University of Illinois Urbana-Champaign, United States · College of Cryptology and Cyber Science, Nankai University, China +1
Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimization
Feb 22, 2026Yanting Yang, Shenyuan Gao, Qingwen Bu +2VLMs for Robotics Large Language Model-Based Robot Planning
Rutgers University · The Hong Kong University of Science and Technology · The University of Hong Kong
OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport
Feb 22, 2026Xiwen Chen, Wenhui Zhu, Gen Li +9Visual Token Pruning Efficient VLM Inference
Morgan Stanley · Clemson University · Arizona State University +4
DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
Feb 21, 2026Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma +2Efficient VLM Inference Multimodal Token Compression
Advanced Micro Devices, Inc. (AMD)
Designing Production-Scale OCR for India: Multilingual and Domain-Specific Systems
Feb 18, 2026Ali Faraz, Raja Kolla, Ashish Kulkarni +1Multilingual OCR Optical Character Recognition
Krutrim AI, Bangalore, India
HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding
Feb 13, 2026Wenhui Liao, Hongliang Li, Pengyu Xie +15Efficient VLM Inference Speculative Decoding
South China University of Technology, Guangzhou, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Nanjing University, Nanjing, China +1
Think Proprioceptively: State-Grounded Visual Token Selection for VLA Policies
Feb 6, 2026Fangyuan Wang, Peng Zhou, Jiaming Qi +4Efficient VLM Inference Visuomotor Policy Learning
The Hong Kong Polytechnic University · Great Bay University · Northeast Forestry University +2
SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs
Feb 6, 2026Niccolo Avogaro, Nayanika Debnath, Li Mi +6Visual Question Answering Efficient VLM Inference
ETH Zürich · IBM Research · MIT-IBM Watson AI Lab.
Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
Jan 30, 2026Yanlong Chen, Amirhossein Habibian, Luca Benini +1Cross-Modal Knowledge Distillation Vision-Language Models
Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Qualcomm AI Research, Amsterdam, the Netherlands +1
DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation
Jan 29, 2026Haozhe Xie, Beichen Wen, Jiarui Zheng +4Inference-Time Optimization Efficient VLM Inference
Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding
Jan 28, 2026Haoyu Cao, Kun Yin, Yunfei Wu +16Efficient VLM Inference Document Parsing
Youtu-Parsing Team*

Mar 11, 2026J/K move · Enter open · S save

Taking Shortcuts for Categorical VQA Using Super Neurons

Pierre Musacchio, Jaeyi Jeong, Dahun Kim, Jaesik Park

Seoul National University · EPFL · Google Deepmind

Efficient VLM Inference Visual Question Answering Representation Probing

Open paper AskPDF arXiv

Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of Vision Language Models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model's prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10x.