Efficient VLM Inference

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

38 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 322

All topics

VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning
Mar 26, 2026Zhe Gao, Shiyu Shen, Taifeng Chai +7Tool Use in VLMs Efficient VLM Inference
Nanjing University, Nanjing, China · Nankai University, Tianjin, China · East China Normal University, Shanghai, China +3
Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding
Mar 19, 2026Yikai Zheng, Xin Ding, Yifan Yang +6Streaming Video Understanding Efficient VLM Inference
Institute for AI Industry Research (AIR), Tsinghua University · University of Science and Technology of China · 4Microsoft Research Asia +1
Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs
Mar 14, 2026Nimrod Shabtay, Moshe Kimhi, Artem Spector +5Tool Use in VLMs Efficient VLM Inference
IBM Research · Tel-Aviv University · Technion +1
MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models
Mar 12, 2026Shengyuan Liu, Zanting Ye, Yunrui Lin +6Efficient VLM Inference 3D Medical Imaging
Chinese University of Hong Kong · Westlake University · Southern Medical University +4
History-Conditioned Spatio-Temporal Visual Token Pruning for Efficient Vision-Language Navigation
Mar 6, 2026Qitong Wang, Yijun Liang, Ming Li +2Efficient VLM Inference Token Pruning
Department of Computer and Information Sciences at the University of Delaware, Newark, DE 19711, USA · University of Maryland’s Department of Computer Science, College Park, MD 20742, USA · Mohamed bin Zayed University of Artificial Intelligence, Masdar City, Abu Dhabi 7909, UAE
FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering
Mar 4, 2026Tatiana Zemskova, Solomon Andryushenko, Ilya Obrubov +4Efficient VLM Inference Egocentric Video QA
AXXX · MIRAI · Yandex +1
Stateful Token Reduction for Long-Video Hybrid VLMs
Feb 27, 2026Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko +7Efficient VLM Inference Video-Language Models
NVIDIA
HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding
Feb 13, 2026Wenhui Liao, Hongliang Li, Pengyu Xie +15Efficient VLM Inference Speculative Decoding
South China University of Technology, Guangzhou, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Nanjing University, Nanjing, China +1
Think Proprioceptively: State-Grounded Visual Token Selection for VLA Policies
Feb 6, 2026Fangyuan Wang, Peng Zhou, Jiaming Qi +4Efficient VLM Inference Visuomotor Policy Learning
The Hong Kong Polytechnic University · Great Bay University · Northeast Forestry University +2
SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs
Feb 6, 2026Niccolo Avogaro, Nayanika Debnath, Li Mi +6Visual Question Answering Efficient VLM Inference
ETH Zürich · IBM Research · MIT-IBM Watson AI Lab.
Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
Jan 30, 2026Yanlong Chen, Amirhossein Habibian, Luca Benini +1Cross-Modal Knowledge Distillation Vision-Language Models
Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Qualcomm AI Research, Amsterdam, the Netherlands +1
DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation
Jan 29, 2026Haozhe Xie, Beichen Wen, Jiarui Zheng +4Inference-Time Optimization Efficient VLM Inference
Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding
Jan 28, 2026Haoyu Cao, Kun Yin, Yunfei Wu +16Efficient VLM Inference Document Parsing
Youtu-Parsing Team*
Efficient-VLN: A Simple yet Strong Baseline for Efficient Vision-Language Navigation
Dec 11, 2025Duo Zheng, Shijia Huang, Yanyang Li +1KV Caching Efficient VLM Inference
The Chinese University of Hong Kong · Weitu AI
AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference via Dynamical Text Guidance
Aug 8, 2025Weichen Zhang, Zhui Zhu, Ningbo Li +5Efficient VLM Inference Visual Token Pruning
Global Innovation Exchange, Tsinghua University, Beijing, China · Department of Automation, Tsinghua University, Beijing, China · School of Computer Science, Peking University, Beijing, China
SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
Aug 5, 2025Yufei Xue, Yushi Huang, Lunjie Zhu +2Vision-Language Models VLM Quantization
Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology
StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
Jul 7, 2025Meng Wei, Chenyang Wan, Xiqian Yu +9Efficient VLM Inference Efficient VLA Models
1Shanghai AI Lab · 2The University of Hong Kong · 3Zhejiang University +1
ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
May 24, 2025Duo Li, Zuhao Yang, Xiaoqin Zhang +2Efficient VLM Inference Large Vision-Language Models
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Vision-Language Models Efficient VLM Inference
1UCLA · 2Panasonic AI Research · 4Salesforce Research +1
AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning
Date pendingKyuan Oh, Bumsoo KimEfficient VLM Inference Large Vision-Language Models
Chung-Ang University, Seoul, Korea
Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs
Date pendingGuiqiu Liao, Matjaz Jogan, Daniel A. HashimotoImage Tokenization Healthcare
GRASP Laboratory, University of Pennsylvania · PCASO Laboratory, Department of Surgery, University of Pennsylvania · Department of Computer and Information Science, University of Pennsylvania
Ambient @ EgoLongQA 2026: Distilling Long-Video perception into a Sub-2B Model
Date pendingLogesh Kumar UmapathiEfficient VLM Inference Egocentric Video QA
Team Ambient

Mar 26, 2026, cs.CVJ/K move · Enter open · S save

VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning

Zhe Gao, Shiyu Shen, Taifeng Chai, Weinong Wang+6

Nanjing University, Nanjing, China · Nankai University, Tianjin, China · East China Normal University, Shanghai, China+3

Tool Use in VLMs Efficient VLM Inference Long-Video Understanding Long-Video QA RL for VLMs

Open paper AskPDF arXiv

Existing Multimodal Large Language Models (MLLMs) often suffer from hallucinations in long video understanding (LVU), primarily due to the imbalance between textual and visual tokens. Observing that MLLMs handle short visual inputs well, recent LVU works alleviate hallucinations by automatically parsing the vast visual data into manageable segments that can be effectively processed by MLLMs. SFT-based tool-calling methods can serve this purpose, but they typically require vast amounts of fine-grained, high-quality data and suffer from constrained tool-calling trajectories. We propose a novel VideoTIR that leverages Reinforcement Learning (RL) to encourage proper usage of comprehensive multi-level toolkits for efficient long video understanding. VideoTIR explores both Zero-RL and SFT cold-starting to enable MLLMs to retrieve and focus on meaningful video segments/images/regions, enhancing long video understanding both accurately and efficiently. To reduce redundant tool-calling, we propose Toolkit Action Grouped Policy Optimization (TAGPO), which enhances the efficiency of the calling process through stepwise reward assignment and reuse of failed rollouts. Additionally, we develop a sandbox-based trajectory synthesis framework to generate high-quality trajectories data. Extensive experiments on three long-video QA benchmarks demonstrate the effectiveness and efficiency of our method.

Momentum

Related topics

Latest papers 322