Efficient VLM Inference

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

38 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 322

All topics

Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning
Aug 10, 2026Jiahao Shao, Yuanbo Yang, Yiyi Liao +3Tool Use in VLMs Efficient VLM Inference
Hong Kong University of Science and Technology · Zhejiang University · Ant Group +1
Not All Visual Tokens Are Equally Safe to Remove:Consequence-Sensitive Visual Token Compression
Aug 10, 2026Jingbo Wen, Liang He, Mingyu Cao +4Efficient VLM Inference Visual Token Compression
1The University of Sydney · 2Tongji University · University of Surrey +3
VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling
Aug 9, 2026Yuqi Zhang, Cheng Chen, Yuyu Guo +6Vision-Language Models Efficient VLM Inference
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University, China · Shanghai Innovation Institute, China · Ant Group, China
How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systems
Aug 8, 2026Henri Vanhuynegem, Weitao Xu, Yiran Shen +1VLM Evaluation Visual Question Answering
Department of Software Technology, Delft University of Technology, Delft, The Netherlands · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Software, Shandong University, Jinan, China +1
A Picture is Worth a Thousand Tokens: How Vision Language Models Cut AI Energy Costs While Improving Accuracy
Aug 7, 2026Bhavika Jalli, Nikhil Korati Prasanna, Jayanta ChoudhuryVision-Language Models Efficient VLM Inference
Ericsson
An AI4AI Framework for Visual Token Pruning
Aug 7, 2026Zhen Liu, Wenli Huang, Wei Song +3Automated Algorithm Discovery Efficient VLM Inference
Xi’an Jiaotong University · Ningbo University of Technology · 3North China University of Technology +3
RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs
Aug 7, 2026Qiyanhui Lu, Han Wu, Rongjian Xu +6Efficient VLM Inference Large Vision-Language Models
City University of Hong Kong · Peking University · Huawei Technologies +1
Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models
Aug 7, 2026Minseok Kang, Hyunwoo Kim, Chanyoung Kim +3Vision-Language Models Efficient VLM Inference
Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea · School of Computer Science and Engineering, Soongsil University, Seoul, Korea · Department of Computer Science, Kookmin University, Seoul, Korea
Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding
Aug 6, 2026Bo Zhang, Wenxin Wang, Feng Chen +4Efficient VLM Inference Long-Video Understanding
1Sichuan University · 2Adelaide University · 3Beijing Institute of Technology
DIVE: Dynamic Iterative Visual Evidence Construction for Efficient Vision-Language Models
Aug 5, 2026Chen Zhong, Xiao An, Zijie Wang +3Efficient VLM Inference Visual Token Pruning
State Key Laboratory of Information Engineering in Surveying Mapping and Remote Sensing, Wuhan University, China · Electronic Information School, Wuhan University, China
Not All Redundant Tokens Are Alike: Analyzing Visual Token Pruning through Token Roles
Aug 5, 2026Hyeonyu Kim, Sehwan Lim, Youngwon Choi +2Efficient VLM Inference VLM Interpretability
Maum AI Inc., Republic of Korea
Q-CueGraph: Query-Conditioned Visual Evidence Graphs for Multimodal Reasoning
Aug 5, 2026Pengcheng Pan, Xinfang ZhangVisual Question Answering Efficient VLM Inference
The University of Tokyo · Tohoku University
TriCLE: Tri-Modal Vision-Language Reasoning for Edge-Deployed Fine-Grained Clustering
Aug 4, 2026Kishor Datta Gupta, Md. Mahfuzur Rahman, Fahad Rahman +4Efficient VLM Inference Clustering
Department of Cyber Physical Systems Clark Atlanta University Atlanta, GA, USA · Department of Computer Science and Engineering United International University Dhaka, Bangladesh · Department of Electrical and Computer Engineering North South University Dhaka, Bangladesh
RUTA: Principled Visual Token Allocation via Rate-Utility Optimization
Aug 4, 2026Jian Zou, Xiaoyu Xu, Zhihua Wang +3Efficient VLM Inference Visual Token Pruning
City University of Hong Kong · Google Inc.
When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding
Aug 4, 2026Ke Li, Jiayu Chen, Maoliang Li +5Efficient VLM Inference Large Vision-Language Models
School of Electronics Engineering and Computer Science, Peking University · School of Computer Science, Peking University
When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware
Aug 4, 2026Hao Dou, Ruiwen TianEfficient VLM Inference Visual Token Pruning
Harbin Institute of Technology, Harbin, China
SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
Aug 4, 2026Yaozhi Wen, Jialong Guo, Zhenliang Ni +2Efficient VLM Inference Structured Pruning
Huawei Technologies
Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning
Aug 4, 2026Yuyao Sun, Tao Deng, Shuang Li +3Visual Attention Efficient VLM Inference
Beihang University · Shanghai EABOT Technology Co., Ltd.
Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models
Aug 4, 2026Paribesh Regmi, Qingshuang Chen, Chi Zhang +3Efficient VLM Inference Video-Language Models
Amazon.com Services LLC
GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models
Aug 4, 2026Mengjie Zhang, Qihui Zhu, Tao Zhang +10Efficient VLM Inference Video-Language Models
University of Science and Technology of China He Fei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center He Fei, China
MoRAL: Sensor-Grounded BEV Reasoning for Compact VLMs toward Edge-Oriented Autonomous Driving
Aug 3, 2026Ambarish Govindarajulu Kaliamurthi, Kaikai LiuVLMs for Autonomous Driving Efficient VLM Inference
San Jose State University · Computer Engineering Department · San José State University (SJSU) +1
EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass
Aug 3, 2026Haomin Peng, Yongkang Li, Zhaoxiang Liu +4Efficient VLM Inference Open-Vocabulary Semantic Segmentation
Huazhong University of Science and Technology · Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom +1
Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
Aug 3, 2026Long Qian, Jiaqi Wei, Bingke Zhu +2Efficient VLM Inference Coreset Selection
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Engineering, Cardiff University, Cardiff, United Kingdom · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China +1
DiffPrune: differentiable information throttling for token pruning in vision-language models
Aug 3, 2026Landi He, Mingde Yao, Shawn Young +1Vision-Language Models Efficient VLM Inference
Shenzhen University of Advanced Technology · CUHK MMLab, CPII under InnoHK
AdaThinkV: Adaptive Thinking for Token-Efficient Video Reasoning
Aug 3, 2026Jingqi Tian, Haoji Zhang, Lin Chen +7Efficient VLM Inference RL for Language Model Reasoning
Tsinghua Shenzhen International Graduate School, Tsinghua University · Alibaba Group · Peking University
DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models
Aug 3, 2026Yongkang Zhou, Xiang Xia, Cheng Yan +2Vision-Language Models Efficient VLM Inference
University of Science and Technology of China Hefei, China
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
Aug 3, 2026Yu Chen, Xiaohong Li, Xiaole Wang +3Efficient VLM Inference Large Vision-Language Models
1Evermind · 2Shanda Group · 3Peking University
Dynamic Resolution Routing for Efficient Egocentric Grounding
Aug 3, 2026Huixin Sun, Wangbo Zhao, Fanyue Wei +3Efficient VLM Inference Vision-Language Grounding
National University of Singapore · The Hong Kong University of Science and Technology · Nanyang Technological University
Linear Multi-Timescale Retention as a Memory-Efficient Vision-Language Bridge
Aug 3, 2026Ashfak Yeafi, Mehedi Hasan, Md Khairul IslamMemory-Augmented VLMs Vision-Language Models
Department of Electrical and Electronic Engineering Khulna University of Engineering & Technology Khulna-9203, Bangladesh · Department of Computer Science and Engineering Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science Hobart and William Smith Colleges Geneva, NY, USA
Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere
Aug 2, 2026Jiayang He, Tianling Xu, Diancheng Kang +4Efficient VLM Inference Video-Language Models
Southern University of Science and Technology · Imperial College London · Jilin University

Aug 10, 2026, cs.CVJ/K move · Enter open · S save

Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning

Jiahao Shao, Yuanbo Yang, Yiyi Liao, Yujun Shen+2

Hong Kong University of Science and Technology · Zhejiang University · Ant Group+1

Tool Use in VLMs Efficient VLM Inference VLM Reasoning

Open paper AskPDF arXiv

Tool-augmented vision-language models increasingly "think with images": they call crop, zoom, or code tools and reason over the returned pixels. However, recent work using blind tests, gain decompositions, and attention analyses has shown that returned images contribute little, raising the question: if pixels do not carry the gain, what does? We hypothesize that the load-bearing signal is the structured text emitted before any returned pixel arrives: tool name, coordinates, target description, and intent. This textual scaffold encodes where to look and what to find. We introduce TextCall (call-but-no-return) to test this: it keeps the scaffold but replaces returned images with the text placeholder [Image output skipped]. Three studies support the hypothesis. (i) Non-necessity of returned pixels: across LoRA, full fine-tuning, and RL, TextCall matches or exceeds full thinking-with-images; under RL it preserves tool use at the reported checkpoint, avoiding the failure mode where, under matched settings, seeing the returned image causes the model to stop calling tools and answer directly. (ii) Sufficiency of the scaffold: on matched training queries, scaffold-only input yields equivalent accuracy to returned-image input. (iii) Component specificity: decomposing the scaffold into reasoning text and spatial code shows both components contribute, with the dominant one varying by task. Together these results support the Tool-Call Scaffold Hypothesis: in current thinking-with-images distributions, the active signal is the structured text emitted at tool-call time; the returned image is a redundant carrier. TextCall preserves accuracy while reducing latency by 29-46% and eliminating tool-execution API calls. Our claims hold for current thinking-with-images benchmarks; constructing tasks where pixels are genuinely load-bearing remains an open direction.

Momentum

Related topics

Latest papers 322