VLM Inference

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model inference is the process of using trained models to interpret images or video and generate answers, descriptions, or actions. Research improves reliability with training-free decoding and attention interventions that strengthen visual evidence, and improves perception through query-guided region selection and temporal frame retrieval. To reduce cost and delay, methods prune visual tokens, reuse caches, route queries across models, or coordinate inference across edge devices and cloud services. The central tension is preserving grounding and answer quality while reducing computation and latency, especially for long videos, high-resolution images, and real-time use.

Latest papers

CardsList

82 papers13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

  1. Shared Context, Separate Engines: Stateful Streaming Inference for Multimodal LLMs on Edge SoCs

    Oct 26, 2026Yilong Li, Chengpo Yan, Yijing Zeng +3Efficient VLM InferenceVLM Inference

    University of Wisconsin–Madison · Meta · Carnegie Mellon University

  2. Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval

    Oct 24, 2026Khang Nhat Hoang Vo, Belal Shoer, Abhirama Subramanyam PenamakuriVision-Language RetrievalCross-Modal Retrieval

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

  3. MJev-Spatial: System One Spatial Intelligence via Native Attention Matching

    Oct 15, 2026Luozheng Qin, Yuhan Zhu, Changhai Zhou +10Spatial Reasoning in Language ModelsVisual Spatial Reasoning

  4. Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

    Oct 15, 2026Emanuel Ben Baruch, Sivan Biham, Alon Shoshan +2VLM InferenceVLM Adaptation

    Amazon

  5. Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms

    Oct 14, 2026Junlin Liao, Harshit Kondle, Di Liu +1Efficient VLM InferenceVLM Inference

    Hofstra University, Hempstead, NY, USA · Bridgewater-Raritan High School, Bridgewater, NJ, USA · Norwegian University of Science and Technology, Trondheim, Norway

  6. MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search

    Oct 13, 2026Minghua Chen, Xingming Liao, Weiqiang Zhong +1GUI GroundingVision-Language Grounding

    School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510090, China

  7. Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering

    Oct 12, 2026Haonan Zhang, Ting Jin, Wenzhong Tang +1Visual Question AnsweringContrastive Decoding

    School of Computer Science and Engineering, Beihang University Beijing 100191, China

  8. MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference

    Oct 9, 2026Wenjuan Zhu, Zheming Yang, Yunqing Hu +1Multimodal Model RoutingVLM Inference

    Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  9. Video Codec-Guided Training-Free Token Compression for Video Large Language Models

    Oct 9, 2026Xihua Sheng, Ruijie Liu, Chang Wen ChenVideo Token CompressionVideo-Language Models

    Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

  10. SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

    Oct 8, 2026Zhimin Li, Pan Wang, Jingxian Chen +3Efficient VLM InferenceVLM Inference

    Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · Department of Computer Science, University of Central Florida

  11. Why VLMs Miss Small Objects, and When Zooming In Is Safe

    Oct 7, 2026Junzhe Shi, Yuan Gan, Shida JiangVision-Language ModelsVLM Inference

    Systems Engineering, University of California, Berkeley, Berkeley, CA, USA · Department of AI, Quotr AI, Emeryville, CA, USA

  12. CVIF: A Criticality-Driven Visual Intervention Framework for Geometric Diagram Understanding in MLLMs

    Oct 5, 2026Jiahui Kang, Bifan Wei, Lingling Zhang +4Test-Time Adaptation of VLMsVisual Reasoning

    School of Computer Science and Technology, Xi’an Jiaotong University · Tencent Hy AI Data

  13. When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs

    Sep 29, 2026Xiang Hu, Jiazuo Yu, Lu Zhang +2Streaming Video UnderstandingEvidence Selection

    IIAU Lab, Dalian University of Technology

  14. ReSight-SMC: Two-Stage Power Sampling via Island SMC with Visual Scouts

    Sep 28, 2026Yaowen Zhang, Xiangyu Qiu, Junyi Hu +5Markov Chain Monte CarloLarge Vision-Language Models

    Independent Researcher · University of Electronic Science and Technology of China · Tsinghua University

  15. The Right Information Extraction Pipeline Depends on the Document: Accuracy-Energy Trade-offs for Small, Local Models

    Sep 25, 2026Christoph Walser, Mauricio Fadel Argerich, Jonathan FürstDocument Information ExtractionEnergy-Efficient ML

    Zurich University of Applied Sciences, Switzerland · Universidad Politécnica de Madrid, Spain

  16. INTCORT: Training-Free Spatial Reasoning Enhancement for Vision-Language Models via Input Transformations and Confidence Routing

    Sep 21, 2026Haoran Sun, Jingqi Xu, Yanhui Li +3Visual Spatial ReasoningVLM Reasoning

    The University of Hong Kong · University of Southern California · China Telecom +3

  17. A Low-Latency Interactive System for Real-Time Video Understanding Based on VLMs

    Sep 12, 2026Punan Dai, Jun Xu, Bingcong Lu +4Streaming Video UnderstandingVideo-Language Models

    School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, China · Ant Group, Shanghai, China

  18. CONDUIT: A Unified Residual-Stream Restoration Framework for KV Cache Reuse in Vision-Language Models

    Sep 5, 2026Pengan Chen, Kaisheng Zheng, Liang Hong +10Efficient VLM InferenceKV-Cache Reuse

    The Chinese University of Hong Kong · Fudan University · LIGHTSPEED +1

  19. Compressing AI Traffic: Standardized Neural Network Coding of Visual-Token Representations in Split Vision-Language Inference

    Sep 1, 2026Reza Heidari, Hamed R. Tavakoli, Juho KannalaVisual Token CompressionVLM Inference

    Aalto University, Nokia Technologies · Nokia Technologies · Espoo, Finland +1

  20. LOCI: A Locator-Critic with Refinement Loop

    Aug 31, 2026Walid Bousselham, Mathilde Caron, Arsha Nagrani +1Visual Question AnsweringVision-Language Grounding

    Google DeepMind

  21. G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification

    Aug 27, 2026Zehua Hao, Fang Liu, Qinliang Wang +3Zero-Shot Image ClassificationVLM Inference

    Xidian University Xi’an, Shaanxi, China

  22. FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

    Aug 24, 2026Hengjie Zhu, Dayan Wu, Zihao Zhang +6Efficient Multimodal InferenceVLM Inference

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · School of Cyber Science and Engineering, Southeast University

  23. A VLM Answer Is Not an Anomaly Score: Rank Compression Across Image and Video Anomaly Detection

    Aug 21, 2026Inpyo Song, Jangwon LeeVideo Anomaly DetectionZero-Shot Anomaly Detection

    SungKyunKwan University South Korea

  24. Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs

    Aug 18, 2026Yifan Lu, Adinath Dukre, Abhijit Das +4VLM Hallucination MitigationMedical VLMs

    Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

  25. MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

    Aug 18, 2026Bonan Zhang, Shiyu Dong, Quan Hung Tran +9Efficient ViTsVLM Inference

    Meta · Work done while at Meta

  26. Your VLM Already Knows When: Training-Free Temporal Grounding by Asking Yes or No

    Aug 8, 2026Ji Huang, Barry Devereux, Hui WangTemporal Video GroundingTemporal Reasoning in Language Models

    Queen’s University Belfast

  27. How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systems

    Aug 8, 2026Henri Vanhuynegem, Weitao Xu, Yiran Shen +1VLM EvaluationVisual Question Answering

    Department of Software Technology, Delft University of Technology, Delft, The Netherlands · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Software, Shandong University, Jinan, China +1

  28. SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation

    Aug 6, 2026Mohamad Zamini, Diksha ShuklaMemory-Augmented VLMsOpen-Vocabulary Semantic Segmentation

    University of Wyoming Laramie, WY

  29. Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

    Jul 25, 2026Sultan Alshehri, Zhantao Yang, Han Zhang +1Vision-Language ModelsVLM Reasoning

    Carnegie Mellon University, Pittsburgh, PA, USA.

  30. A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi

    Jul 16, 2026A. C. Opus, J. Q. LuEfficient Transformer InferenceGPU Kernel Optimization

    Department of Physics, University of Puerto Rico, Mayaguez, PR 00680, USA