Efficient VLM Inference

VLM: Vision-Language Model

Overview

The field studies how to make vision-language models answer faster and use less memory and energy, especially for high-resolution images, long videos, and edge devices. Common approaches prune or merge visual tokens and frames, adapt resolution or retrieve regions on demand, reuse or compress KV caches, and use quantization, efficient kernels, routing, or speculative decoding. Methods are evaluated on tasks such as VQAv2, TextVQA, OCRBench, VideoMME, and LongVideoBench, measuring accuracy alongside time-to-first-token, throughput, and memory use. The central tension is preserving fine-grained evidence and reliable grounding while reducing computation: fewer tokens do not always mean lower end-to-end latency, and generating long answers can dominate cost on some devices.

Latest papers

CardsList

530 papers38 papers in the last four weeks, down 47% on the four weeks before. 0.3% of all new papers.

  1. Raptor: Rejection-aware Adaptive Tree based on Speculative Decoding

    Nov 19, 2026Qishu Hu, Haozhao Wang, Zicong Hong +2Efficient VLM InferenceSpeculative Decoding

  2. DRIFT: Detachable Reasoning for Runtime-Flexible and Efficient End-to-End Driving

    Nov 18, 2026Emad Aghajanzadeh, Bethany Anne Wong, Leonhard Kunze +8End-to-End Autonomous DrivingAutonomous Driving

  3. Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs

    Nov 18, 2026Xiaolin Lin, Kai-Chiang Wu, Bing Li +1Efficient VLM InferenceLong-Context VLMs

  4. CanopyKV: Remember beyond the Crop for Fast Active Visual Reasoning

    Nov 18, 2026Lei Yang, Dayan Wu, Yiliang Zhu +6KV-Cache CompressionEfficient VLM Inference

  5. Do Thinking VLMs Need to Keep Looking at Long Videos? Write It Down, Then Look Away

    Nov 18, 2026Pengyu Li, Zhitao Gao, Lingling Zhang +6Efficient VLM InferenceLong-Context VLMs

  6. AVTR: Adaptive Multimodal Reasoning with Visual Imagination

    Nov 18, 2026Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao +15Reasoning for VLMsVLM Reasoning

  7. Keep the Reasoning, Read the Image: On-Demand Visual KV for Multi-Turn Reasoning VLMs

    Nov 18, 2026Ziyi Wang, Xiyue Zhang, Bing Hu +3Efficient VLM InferenceKV-Cache Management

  8. MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models

    Nov 18, 2026Dongkeun Lee, Wonjun LeeEfficient VLM InferenceVLM Compression

  9. Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

    Nov 17, 2026Rui Ning, Om Chabra, Hari BalakrishnanEfficient VLM InferenceVLM Inference

  10. LoopZip: Visual Token Compression for Efficient Looped Vision-Language Models

    Nov 17, 2026Weizhong Huang, Zhicheng Liu, Yan Zhang +3Efficient VLM InferenceVisual Token Compression

  11. Know When to Zoom: How Small Is Too Small for Vision-Language Models?

    Nov 17, 2026Siddharth Vohra, Min XuVision-Language ModelsVisual Evidence Sensitivity in VLMs

  12. Semantic-Guided Geometric Decoding for Token-Efficient Visual Grounding

    Nov 17, 2026Zhaojie Chen, Junyan Yang, Le Yang +6Vision-Language GroundingVisual Grounding

  13. GridDriver: On-Board BEV-Based Language Planning and Reasoning

    Nov 16, 2026Katharina Winter, William Engel, Zain A. Alishammat +1VLMs for Autonomous DrivingAutonomous Driving Planning

  14. VisHead-Spec: Head-Routed Visual Attention for Multimodal Speculative Decoding

    Nov 16, 2026Xin Guo, Arushi Rai, Xiang Zhou +4Efficient VLM InferenceSpeculative Decoding

  15. MedJev: A System One Vision-Language Decision Model for Fast and Effective Medical Image Understanding

    Nov 16, 2026Fan Zhang, Hao Wu, Jinyang Wu +5Medical VLMsVLM Inference

  16. ReAccess: Look Once, Stay Focused with Efficient Test Time Memory

    Nov 16, 2026Tianhao Chen, Yilong Fan, Yuheng Wu +2Memory-Augmented VLMsEfficient VLM Inference

  17. From Cloud to Edge: A Fully Local Pipeline for Pothole Risk Assessment and Driver Warning Generation

    Nov 15, 2026Abdalmalek Aburaddaha, Samir A. RawashdehVLMs for Autonomous DrivingEfficient VLM Inference

  18. MMFlash: Faster Speculative Decoding for Vision-Language Models

    Nov 13, 2026Sicheng Feng, Zigeng Chen, Song Wang +2Efficient VLM InferenceSpeculative Decoding

  19. Report of the 8th LSVOS Challenge: Mobile Unified Multimodal Understanding

    Nov 12, 2026Chang Liu, Henghui Ding, Tong Yang +10Unified Multimodal ModelsMultimodal Model Evaluation

  20. TraceKV: Policy-Preserving Low-Rank KV Cache Compression for Reasoning Vision-Language Models

    Nov 11, 2026Ke Li, Zishan Shao, Yiran ChenKV-Cache CompressionEfficient VLM Inference

  21. Edge-Native Multimodal RAG: Hardware-Aware Quantization and Architectural Optimization for Consumer GPUs

    Nov 11, 2026Zahraa Selim, Menna Hamed, Wesam Ahmed +3Multimodal RAGRetrieval-Augmented Generation

  22. Learning When to Think: Dual-Reference Offline Optimization for Adaptive VLM Reasoning

    Nov 6, 2026Hongbin Lin, Sizhe Zou, Juangui Xu +6Reasoning for VLMsVLM Reasoning

  23. ProQuant: Progressive Quantization-aware Training for Edge MLLMs

    Nov 5, 2026Yufei Xue, Yushi Huang, Jiawei Shao +5LLM QuantizationQuantization-Aware Training