Efficient Multimodal Inference

Ask about this topic

Overview

Efficient multimodal inference studies how to make models process mixed inputs with less computation, memory, latency, and energy. Common approaches prune or compress visual and audio tokens, select useful video frames or input modalities, compress KV caches, and adapt reasoning depth to query difficulty. Systems work also schedules inference across heterogeneous GPUs and edge devices, with evaluations spanning image and video question answering, long-context tasks, and real-time applications. Many methods preserve accuracy at substantial compression, but an open tension is whether they retain fine-grained evidence and grounding across tasks while improving end-to-end latency and cost.

Latest papers

CardsList

231 papers27 papers in the last four weeks, down 4% on the four weeks before. 0.2% of all new papers.

  1. Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference

    Oct 29, 2026Wenjie Huang, Jun Zhang, Hongyaoxing Gu +10Efficient Multimodal InferenceMultimodal Large Language Models

    Zhejiang University · Alibaba Group · University of the Chinese Academy of Sciences +4

  2. Calibrating Attention-Based Token Pruning for Efficient Multi-Image Synthesis in UMMs

    Oct 29, 2026Zimeng Wu, Qijian Tian, Penghui Du +3Efficient Multimodal InferenceMultimodal Token Compression

    Beihang University · KlingAI Research · Shanghai Jiao Tong University

  3. Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

    Oct 28, 2026Yi Wang, Tianzi Wang, Peter BellAudio-Visual QAMultimodal Agents

    School of Informatics, University of Edinburgh, UK · Department of Systems Engineering and Engineering Management, Chinese University of Hong Kong, Hong Kong SAR

  4. EmbeddingGemma 2: Unified and Efficient Multimodal Representations

    Oct 27, 2026Henrique Schechter Vera, Sahil Dua, Jovan Andonov +70Multimodal EmbeddingUnified Multimodal Models

  5. Efficient Omni-modal Large Language Model Inference via Per-Query Modality Selection

    Oct 22, 2026Shuqi Liu, Yuzhou Cao, Luke OngEfficient Multimodal InferenceMultimodal Large Language Models

    College of Computing and Data Science, Nanyang Technological University

  6. Look Before You Reason: Implicit Visual Thinking for Efficient Multimodal Reasoning

    Oct 19, 2026Zerui Chen, Changrui Chen, Fei Ni +1Multimodal ReasoningEfficient Multimodal Inference

    MVP Lab, Imperial College London

  7. TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning

    Oct 19, 2026Xiyue Zhang, Keru Zhou, Ziyi Wang +2Mixture-of-Experts InferenceEfficient Multimodal Inference

    Zhejiang University · Tsinghua University · Wenzhou Business College

  8. LightCRS: Lightweight Multi-Domain Conversational Recommendation System

    Oct 18, 2026He Ma, Lan You, Qiyang Wu +1Recommender SystemsMultimodal Recommendation

    University of Sydney Sydney, NSW, Australia · Nankai University Tianjin, China

  9. Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling

    Oct 15, 2026Tong Zhu, Zhe Xu, Zefeng He +7LLM Decision-MakingEfficient Multimodal Inference

    Shanghai AI Lab · Shanghai Jiao Tong University · Fudan University +1

  10. Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

    Oct 13, 2026Aimon Rahman, Joymallya Chakraborty, Vidit Bansal +1Multi-Image ReasoningEfficient Multimodal Inference

    Amazon

  11. Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning

    Oct 11, 2026Quanlin Chen, Xiuwei Chen, Kun Xiang +5Multimodal Large Language ModelsMultimodal Reasoning

    Sun Yat-sen University · Shenzhen Loop Area Institute · Beijing Zhongguancun Academy +2

  12. Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

    Oct 8, 2026Yicheng Xue, Han Wu, Jufeng Yang +4VLM ReasoningEfficient Multimodal Inference

    City University of Hong Kong · Zhejiang University · Peking University +2

  13. FastJEV: Understanding Redundancy for Compact JEV Inference

    Oct 8, 2026Jie Ma, Jie Gao, Yihang Liu +5Efficient Multimodal InferenceEfficient VLA Model Inference

    Xiamen University

  14. RSJEV: Discriminative Remote Sensing Scene Classification with Multimodal Large Language Models

    Oct 6, 2026Dongchen Si, Di Wang, Mingzhen Xu +3Remote Sensing Image UnderstandingEfficient Multimodal Inference

    School of Computer Science, Wuhan University, Wuhan 430072, China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

  15. VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

    Oct 6, 2026Yuan Feng, Qize Yang, Ruizhe Chen +9Efficient Multimodal InferenceMultimodal Large Language Models

    University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · The Chinese University of Hong Kong +1

  16. Decide Before You Look: Learning Which Retrieved Memories Deserve Pixels

    Oct 6, 2026Youxing LIMemory-Augmented VLMsEfficient Multimodal Inference

    The Chinese University of Hong Kong

  17. Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

    Oct 5, 2026Payal Mohapatra, Haodong Yang, Yueyuan Sui +3Efficient Multimodal InferenceMultimodal Classification

    Arm Inc., USA · Northwestern University, USA

  18. Conditional Trajectory Peaks: Single-Pass Multimodal Policies over Action Chunks

    Oct 5, 2026Di Wu, Rongtian Shen, Ping Liu +4Efficient Multimodal InferenceRobot Policy Learning

    Magiclab Robotics Technology Co., Ltd., China · Southeast University, Nanjing, China

  19. ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory

    Oct 4, 2026Hao Jiang, Zhanyu Guo, Chenwei Wu +11Memory-Augmented VLMsEfficient Multimodal Inference

    University of Electronic Science and Technology of China · University of Michigan, Ann Arbor · Nanyang Technological University +5

  20. Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition

    Oct 2, 2026Eeshaan Jain, Linus Bleistein, Bart Deplancke +1Efficient Multimodal InferenceMultimodal Foundation Models

    School of Computer and Communication Sciences, EPFL · School of Life Sciences, EPFL

  21. Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents

    Oct 1, 2026Feiyu Gavin Zhu, Xiaoyu Zhu, Jiqi Yang +9Efficient Multimodal InferenceMultimodal Reasoning

    Apple · Carnegie Mellon University

  22. HAWK: Rethinking Multimodal Drafting for Speculative Decoding

    Sep 30, 2026Wenhan Yang, Anirudh Rao, Ashwin ChandraEfficient Multimodal InferenceVLM Distillation

    University of California, Los Angeles · Samsung Research America

  23. Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillation

    Sep 30, 2026Jianghao Wang, Ke Meng, Jian Li +7Efficient Multimodal InferenceMultimodal Large Language Models

    School of Computer Science, Wuhan University · Didi Chuxing

  24. DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference

    Sep 30, 2026Xudong Tan, Peng Ye, Ming Xie +4Mixture-of-Experts PruningEfficient Multimodal Inference

    Fudan University · The Chinese University of Hong Kong

  25. Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inference

    Sep 29, 2026Luning Pang, Cheng Yuan, Jiawei Shao +2Efficient Multimodal InferenceVisual Token Compression

    Department of Electronic Engineering, and Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China · Institute of Artificial Intelligence (TeleAI), China Telecom, Beijing 100033, China

  26. OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models

    Sep 29, 2026Yuchen Deng, Zidang Cai, Feidiao Yang +4Efficient Multimodal InferenceAudio Token Compression

    Shenzhen International Graduate School, Tsinghua University, China · Pengcheng Laboratory, China

  27. SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models

    Sep 28, 2026Tianxiang Chen, Zhentao Tan, Zi Ye +9Efficient Multimodal InferenceTransformer

    End-User Intelligent Computing BU, Alibaba Cloud · Alibaba Group · Department of Computer Science, Maynooth University +2

  28. Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

    Sep 28, 2026Jingdi lei, Junxian Li, Di Zhang +3Efficient Multimodal InferenceMultimodal Large Language Models

    Nanyang Technological University · Shanghai Jiao Tong University · Fudan University +2

  29. Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models

    Sep 28, 2026Zhiqiang Xia, Yang Li, Xinyuan Zhang +5Efficient Multimodal InferenceEfficient VLM Inference

    Xiaomi Corporation, Beijing, China

  30. OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming

    Sep 28, 2026Zongshang Shen, Wangsong Yin, Daliang Xu +2Efficient Multimodal InferenceOn-Device Language Model Inference

    Key Lab of High Confidence Software Technologies (Peking University), Beijing, China · State Key Laboratory of Networking and Switching Technology (BUPT), Beijing, China