Efficient Multimodal Inference

Follow this topic Ask about this topic

Efficient multimodal inference studies how to make models process mixed inputs with less computation, memory, latency, and energy. Common approaches prune or compress visual and audio tokens, select useful video frames or input modalities, compress KV caches, and adapt reasoning depth to query difficulty. Systems work also schedules inference across heterogeneous GPUs and edge devices, with evaluations spanning image and video question answering, long-context tasks, and real-time applications. Many methods preserve accuracy at substantial compression, but an open tension is whether they retain fine-grained evidence and grounding across tasks while improving end-to-end latency and cost.

Latest papers

231 papers27 papers in the last four weeks, down 4% on the four weeks before. 0.2% of all new papers.

StreamWise: Serving Multi-Modal Generation in Real-Time at Scale
Mar 6, 2026Haoran Qiu, Gohar Irfan Chaudhry, Chaojie Zhang +4Efficient Multimodal Inference Streaming Video Generation
Microsoft Azure Research Redmond, USA · MIT CSAIL Cambridge, USA
AutothinkRAG: Complexity-Aware Control of Retrieval-Augmented Reasoning for Image-Text Interaction
Mar 5, 2026Jiashu Yang, Chi Zhang, Abudukelimu Wuerkaixi +5Multimodal RAG Multimodal QA
Dalian University of Technology · Meituan LongCat Interaction Team · Tsinghua University
PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Driving
Mar 4, 2026Yi Zhang, Xian Zhang, Saisi Zhao +4Multimodal Model Routing Autonomous Driving
Chair of Robotics, Artificial Intelligence and Embedded Systems Technical University of Munich (TUM) Munich, Germany
Phi-4-reasoning-vision-15B Technical Report
Mar 4, 2026Jyoti Aneja, Michael Harrison, Neel Joshi +3VLM Reasoning MLLM Reasoning
Microsoft Research
TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval
Mar 3, 2026Xiangzhao Hao, Shijie Wang, Tianyu Yang +3Multimodal IR Multimodal Embedding
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Orchestrating Multimodal DNN Workloads in Wireless Neural Processing
Mar 2, 2026Sai Xu, Kai-Kit Wong, Yanan Du +1Efficient Multimodal Inference Edge Inference
Department of Electronic and Electrical Engineering, University College London, WC1E 7JE, London, UK · Department of Electronic Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do 17104, Republic of Korea · School of Electrical and Electronic Engineering, the University of Sheffield, Sheffield S10 2TN, UK +1
Proof-of-Perception: Certified Tool-Using Multimodal Reasoning with Compositional Conformal Guarantees
Feb 27, 2026Arya Fayyazi, Haleh AkramiMultimodal Reasoning Multimodal Tool Use
University of Southern California, Los Angeles, CA · Nuro, Mountain View, CA
Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
Feb 27, 2026Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1Multimodal Learning Unified Multimodal Models
City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom
HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework
Feb 27, 2026Jiacheng Yang, Anqi Chen, Yunkai Dang +5Efficient Multimodal Inference VLM Reasoning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Institute of Brain-inspired Intelligence, Nanjing University, Nanjing, China · Shenzhen Research Institute of Nanjing University, Shenzhen, China
Efficient Encoder-Free Fourier-based 3D Large Multimodal Model
Feb 26, 2026Guofeng Mei, Wei Lin, Luigi Riz +33D VLMs Efficient Multimodal Inference
Fondazione Bruno Kessler, Italy · JKU Linz, Austria · CSIRO, Australia
Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device
Feb 23, 2026Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad +8Unified Multimodal Models Multimodal Generation
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University · Linköping University
IDSelect: A RL-Based Cost-Aware Selection Agent for Video-based Multi-Modal Person Recognition
Feb 22, 2026Yuyang Ji, Yixuan Shen, Kien Nguyen +2Efficient Multimodal Inference Model Selection
Department of Computer Science, Drexel University · School of Electrical Engineering and Robotics, Queensland University of Technology · Department of Electrical and Computer Engineerin, Drexel University
EntropyPrune: Matrix Entropy Guided Visual Token Pruning for Multimodal Large Language Models
Feb 19, 2026Yahong Wang, Juncheng Wu, Zhangkai Ni +6Token Pruning Visual Token Pruning
Tongji University · University of California, Santa Cruz · East China Normal University +1
HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis
Feb 18, 2026J. Dhar, M. K. Pandey, D. Chakladar +4Multimodal Fusion Multimodal Learning
Indian Institute of Technology Ropar, India · RoentGen Health, India · Lulea University of Technology, Sweden +4
Effective and Robust Multimodal Medical Image Analysis
Feb 17, 2026Joy Dhar, Nayyar Zaidi, Maryam HaghighatMedical Image Analysis Multimodal Fusion
Indian Institute of Technology Ropar, Rupnagar, Punjab, India · Deakin University, Melbourne, Victoria, Australia · Queensland University of Technology, Brisbane, QLD, Australia
Multi-Agentic AI for Fairness-Aware and Accelerated Multi-modal Large Model Inference in Real-world Mobile Edge Networks
Feb 6, 2026Haiyuan Li, Hari Madhukumar, Shuangyi Yan +2Efficient Multimodal Inference LLM Inference Scheduling
Smart Internet Lab, Department of Electrical and Electronic Engineering, University of Bristol, BS8 1QU, U.K.
Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs
Feb 5, 2026Qi Li, Yanzhe Zhao, Yongxin Zhou +4Efficient Multimodal Inference Multimodal Large Language Models
Honor Device Co., Ltd., China
EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation
Sep 15, 2025Jingyu Xiao, Zhongyi Zhang, Yuxuan Wan +3Efficient Multimodal Inference Code Generation
The Chinese University of Hong Kong, Hong Kong, China · Huazhong University of Science and Technology, Wuhan, China · Singapore Management University, Singapore, Singapore +1
Multimodal Remote Inference
Aug 11, 2025Keyuan Zhang, Yin Sun, Bo JiSemi-Markov Decision Process Efficient Multimodal Inference
Department of Computer Science, Virginia Tech · Department of Electrical and Computer Engineering, Auburn University
Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation
Aug 7, 2025Haonan Shangguan, Xiaocui Yang, Shi Feng +3Efficient Multimodal Inference Sentiment Analysis
School of Computer Science and Engineering, Northeastern University, Shenyang, China
Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment
Jan 30, 2025Joanna Hong, Sanjeel Parekh, Honglie Chen +4Missing-Modality Learning Audio-Visual Understanding
Meta

Mar 6, 2026J/K move · Enter open · S save

StreamWise: Serving Multi-Modal Generation in Real-Time at Scale

Haoran Qiu, Gohar Irfan Chaudhry, Chaojie Zhang, Íñigo Goiri+3

Microsoft Azure Research Redmond, USA · MIT CSAIL Cambridge, USA

Efficient Multimodal Inference Streaming Video Generation Multimodal Generation

Open paper AskPDF arXiv

Advances in multi-modal generative models are enabling new applications, from storytelling to automated media synthesis. Most current workloads generate simple outputs (e.g., image generation from a prompt) in batch mode, often requiring several seconds even for basic results. Serving real-time multi-modal workflows at scale is costly and complex, requiring efficient coordination of diverse models (each with unique resource needs) across language, audio, image, and video, all under strict latency and resource constraints. We tackle these challenges through the lens of real-time podcast video generation, integrating LLMs, text-to-speech, and video-audio generation. To meet tight SLOs, we design an adaptive, modular serving system, StreamWise, that dynamically manages quality (e.g., resolution, sharpness), model/content parallelism, and resource-aware scheduling. We leverage heterogeneous hardware to maximize responsiveness and efficiency. For example, the system can lower video resolution and allocate more resources to early scenes. We quantify the trade-offs between latency, cost, and quality. The cheapest setup generates a 10-minute podcast video on A100 GPUs in 1.4 hours (8.4x slower than the real-time) for less than $25. StreamWise enables high-quality real-time streaming with a sub-second startup delay under $45.