Efficient Multimodal Inference

Follow this topic Ask about this topic

Momentum

28 papers in the last four weeks, up 211% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 157

All topics

SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning
Mar 24, 2026Haoyu Huang, Jinfa Huang, Zhongwei Wan +3Efficient Multimodal Inference LLM Inference Acceleration
Xiamen University, Xiamen, Fujian, China · University of Rochester, Rochester, NY, USA · The Ohio State University, Columbus, OH, USA
DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression
Mar 15, 2026Bingzhou Li, Tao HuangAudio-Visual Understanding Efficient Multimodal Inference
Shanghai Jiao Tong University, Shanghai, China · Tongji University, Shanghai, China
HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework
Feb 27, 2026Jiacheng Yang, Anqi Chen, Yunkai Dang +5Efficient Multimodal Inference VLM Reasoning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Institute of Brain-inspired Intelligence, Nanjing University, Nanjing, China · Shenzhen Research Institute of Nanjing University, Shenzhen, China
Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs
Feb 5, 2026Qi Li, Yanzhe Zhao, Yongxin Zhou +4Efficient Multimodal Inference Multimodal Large Language Models
Honor Device Co., Ltd., China
EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation
Sep 15, 2025Jingyu Xiao, Zhongyi Zhang, Yuxuan Wan +3Efficient Multimodal Inference Code Generation
The Chinese University of Hong Kong, Hong Kong, China · Huazhong University of Science and Technology, Wuhan, China · Singapore Management University, Singapore, Singapore +1
Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation
Aug 7, 2025Haonan Shangguan, Xiaocui Yang, Shi Feng +3Efficient Multimodal Inference Sentiment Analysis
School of Computer Science and Engineering, Northeastern University, Shenyang, China
Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment
Jan 30, 2025Joanna Hong, Sanjeel Parekh, Honglie Chen +4Missing-Modality Learning Audio-Visual Understanding
Meta

Mar 24, 2026, cs.CVJ/K move · Enter open · S save

SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

Haoyu Huang, Jinfa Huang, Zhongwei Wan, Xiawu Zheng+2

Xiamen University, Xiamen, Fujian, China · University of Rochester, Rochester, NY, USA · The Ohio State University, Columbus, OH, USA

Efficient Multimodal Inference LLM Inference Acceleration Multimodal Agents

Open paper AskPDF arXiv

Agentic multimodal large language models (MLLMs) (e.g., OpenAI o3 and Gemini Agentic Vision) achieve remarkable reasoning capabilities through iterative visual tool invocation. However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead. This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level concurrency. To this end, we propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck. Our key insight is that a lightweight, tool-free MLLM can serve as a speculative planner to predict the execution trajectory, enabling early termination of expensive tool chains without sacrificing accuracy. To regulate this speculative planning, we introduce a cognitive gating mechanism based on answer separability, which quantifies the model's confidence for self-verification without requiring oracle labels. Furthermore, we design a heterogeneous parallel funnel that exploits the stateless concurrency of the small model to mask the stateful serial execution of the large model, maximizing system throughput. Extensive experiments on V* Bench, HR-Bench, and POPE demonstrate that SpecEyes achieves 1.1-3.35x speedup over the agentic baseline while preserving or even improving accuracy (up to +6.7%), thereby boosting serving throughput under concurrent workloads.

Momentum

Related topics

Latest papers 157