Multimodal Agents

Follow this topic Ask about this topic

Multimodal agents combine language models with visual, audio, and video perception to answer questions, use tools, and interact with apps, websites, robots, or other environments. Common approaches include planning and tool orchestration, specialized collaborating agents, active evidence gathering, and memory or reusable skills for long tasks; reinforcement learning and execution feedback are increasingly used to improve these behaviors. Benchmarks now test browsing, GUI use, video reasoning, and embodied tasks, often evaluating action traces and process reliability as well as final answers. Despite progress, agents remain brittle over long horizons, and more modalities or more agents do not automatically improve grounding, safety, efficiency, or task success.

Latest papers

396 papers38 papers in the last four weeks, down 19% on the four weeks before. 0.2% of all new papers.

VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks
Nov 3, 2026Tengju Ru, Weitong Lian, Zecong Tang +12Multimodal Dataset Multimodal Instruction Tuning
Zhejiang University · The University of Hong Kong · Institute of Automation, Chinese Academy of Sciences +1
Omni-Life: Benchmarking Omnimodal Real-Time AI Assistants in People's Daily Lives
Nov 2, 2026Mengze Hong, Ziwei Shang, Di Liang +9Multimodal Agent Evaluation Multimodal Agents
Hong Kong Polytechnic University · Today AI · Fudan University +1
RxnMiner: A Multimodal Agent for Structured Reaction Data Extraction from Chemistry Figures
Nov 2, 2026Jinbo Hu, Songren Zhang, Chuanxun Fu +19Document Information Extraction Multimodal Agents
DP Technology · Peking University · The Hong Kong University of Science and Technology +1
Multi-Modal Multi-Agent System for Bridge Operation and Maintenance based on Large Language Model
Nov 2, 2026Xiangyun Luo, Yunkai Zhang, Yong Huang +2Multi-Agent LLM Systems Multimodal Agents
Key Lab of Smart Prevention and Mitigation of Civil Engineering Disasters of the Ministry of Industry and Information Technology, Harbin Institute of Technology, Harbin 150090, China · Key Lab of Structures Dynamic Behavior and Control of the Ministry of Education, School of Civil Engineering, Harbin Institute of Technology, Harbin 150090, China · The Institute of Statistical Mathematics, Research Organization of Information and Systems, 10-3 Midori-cho, Tachikawa, Tokyo 190-8562, Japan +1
Parallelism-Aligned Pruning and Round-Aware Context Reuse for On-Device Multimodal Agents
Nov 2, 2026Peipei Jiao, Mingkang Liu, Weiming Meng +3Efficient Multimodal Inference Multimodal Agents
Juhaokan Technology Co., Ltd.
MAGAR: Multimodal Agents for GUI-Based 3D Mechanical Assembly and Minimal Rework
Nov 2, 2026Jingxuan Wei, Jiahao Fu, Fuqiang Wang +5Multimodal Agents AI Agent Reliability
University of Chinese Academy of Sciences, Beijing, China · Qilu University of Technology (Shandong Academy of Sciences), Shandong Computer Science Center, Jinan, China · China University of Petroleum (East China), China +2
MAPLE: A Multimodal Mobile Agent with Playbook-Learned Experience
Oct 31, 2026Jun Wang, Aurora Wei, Lucas Spangher +3Mobile GUI Agents Multimodal Agents
Google · Washington University in St. Louis
MRC-MMAgent: Reinforcement Learning for Multi-Role Collaboration in Multimodal Tool-Using Agents
Oct 29, 2026Shengqin Wang, Jie Jin, Xiaofan Li +8Multimodal Agents RL for Tool Use
Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation
Oct 28, 2026Yi Wang, Tianzi Wang, Peter BellAudio-Visual QA Multimodal Agents
School of Informatics, University of Edinburgh, UK · Department of Systems Engineering and Engineering Management, Chinese University of Hong Kong, Hong Kong SAR
Live-OPSD: On-Policy Self-Distillation for Proactive Streaming Large Video Models
Oct 27, 2026Kaixuan Du, Xin Wan, Dai Guan +3On-Policy Distillation Multimodal Agents
Qwen Business Unit of Alibaba
The Trinity: Capability-Driven Recursive Self-Improvement for Multimodal Agents
Oct 26, 2026Weidong Tang, Ruiheng Wang, Jingxuan Zhang +18Self-Improving Agents Multimodal Agents
Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams
Oct 26, 2026Yinsong Yan, Ling Wang, Yuxuan Wang +11Multimodal Agent Evaluation MLLM Evaluation
The Hong Kong Polytechnic University · Alibaba Token Hub, Alibaba Group · Shanghai Jiaotong University +2
Adaptive Control of Scientific Instruments with Nonstationary Responses through Multimodal Agents
Oct 23, 2026Guoyang Shi, Hongwen Li, Yiding Yu +6Adaptive Control Language Model-Based Control
Beijing ENN Fusion Energy Science and Technology Co., Ltd., Beijing 101111, China · Beijing Key Laboratory of High Magnetic Field Spherical Torus Fusion Energy, Beijing 101111, China · Hebei Key Laboratory of Compact Fusion, Langfang 065000, China
Dual Exam: Controlled Evaluation of Multimodal Agents and World Models
Oct 22, 2026Zhefan Rao, Meng Chu, Xuan Billy Zhang +89Multimodal Agent Evaluation Agent Evaluation
Secure Agentic AI and Multimodal Intelligence in 6G Communication Systems
Oct 21, 2026Satyam Gaba, Devanshu Motwani, Siva Sai +3AI Agent Security Multimodal Agents
Qualcomm Wireless Research, San Diego, CA 92121, USA · Department of Computer Science and Engineering, BITS Pilani, Pilani Campus, Rajasthan 333031, India · Department of Electrical and Computer Engineering, National University of Singapore 117583, Singapore +1
Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use
Oct 21, 2026Yanzhe Chen, Ziyi Yang, Jifeng Zhu +14Multimodal Agents Embodied AI
Show Lab, National University of Singapore
TARSphere: A Robot-Oriented Full-Stack Framework for Long-Horizon Embodied Agents in Evolving Physical and Social Environments
Oct 21, 2026Yicheng Xu, Zhikai Xu, Ziyuan Chen +5Long-Horizon Agent Evaluation Embodied Agent Benchmarks
Zhejiang University · Fudan University
Remembering Through Change: Provenance-Preserving Hypergraph Memory for Multimodal Agents
Oct 21, 2026Jielong Tang, Ling-An Zeng, Xujie Yuan +7Agent Memory Multimodal Memory
Sun Yat-sen University · Huawei Technologies, Noah Arks’ Lab · Shandong Normal University +1
DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams
Oct 20, 2026Yuhang Chen, Guangyue Liu, Shilong LiuMultimodal Model Evaluation Multimodal Large Language Models
Huazhong University of Science and Technology · Princeton University
Reusing Multimodal Agents: Efficient Evaluation of Sound Designs in Games
Oct 20, 2026Thai Nguyen Van, Ibrahim Khan, Mury Fajar Dewantoro +3Automated Agent Evaluation Multimodal Agents
Graduate School of Information Science and Engineering, Ritsumeikan University, Japan · College of Information Science and Engineering, Ritsumeikan University, Japan · Independent Researcher +1
Focused Omni-Modal GUI Agents
Oct 19, 2026Fuyao Zhang, Wenwen He, Jiaming Zhang +6GUI Agents Multimodal Agents
Nanyang Technological University · Alibaba Group
MIRA: A Multimodal Illustrated Deep Research Agent for Controllable Reports
Oct 19, 2026Hongrui Wu, Kaiwen Tuo, Congcong Wang +7Deep Research Agents Multimodal Agents
Stanford University · The Hong Kong University of Science and Technology · Fudan University +2
InteractAgent: Agentic Human Motion Interaction with Memory-Augmented Reflection
Oct 19, 2026Senhe Zhang, Tao Zhang, Yujia Zhang +2Long-Horizon Agent Tasks Multimodal Agents
AI Geeks · UNSW Sydney
ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games
Oct 16, 2026Dawei Wang, Di Zhao, Jianran Yu +8AI Agent Benchmarks Multimodal Agents
Newcastle University · University of Auckland · CTW Inc.
Jarvis: Coordination Deserves a Brain of Its Own in Agentic Interactive Models
Oct 16, 2026Yuxiang Wang, Shengbo Cai, Zhixian Zhao +7LLM Agent Harnesses Multi-Agent Coordination
The Chinese University of Hong Kong, Shenzhen · Tencent Hunyuan · Amphion Technology Co., Ltd.
MobileInject: Evaluating Reasoning-to-Action Security of MLLM-Based Mobile Agents under Cross-Modal Consistency Injection
Oct 16, 2026Songping Wang, Peiyang Li, Yi Huang +5LLM Agent Security AI Agent Security Benchmarks
Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?
Oct 15, 2026Jingyu Tang, Xu Yang, Zhehao Zhang +7Agent Evaluation AI Agent Benchmarks
University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign +3
DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding
Oct 15, 2026Zhuowen Liang, Xiaotian Lin, Yuyu Luo +2Document QA Multimodal Agents
The Hong Kong University of Science and Technology (Guangzhou) · EvenUp, USA
EvolvingBrain: An Adaptive Skill System for Multi-modal Agents Self-Evolution in Open Worlds
Oct 14, 2026Runxuan Liu, Jingqi Shi, Baoxin Wang +10Self-Evolving Agents Agent Skill Learning
Harbin Institute of Technology · iFLYTEK · Meituan +1
HM3D-M3: A Benchmark for Multi-Agent, Multi-Modal, Multi-Object Goal Navigation
Oct 14, 2026Hongyu Ding, Kanyu Xu, Mingjia Chen +5Robot Navigation Benchmarks Robot Navigation
Hongyu Ding and Jing Huo are with the School of Computer Science, Nanjing University, China. · Hongyu Ding and Yifan Zhang are with the Institute of Automation, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences, Nanjing, China. · Kanyu Xu, Zixuan Chen, Jieqi Shi and Yang Gao are with the School of Intelligence Science and Technology, Nanjing University, China. +1

Nov 3, 2026J/K move · Enter open · S save

VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

Tengju Ru, Weitong Lian, Zecong Tang, Lingyi Meng+11

Zhejiang University · The University of Hong Kong · Institute of Automation, Chinese Academy of Sciences+1

Multimodal Dataset Multimodal Instruction Tuning Multimodal Agents

Open paper AskPDF arXiv

The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data. While raw videos offer an abundant and dynamic source of visual knowledge, transforming them into precise instruction data remains highly challenging. To bridge this gap, we present VidUEU-Agent, an automated pipeline that synthesizes instruction data from raw videos for multimodal understanding, editing, and unified tasks. VidUEU-Agent follows a four-stage pipeline: it performs global perception to filter low-quality shots, mines keyframes and multimodal signals, constructs metadata and routes the metadata to synthesize instruction samples. More importantly, our agent framework can autonomously iterate the data distribution based on downstream training results, thereby optimizing data quality, balance, and training effectiveness. Scaling this pipeline, we construct VidUEU-100K, a large-scale dataset of 100K training samples spanning multiple domains and tasks, providing a robust foundation for developing comprehensive multimodal capabilities. Extensive experiments demonstrate that fine-tuning baseline models on VidUEU-100K yields significant and consistent performance gains across diverse representative benchmarks, validating the effectiveness of our proposed agent framework and dataset.