Multimodal Agents

Follow this topic Ask about this topic

Multimodal agents combine language models with visual, audio, and video perception to answer questions, use tools, and interact with apps, websites, robots, or other environments. Common approaches include planning and tool orchestration, specialized collaborating agents, active evidence gathering, and memory or reusable skills for long tasks; reinforcement learning and execution feedback are increasingly used to improve these behaviors. Benchmarks now test browsing, GUI use, video reasoning, and embodied tasks, often evaluating action traces and process reliability as well as final answers. Despite progress, agents remain brittle over long horizons, and more modalities or more agents do not automatically improve grounding, safety, efficiency, or task success.

Latest papers 367

49 papers in the last four weeks, up 20% on the four weeks before. 0.3% of all new papers.

All topics

Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution
May 9, 2026Feng Xiong, Zengbin Wang, Yong Wang +5Continual Learning for LLM Agents Agent Skill Learning
1AMAP, Alibaba Group · 2CASIA · 3BNU
Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck
May 8, 2026Zihan Huang, Junda Wu, Tong Yu +6LLM Agent Skill Learning Multimodal Agents
UC San Diego · Adobe Research · UC Santa Barbara +1
EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams
May 8, 2026Dongchuan Ran, Linyu Ou, Xueheng Li +5Egocentric Video Understanding AI Agent Benchmarks
Sensetime Reasearch · Beijing Institute of Technology · University of Science and Technology of China
HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents
May 8, 2026Guankai Li, Jiabin Chen, Yi Xu +2Efficient Multimodal Inference Multimodal Search Agents
Xiaohongshu Inc. · University of Cambridge
Cross-Modal Navigation with Multi-Agent Reinforcement Learning
May 7, 2026Shuo Liu, Xinzichen Li, Christopher AmatoCross-Modal Learning Multi-Agent Collaboration
Khoury College of Computer Sciences Northeastern University Boston, MA, 02120
X-OmniClaw Technical Report: A Unified Mobile Agent for Multimodal Understanding and Interaction
May 7, 2026Xiaoming Ren, Ru Zhen, Chao Li +11Multimodal Grounding GUI Agents
Multi-X Team, OPPO AI Center
OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents
May 6, 2026Shuang Chen, Kaituo Feng, Hangting Chen +7Multimodal IR Multimodal Search Agents
Tencent Hunyuan · University of California, Los Angeles · The Chinese University of Hong Kong +1
Bridging Perception and Action: A Lightweight Multimodal Meta-Planner Framework for Robust Earth Observation Agents
May 6, 2026Jinghui Xu, Boyi Shangguan, Mengke Zhu +10Remote Sensing Tool-Use Planning
State Key Laboratory of Space Information System and Integrated Application, Beijing 100095, China · Beijing Institute of Satellite Information Engineering, Beijing 100095, China · Shenzhen International Graduate School, Tsinghua University, Shenzhen, China +2
Pro $^2$ Assist: Continuous Step-Aware Proactive Assistance with Multimodal Egocentric Perception for Long-Horizon Procedural Tasks
May 5, 2026Lilin Xu, Bufang Yang, Siyang Jiang +6Proactive Assistance Multimodal Agents
Columbia University, United States · The Chinese University of Hong Kong, China
Enwar 3.0: An Agentic Multi-Modal LLM Orchestrator for Situation-Aware Beamforming, Blockage Prediction, and Handover Management
May 4, 2026Ahmad M. Nazar, Abdulkadir Celik, Asmaa Abdallah +3Adaptive Model Routing Wireless Communications
Department of Electrical and Computer Engineering, Iowa State University (ISU), Ames, IA, 50014, USA · Gladiolus Technological Institute, USA · School of Electronics and Computer Science, University of Southampton, Southampton SO17 1BJ, U.K. +2
BaRA: Budget-constrained and Reliable Web Data Collection Agent
May 2, 2026Soojeong Lee, Joseph Lee, Yongseong Cho +3Multimodal Agents Web Agents
Department of Statistics and Data Science, Yonsei University, Seoul, Republic of Korea · Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea · HUSTLERS Corp., Seoul, Republic of Korea
From Static Analysis to Audience Dissemination: A Training-Free Multimodal Controversy Detection Multi-Agent Framework
May 1, 2026Zihan Ding, Ziyuan Yang, Yi ZhangSocial Media Analysis Multimodal Classification
Sichuan University, Chengdu, Sichuan, China · Nanyang Technological University, Singapore
ARIS: Agentic and Relationship Intelligence System for Social Robots
May 1, 2026Stavya Datta, Fucai Ke, Leimin Tian +1Agentic RAG Human-Robot Interaction
Monash University
Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation
Apr 30, 2026Jing Zhang, Wentao Jiang, Tao Huang +8Medical Image Analysis Multimodal Reasoning
School of Computer Science, Wuhan University, China · The Central Hospital of Wuhan, China · Taizhou Hospital of Zhejiang Province, China +1
A Pattern Language for Resilient Visual Agents
Apr 30, 2026Habtom Kahsay Gidey, Alexander Lenz, Alois KnollVision-Language-Action Models Multimodal Foundation Models
Technische Universität München, München, Germany
MM-StanceDet: Retrieval-Augmented Multi-modal Multi-agent Stance Detection
Apr 30, 2026Weihai Lu, Zhejun Zhao, Yanshu Li +1Multimodal Reasoning Multimodal Classification
Peking University · Baidu Inc · Brown University +1
InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
Apr 30, 2026Qiyao Wang, Haoran Hu, Longze Chen +4Web Application Generation Web Agent Benchmarks
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Dalian University of Technology +2
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
Apr 30, 2026Junbo Cui, Bokai Xu, Chongyi Wang +33Efficient Multimodal Inference Unified Multimodal Models
MiniCPM-o Team, OpenBMB
GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
Apr 29, 2026GLM-V Team, :, Wenyi Hong +95Multimodal Large Language Models Multimodal Reasoning
Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces
Apr 26, 2026Zijing Shi, Meng Fang, Ling ChenWeb Agent Benchmarks AI Agent Safety
AAII, University of Technology Sydney, NSW, Australia · University of Liverpool, Liverpool, UK
ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Apr 26, 2026Fanqing Meng, Lingxiao Du, Zijian Wu +46Computer-Use Agent Benchmarks LLM Agent Evaluation
Sema: Semantic Transport for Real-Time Multimodal Agents
Apr 22, 2026Jiaying Meng, Bojie LiSemantic Communication Task-Oriented Semantic Communication
Unaffiliated · Pine AI
ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
Apr 22, 2026Wentao Yan, Shengqin Wang, Huichi Zhou +4Multimodal IR Process Reward Models
East China Normal University · Huawei Noah’s Ark Lab · University College London +2
A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding
Apr 21, 2026Shuai Wang, Hongyi Zhu, Jia-Hong Huang +6Multimodal RAG Multimodal Grounding
University of Amsterdam · Part of this work was done during interning at Amazon. · Also with Amazon AGI. +2
DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents
Apr 21, 2026Shengqin Wang, Wentao Yan, Huichi Zhou +4Reward Shaping RL for Language Model Reasoning
East China Normal University · University College London · Huawei Noah’s Ark Lab +2
A Multi-Agent Framework with Structured Reasoning and Reflective Refinement for Multimodal Empathetic Response Generation
Apr 21, 2026Liping Wang, Cheng Ye, Weidong Chen +3LLM Self-Refinement Multimodal Reasoning
School of Information Science and Technology, USTC Hefei, China
Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots
Apr 20, 2026Shiquan Zhang, Tianyi Zhang, Le Fang +3Mobile GUI Automation LLM Agent Evaluation
School of Computing and Information Systems, University of Melbourne, Melbourne, Australia · School of Computer Science, University of Auckland, Auckland, New Zealand
SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topology
Apr 19, 2026Zheng Nie, Ruolin Shen, Xinlei Yu +3Multi-Agent Collaboration Dynamic Graph Learning
National University of Singapore, Singapore · Technical University of Munich, Munich, Germany · Zhejiang University, China
PersonalHomeBench: Evaluating Agents in Personalized Smart Homes
Apr 18, 2026Manasa Bharadwaj, Yolanda Liu, InJung Yang +5AI Agent Evaluation Multimodal Agents
LG Toronto AI Lab · LG Electronics, Korea.
MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation
Apr 16, 2026Yan Li, Zezi Zeng, Yifan Yang +12Web Application Generation Web Agent Benchmarks
Shanghai Jiao Tong University · Xi’an Jiaotong University · Microsoft Corporation +1

May 9, 2026J/K move · Enter open · S save

Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution

Feng Xiong, Zengbin Wang, Yong Wang, Xuecai Hu+4

1AMAP, Alibaba Group · 2CASIA · 3BNU

Continual Learning for LLM Agents Agent Skill Learning Multimodal Agents Memory-Augmented Agents Self-Evolving Agents

Open paper AskPDF arXiv

Self-evolving agents present a promising path toward continual adaptation by distilling task interactions into reusable knowledge artifacts. In practice, this paradigm remains hindered by two coupled bottlenecks: data inefficiency, where costly rollout effort is disproportionately spent on low-value samples rather than informative ones, and knowledge interference, where heterogeneous knowledge stored in shared repositories leads to noisy retrieval and task-misaligned guidance. Together, these issues form a self-reinforcing failure loop in which uninformative rollouts yield noisy knowledge, which in turn degrades subsequent rollouts. In this work, we introduce Ace-Skill, a co-evolutionary framework that jointly optimizes rollout allocation and knowledge organization for self-evolving multimodal agents. Specifically, Ace-Skill combines aprioritized sampler with lazy-decay proficiency tracking to focus rollouts on informative and insufficiently mastered samples, and a clustered organizer that semantically clusters knowledge for cleaner retrieval and more reliable adaptation. By improving sampling and organization together, Ace-Skill turns self-evolution into a virtuous cycle in which more informative rollouts produce higher-quality knowledge that supports stronger subsequent rollouts. Across four multimodal tool-use benchmarks, Ace-Skill delivers strong gains (e.g., +35.46% relative improvement in Avg@4 accuracy), enabling an opensource 35B MoE model to match or surpass proprietary models. The acquired knowledge also transfers effectively in a zero-shot manner to smaller 9B and 4B models, allowing resource-constrained agents to inherit advanced capabilities without additional training. The code has been publicly available at https://github.com/AMAP-ML/Ace-Skill.