Multimodal Agents

Follow this topic Ask about this topic

Latest papers 367

49 papers in the last four weeks, up 20% on the four weeks before. 0.3% of all new papers.

All topics

XSkill: Continual Learning from Experience and Skills in Multimodal Agents
Mar 12, 2026Guanyu Jiang, Zhaochen Su, Xiaoye Qu +1Agent Skill Learning Multimodal Agents
Hong Kong University of Science and Technology · Zhejiang University · Huazhong University of Science and Technology
One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries
Mar 12, 2026Mayank Saini, Arit Kumar BishwasMultimodal Agents LLM Agent Orchestration
PwC US
Agentic AI for Embodied-enhanced Beam Prediction in Low-Altitude Economy Networks
Mar 12, 2026Min Hao, Zhizhuo Li, Zirui Zhang +3Wireless Communications Multimodal Agents
School of Electronic Science and Engineering, South China Normal University, Foshan, China · School of Intelligent Engineering, Shaoguan university, Shaoguan, China · School of Automation, Guangdong University of Technology, Guangzhou, China
MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems
Mar 10, 2026Yunhang Qian, Xiaobin Hu, Jiaquan Yu +6Multi-Agent Systems Multimodal Agents
National University of Singapore · University of Science and Technology of China · Fudan University +3
MM-tau-p $^2$ : Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings
Mar 10, 2026Anupam Purwar, Aditya ChoudharyAgent Evaluation AI Agent Benchmarks
Sprinklr AI, Gurugram, India
Meissa: Multi-modal Medical Agentic Intelligence
Mar 9, 2026Yixiong Chen, Xinyi Bai, Yue Pan +2Medical LLMs Multimodal Agents
Johns Hopkins University, Baltimore, MD, USA · Cornell University, Ithaca, NY, USA
ImageEdit-R1: Boosting Multi-Agent Image Editing via Reinforcement Learning
Mar 9, 2026Yiran Zhao, Yaoqi Ye, Xiang Liu +2Image Editing RL for Image Editing
National University of Singapore · Adobe Research
PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents
Mar 9, 2026Yuxiang Chai, Shunye Tang, Han Xiao +2GUI Agents AI Agent Benchmarks
Nankai University · Huawei Research
DERM-3R: A Resource-Efficient Multimodal Agents Framework for Dermatologic Diagnosis and Treatment in Real-World Clinical Settings
Mar 6, 2026Jirui Dai, Zhendong Wang, Chongjing Wang +7Multimodal Agents Clinical Decision-Making
Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline
Mar 5, 2026Guo Chen, Lidong Lu, Yicheng Liu +17Temporal Video Understanding Video Understanding
1Nanjing University · 2NVIDIA · 3Zhejiang University +2
Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks
Mar 4, 2026Haoyu Liu, Dingcheng Li, Lukas Rutishauser +1Adversarial Training Multimodal Robustness
UC Berkeley, IEOR & BAIR · Google · Google Deepmind
Proact-VL: A Proactive VideoLLM for Real-Time AI Companions
Mar 3, 2026Weicai Yan, Yuhong Dai, Qi Ran +6AI Companions Vision-Language Agents
Zhejiang University · Shenzhen University · South China University of Technology +2
PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR
Mar 2, 2026Esen K. Tütüncü, Qian Zhou, Frederik Brudy +2Interactive Storytelling Human-AI Co-Creation
Institute of Neurosciences of the University of Barcelona, Barcelona, Spain · Autodesk Research, San Francisco, USA · Autodesk Research, Toronto, Canada
CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework
Mar 2, 2026Yuexi Du, Jinglu Wang, Shujie Liu +2Medical VLMs Medical Reasoning
Microsoft Research Asia · Department of Biomedical Engineering, Yale University · Department of Radiology & Biomedical Imaging, Yale University
SciDER: Scientific Data-centric End-to-end Researcher
Mar 2, 2026Ke Lin, Owais Aijaz, Yilin Lu +3AI Agents for Scientific Discovery Scientific Workflow Automation
Data Science, College of William & Mary, Williamsburg, USA · Natural Language Processing, MBZUAI, Abu Dhabi, UAE · Computer Science, University of Minnesota, Minneapolis, USA +1
U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation
Feb 27, 2026Xiang Deng, Feng Gao, Yong Zhang +5Multimodal Agents Unified Multimodal Models
Tsinghua University · Meituan
AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
Feb 26, 2026Zhaochen Su, Jincheng Gao, Hangyu Guo +10AI Agent Benchmarks Tool-Using Vision-Language Agents
Hong Kong University of Science and Technology · University of North Carolina at Chapel Hill · Zhejiang University +1
Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study
Feb 26, 2026Zihao Zhao, Frederik Hauke, Juliana De Castilhos +2Medical Image Classification Multimodal Agents
Department of Diagnostic and Interventional Radiology, University Hospital Aachen, 52074 Aachen, Germany
WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents
Feb 26, 2026Runwei Guan, Shaofeng Liang, Ningwei Ouyang +9Video QA Video Reasoning
Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Hubei Key Laboratory of Inland Shipping Technology (Wuhan University of Technology), Wuhan, China · School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China. +5
OmniGAIA: Towards Native Omni-Modal AI Agents
Feb 26, 2026Xiaoxi Li, Wenxiang Jiao, Jiarui Jin +10Multimodal Reasoning AI Agent Benchmarks
1Renmin University of China · 2Xiaohongshu Inc. · 3Peking University +2
ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices
Feb 25, 2026Dezhi Kong, Zhengzhao Feng, Qiliang Liang +12AI Agent Benchmarks Agentic AI
HyperAI Team, Xiaomi Corporation · Zhejiang University · Peking University +1
Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Feb 24, 2026Inderjeet Singh, Vikas Pahuja, Aishvariya Priya Rathina Sabapathy +8AI Agent Security RAG Security
Fujitsu Research of Europe, UK · Fujitsu Limited, Japan
PyVision-RL: Forging Open Agentic Vision Models via RL
Feb 24, 2026Shitian Zhao, Shaoheng Lin, Ming Li +4Multimodal Agents RL for VLMs
Shanghai AI Lab · UMD · CUHK +3
MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents
Feb 24, 2026Zhenyu Wang, Xiaofen Xing, Yirong Chen +1Agent Evaluation Multimodal Agents
South China University of Technology, Guangzhou 510641, China · Foshan University, Foshan 528225, China
MMA: Multimodal Memory Agent
Feb 18, 2026Yihao Lu, Wanru Cheng, Zeyu Zhang +1Multimodal Agents Memory-Augmented Agents
School of Computer Science, Peking University
Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU
Feb 17, 2026Rehana Mahfuz, Yinyi Guo, Erik Visser +1Conversational Agents Task-Oriented Dialogue Systems
Qualcomm Technologies, Inc.
Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation
Feb 17, 2026Kaito Baba, Risa Kishikawa, Satoshi KoderaRadiology Report Generation Multimodal Agents
Department of Cardiovascular Medicine The University of Tokyo Hospital, Tokyo, Japan
Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems
Feb 17, 2026Xiaoze Liu, Ruowang Zhang, Weichen Yu +7Multi-Agent Systems Multimodal Agents
Purdue University · Contextual AI · Carnegie Mellon University +1
EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use
Feb 17, 2026Siwei Wen, Zhangcheng Wang, Xingjian Zhang +2Streaming Video Understanding Multimodal Agents
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University, Beijing, China · 4Paradigm Inc., Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
ProAct: Harnessing Streaming Motion Generation and Agentic Reasoning for Real-Time Embodied Social Interaction
Feb 15, 2026Zeyi Zhang, Zixi Kang, Ruijie Zhao +4Human Motion Generation Human-Robot Interaction
Peking University, China

Mar 12, 2026J/K move · Enter open · S save

XSkill: Continual Learning from Experience and Skills in Multimodal Agents

Guanyu Jiang, Zhaochen Su, Xiaoye Qu, Yi R. Fung

Hong Kong University of Science and Technology · Zhejiang University · Huazhong University of Science and Technology

Agent Skill Learning Multimodal Agents Memory-Augmented Agents

Open paper AskPDF arXiv

Multimodal agents can now tackle complex reasoning tasks with diverse tools, yet they still suffer from inefficient tool use and inflexible orchestration in open-ended settings. A central challenge is enabling such agents to continually improve without parameter updates by learning from past trajectories. We identify two complementary forms of reusable knowledge essential for this goal: experiences, providing concise action-level guidance for tool selection and decision making, and skills, providing structured task-level guidance for planning and tool use. To this end, we propose XSkill, a dual-stream framework for continual learning from experience and skills in multimodal agents. XSkill grounds both knowledge extraction and retrieval in visual observations. During accumulation, XSkill distills and consolidates experiences and skills from multi-path rollouts via visually grounded summarization and cross-rollout critique. During inference, it retrieves and adapts this knowledge to the current visual context and feeds usage history back into accumulation to form a continual learning loop. Evaluated on five benchmarks across diverse domains with four backbone models, XSkill consistently and substantially outperforms both tool-only and learning-based baselines. Further analysis reveals that the two knowledge streams play complementary roles in influencing the reasoning behaviors of agents and show superior zero-shot generalization.