Multimodal Agents

Ask about this topic

Overview

Multimodal agents combine language models with visual, audio, and video perception to answer questions, use tools, and interact with apps, websites, robots, or other environments. Common approaches include planning and tool orchestration, specialized collaborating agents, active evidence gathering, and memory or reusable skills for long tasks; reinforcement learning and execution feedback are increasingly used to improve these behaviors. Benchmarks now test browsing, GUI use, video reasoning, and embodied tasks, often evaluating action traces and process reliability as well as final answers. Despite progress, agents remain brittle over long horizons, and more modalities or more agents do not automatically improve grounding, safety, efficiency, or task success.

Latest papers 367

49 papers in the last four weeks, up 20% on the four weeks before. 0.3% of all new papers.

All topics
CardsList

  1. An Edge-Host-Cloud Architecture for Robot-Agnostic, Caregiver-in-the-Loop Personalized Cognitive Exercise: Multi-Site Deployment in Dementia Care

    Apr 1, 2026Wenzheng Zhao, Ruth Palan Lopez, Shu Fen Wung +1Assistive RoboticsHealthcare

    Worcester Polytechnic Institute

  2. Collaborative AI Agents and Critics for Fault Detection and Cause Analysis in Network Telemetry

    Mar 31, 2026Syed Eqbal Alam, Zhan ShuMulti-Agent CollaborationMultimodal Agents

    Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Alberta, Canada · SheQAI Research, Edmonton, Alberta, Canada

  3. AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction

    Mar 31, 2026Harsh Mankodiya, Chase Gallik, Theodoros Galanos +1AI Agent BenchmarksMultimodal Agents

  4. MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

    Mar 30, 2026Fangda Ye, Yuxin Hu, Pengxiang Zhu +19AI Agent BenchmarksMultimodal Agents

  5. HeteroHub: An Applicable Data Management Framework for Heterogeneous Multi-Embodied Agent System

    Mar 30, 2026Xujia Li, Xin Li, Junquan Huang +3Multi-Agent SystemsEmbodied AI

    HKUST Hong Kong SAR, China · HKUST (GZ) GuangZhou, China · HKUST (GZ) & HKUST GuangZhou, China

  6. MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences

    Mar 29, 2026Shijian Wang, Jiarui Jin, Runhao Fu +11Multimodal AgentsMultimodal Reasoning

    Southeast University · Monash University · Xiaohongshu Inc. +4

  7. Communicating about Space: Language-Mediated Spatial Integration Across Partial Views

    Mar 28, 2026Ankur Sikarwar, Debangan Mishra, Sudarshan Nikhil +2Spatial ReasoningMultimodal Agents

    Mila – Quebec AI Institute · Université de Montréal · IIIT Hyderabad +1

  8. FusionAgent: A Multimodal Agent with Dynamic Model Selection for Human Recognition

    Mar 27, 2026Jie Zhu, Xiao Guo, Yiyang Su +2Multimodal FusionBiometric Identification

    Michigan State University · University of North Carolina at Chapel Hill

  9. CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare

    Mar 25, 2026Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh +4Computer-Use AgentsLong-Horizon Agent Tasks

    Indian Institute of Technology Patna · Mohamed bin Zayed University of AI (MBZUAI)

  10. SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

    Mar 24, 2026Haoyu Huang, Jinfa Huang, Zhongwei Wan +3Efficient Multimodal InferenceLLM Inference Acceleration

    Xiamen University, Xiamen, Fujian, China · University of Rochester, Rochester, NY, USA · The Ohio State University, Columbus, OH, USA

  11. A Multimodal Framework for Human-Multi-Agent Interaction

    Mar 24, 2026Shaid Hasan, Breenice Lee, Sujan Sarker +1Human-Robot InteractionHuman-AI Interaction

    University of Virginia

  12. GEM: Guided Expectation-Maximization for Behavior-Normalized Candidate Action Selection in Offline RL

    Mar 24, 2026Haoyu Wang, Jingcheng Wang, Shunyu Wu +1Offline RLExpectation-Maximization

    School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China

  13. Task-Aware Positioning for Improvisational Tasks in Mobile Construction Robots via an AI Agent with Multi-LMM Modules

    Mar 24, 2026Seongju Jang, Francis Baek, SangHyun LeeRobotic ConstructionRobot Navigation

    Ph.D. Student, Department of Civil and Environmental Engineering, University of Michigan, 2350 Hayward St, Ann Arbor, MI 48109 · Assistant Professor, School of Civil and Environmental Engineering, Georgia Institute of Technology, North Avenue, Atlanta GA 30332 · Robert B. Harris Collegiate Professor, Department of Civil and Environmental Engineering, University of Michigan, 2350 Hayward St, Ann Arbor, MI 48109

  14. PhotoAgent: A Robotic Photographer with Spatial and Aesthetic Understanding

    Mar 24, 2026Lirong Che, Zhenfeng Gan, Yanbo Chen +2Multimodal AgentsVision-Based Robot Control

    Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China

  15. StreamingClaw Technical Report

    Mar 23, 2026Jiawei Chen, Zhe Chen, Chaoqun Du +21Multimodal AgentsAgentic AI

    Li Auto Inc.

  16. Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment

    Mar 23, 2026Sheng Liu, Long Chen, Zeyun Zhao +16Clinical Risk PredictionClinical Decision-Making

    Department of Biomedical Data Science, Stanford University · Department of Radiation Oncology, Stanford University · Center for Data Science, New York University +13

  17. Seed1.8 Model Card: Towards Generalized Real-World Agency

    Mar 21, 2026Bytedance SeedLLM AgentsTool-Using Vision-Language Agents

    Bytedance Seed

  18. PersonaVLM: Long-Term Personalized Multimodal LLMs

    Mar 20, 2026Chang Nie, Chaoyou Fu, Yifan Zhang +2Personalized Language ModelsMultimodal Agents

    Nanjing University · ByteDance

  19. ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agents

    Mar 20, 2026Zijian Lu, Yiping Zuo, Yupeng Nie +4LLM Agent Skill LearningWeb Agents

    Nanjing University of Posts and Telecommunications, Nanjing, China · China University of Petroleum (East China), Qingdao, China · Southeast University, Nanjing, China

  20. Teaching an Agent to Sketch One Part at a Time

    Mar 19, 2026Xiaodan Du, Ruize Xu, David Yunis +2Generative ModelingLLM Agent Training

    TTI-Chicago · University of Chicago · MIT CSAIL

  21. ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models

    Mar 19, 2026Thomas De Min, Subhankar Roy, Stéphane Lathuilière +2Multimodal Large Language ModelsAI Agent Benchmarks

    University of Trento · University of Bergamo · Inria, Univ. Grenoble Alpes, CNRS, LJK +1

  22. FailureMem: A Failure-Aware Multimodal Framework for Autonomous Software Repair

    Mar 18, 2026Ruize Ma, Yilei Jiang, Shilin Zhang +7Automated Program RepairMultimodal Agents

    Nanjing University · SenseTime · The Chinese University of Hong Kong +1

  23. Anticipatory Planning for Multimodal AI Agents

    Mar 17, 2026Yongyuan Liang, Shijie Zhou, Yu Gu +6Multimodal AgentsComputer-Use Agents

    University of Maryland, College Park · State University of New York at Buffalo · The Ohio State University +1

  24. MSRAMIE: Multimodal Structured Reasoning Agent for Multi-instruction Image Editing

    Mar 17, 2026Zhaoyuan Qiu, Ken Chen, Xiangwei Wang +3Image EditingMultimodal Agents

    University Of Melbourne, Grattan Street, Parkville, Victoria 3010, Australia

  25. Intelligent Co-Design: An Interactive LLM Framework for Interior Spatial Design via Multi-Modal Agents

    Mar 16, 2026Ren Jian Lim, Rushi Dai3D Scene Layout GenerationHuman-AI Co-Creation

    Hong Kong Center for Construction Robotics, The Hong Kong University of Science and Technology

  26. Advancing Multimodal Agent Reasoning with Long-Term Neuro-Symbolic Memory

    Mar 16, 2026Rongjie Jiang, Jianwei Wang, Gengda Zhao +3Multimodal MemoryMultimodal Agents

    University of New South Wales Sydney, Australia · Zhejiang University Hangzhou, China · Shanghai Jiao Tong University Shanghai, China

  27. Mind-of-Director: Multi-modal Agent-Driven Film Previsualization via Collaborative Decision-Making

    Mar 16, 2026Shufeng Nan, Mengtian Li, Sixiao Zheng +3Multimodal AgentsHuman-AI Co-Creation

    Fudan University, Shanghai, China · Shanghai University, Shanghai, China

  28. A Skill-augmented Agentic Framework and Benchmark for Multi-Video Understanding

    Mar 16, 2026Yue Zhang, Liqiang Jing, Jia Li +4Video UnderstandingMultimodal QA

    University of Texas at Dallas, USA

  29. InterventionLens: A Multi-Agent Framework for Detecting ASD Intervention Strategies in Parent-Child Shared Reading

    Mar 14, 2026Xiao Wang, Lu Dong, Ifeoma Nwogu +2Autism Spectrum DisorderTemporal Action Segmentation

    State University of New York at Buffalo, Buffalo, NY 14260, USA

  30. A Discordance-Aware Multimodal Framework with Multi-Agent Clinical Reasoning

    Mar 12, 2026Pegah Ahadian, Mingrui Yang, Sixu Chen +2Clinical Outcome PredictionMedical Diagnosis

    Department of Computer Science, Kent State University, Ohio, USA · Department of Biomedical Engineering, Cleveland Clinic, Ohio, USA