Multimodal Agents

Follow this topic Ask about this topic

Multimodal agents combine language models with visual, audio, and video perception to answer questions, use tools, and interact with apps, websites, robots, or other environments. Common approaches include planning and tool orchestration, specialized collaborating agents, active evidence gathering, and memory or reusable skills for long tasks; reinforcement learning and execution feedback are increasingly used to improve these behaviors. Benchmarks now test browsing, GUI use, video reasoning, and embodied tasks, often evaluating action traces and process reliability as well as final answers. Despite progress, agents remain brittle over long horizons, and more modalities or more agents do not automatically improve grounding, safety, efficiency, or task success.

Latest papers 367

49 papers in the last four weeks, up 20% on the four weeks before. 0.3% of all new papers.

All topics

DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning
Jun 29, 2026Xinxin Chen, Yuchen Li, Zihan Wang +3Efficient Multimodal Inference Multi-Agent Collaboration
University of Chinese Academy of Sciences
Hybrid Retriever Evolution for Multimodal Document Reasoning Agents
Jun 28, 2026Bohan Yao, Shruthan Radhakrishna, Vikas YadavMultimodal RAG Multimodal QA
1ServiceNow · University of Washington
Agent-Computer Observation Interfaces Enable Dynamic Computer Use
Jun 28, 2026Bojie Li, Noah ShiAudio-Visual Understanding Computer-Use Agents
Pine AI · University of Washington
GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodes
Jun 26, 2026Amit Parekh, Sabrina McCallum, Kareem Al-Hasan +3Game-Playing Agents Multi-Agent Collaboration
Heriot-Watt University · University of Edinburgh
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
Jun 26, 2026ZhengXian Wu, Hangrui Xu, Kai Shi +8Multimodal RAG Multimodal Search Agents
OPPO AI Center, OPPO Inc. China · The Shenzhen International Graduate School, Tsinghua University · Nanyang Technological University, Singapore
NormAct: A Benchmark for Hidden Social Norm Compliance in Embodied Planning
Jun 26, 2026Shiyun Zhao, Xinwei Song, Tianyu Guo +7AI Agent Benchmarks Multimodal Agents
1State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI) · 3ShanghaiTech University · 2China Academy of Information and Communications Technology
DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection
Jun 25, 2026Yujin Tang, Chenming Shang, Ruize Xu +1Multimodal Memory Long-Horizon Agent Evaluation
Dartmouth College
Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
Jun 25, 2026Tianyi Men, Zhuoran Jin, Pengfei Cao +3GUI Agents LLM Agent Training
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy
Jun 25, 2026Junhao Shi, Zezheng Huai, Siyin Wang +7LLM Agent Orchestration Robotics
Fudan University · Shanghai Innovation Institute · Jilin University
iFLYTEK-Embodied-Omni Technical Report
Jun 24, 2026Yuan Zhang, Jingfei Ni, Guanchen Lu +12Unified Multimodal Models Robot Motion Generation
1iFLYTEK · University of Science and Technology of China · 2LindenBot
Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Jun 23, 2026Lianghua Huang, Zhi-Fan Wu, Wei Wang +22Audio-Video Generation Unified Multimodal Models
Wan Team, Alibaba Group
ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection
Jun 23, 2026Chenhao Dang, Dantong Zhu, Jun Yang +2Multimodal Agents Multimodal Model Evaluation
Shanghai Jiaotong University · Shanghai Artificial Intelligence Laboratory · Central South University +2
Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse
Jun 22, 2026Bole Ma, Jan Eitzinger, Harald Koestler +1Efficient Multimodal Inference KV Caching
When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents
Jun 22, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangAI Agent Evaluation Indirect Prompt Injection
Northeastern University Boston, Massachusetts, USA · University of Illinois Urbana-Champaign Urbana, Illinois, USA · Southern Methodist University Dallas, Texas, USA
Holmes: Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale
Jun 20, 2026Jia Li, Wenyuan Ma, Ting Peng +2Multi-Agent LLM Systems Fault Localization
Tencent Inc. Shenzhen, China
RAPID: A Reproducible Multi-Agent Pipeline for Interpretable Disaster Damage Assessment from Satellite and Street-View Imagery
Jun 20, 2026Yifan Yang, Wenjing Gong, Kaili Zhang +5Remote Sensing Image Understanding Disaster Damage Assessment
Texas A&M University College Station, Texas, USA · National University of Singapore Singapore, Singapore · The University of Alabama Tuscaloosa, Alabama, USA
Dementia-Agents: A Multi-Modal Multi-Agent System for Dementia Staging and Phenotyping
Jun 19, 2026Yaling Shen, Maja Christensen, Yiwen Jiang +4Medical Diagnosis Multimodal Agents
Monash University · Eastern Health · Lived Experience Advisor
Native Active Perception as Reasoning for Omni-Modal Understanding
Jun 17, 2026Zhenghao Xing, Ruiyang Xu, Yuxuan Wang +8Audio-Visual Understanding Active Perception
1The Chinese University of Hong Kong · †Work done during an internship at Qwen Team, Alibaba Group. · 2Shanghai Jiao Tong University +2
Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games
Jun 17, 2026Shengyuan Ding, Xilin Wei, Xinyu Fang +4Multimodal Memory Sequential Decision Making
1Fudan University · 2Shanghai Innovation Institute · 3Shanghai Artificial Intelligence Laboratory +2
OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation
Jun 16, 2026Zijie Meng, Yufei Liu, Chengqian Ma +8Multi-View Consistency Multi-View Video Generation
Peking University · Xiamen University · Korea Advanced Institute of Science and Technology (KAIST) +5
LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control
Jun 15, 2026Anqi Zou, Han Deng, Chengyu Zhang +9Computer-Use Agent Benchmarks Computer-Use Agents
Shenzhen Loop Area Institute · Dalian University of Technology · The Chinese University of Hong Kong +1
Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
Jun 13, 2026Zhengbo Zhang, Changtao Miao, Jinbo Su +10Multimodal IR Multimodal Grounding
School of Artificial Intelligence, UCAS · Institute of Automation, CAS · Ant Digital Technologies, Ant Group +2
Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation
Jun 13, 2026Shijun Wan, Xuehai Wu, Jiwen Zhang +2LLM Agent Evaluation AI Agent Benchmarks
Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute
A Multi-Agent AI System for Automated High School Transcript Processing: Collaborative Document Analysis at Scale
Jun 11, 2026Ben Torkian, Jun ZhouDocument Understanding Multi-Agent Coordination
University of South Carolina, USA · University of South Carolina, Columbia, South Carolina, USA · University of South Carolina, Columbia, USA
IterCAD: An Iterative Multimodal Agent for Visually-Grounded CAD Generation and Editing
Jun 11, 2026Tao Hu, Jiaxin Ai, Licheng Wen +12Parametric CAD Modeling Text-to-CAD Generation
Shanghai Artificial Intelligence Laboratory · University of Science and Technology of China · Shanghai Innovation Institute +4
Y-BotFrame: An Extensible Embodied Agent Framework for Quadruped Robot Assistants
Jun 11, 2026Luyao Zhang, Ke Li, Yuan Ding +9Assistive Robotics Language-Conditioned Robot Control
Xidian University Xi’an, China
Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Models
Jun 11, 2026Quan QuanIndustrial Anomaly Detection Multimodal Reasoning
InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning
Jun 10, 2026Ziang Yan, Sheng Xia, Jiashuo Yu +10Efficient Multimodal Inference Multimodal Reasoning
Shanghai AI Laboratory · Shanghai Innovation Institute · Nanjing University
MODF-SIR: A Multi-agent Omni-modal Distilled Framework for Social Intelligence Reasoning
Jun 10, 2026Shang Ma, Jisheng Dang, Wencan Zhang +6Multi-Agent Reasoning Social Reasoning
School of information Science and Engineering, Lanzhou University, LanZhou, China · School of Computing, National University of Singapore, Singapore 119077 · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China +1
ParseFixer: An Agentic Framework for Document Parsing via Selective Multimodal Correction
Jun 10, 2026LeKai Yu, Hao Liu, Kun Wang +4Document Parsing Document Layout Analysis
Shandong University · Southeast University

Jun 29, 2026J/K move · Enter open · S save

DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning

Xinxin Chen, Yuchen Li, Zihan Wang, Haoyu Zhang+2

University of Chinese Academy of Sciences

Efficient Multimodal Inference Multi-Agent Collaboration Multimodal Reasoning Multimodal Agents Multimodal Fusion

Open paper AskPDF arXiv

Current multimodal fusion approaches, particularly those based on static Mixture-of-Experts (MoE) architectures, often struggle to provide the adaptive and efficient collaborative reasoning required by complex real-world applications. We introduce the Dynamic Agent-based Interaction Network (DAIN), which reconceptualizes multimodal fusion as a dynamic, multi-agent collaborative process. DAIN employs a context-aware Meta-Controller that dynamically schedules sparse activation of specialized interaction agents and orchestrates compressed inter-agent communication for consensus-building. The framework is guided by a multi-objective loss function that jointly optimizes task accuracy, agent specialization, and operational efficiency through sparse activation and communication regularization. Comprehensive evaluations across five diverse benchmarks -- ADNI, MIMIC-IV, MM-IMDB, CMU-MOSI, and ENRICO -- establish DAIN as a new state-of-the-art, delivering significant performance improvements including a 2.6% accuracy gain on ADNI. Ablation studies verify the critical roles of both dynamic scheduling and agent communication. Furthermore, DAIN offers enhanced interpretability by exposing context-dependent agent roles and collaboration patterns while maintaining computational efficiency through sample-wise sparse agent activation. Our work demonstrates the promise of dynamic, agent-based paradigms for multimodal reasoning.