Overview

Multimodal tool-use research asks how AI agents can inspect visual and audio evidence, search for information, and act through tools such as browsers, code, and graphical interfaces. Common approaches include tool routing and planning, iterative perception–action loops, evidence tracking and verification, and supervised or reinforcement learning on tool-use trajectories. Benchmarks such as AgentVista, VTC-Bench, and OSWorld test agents on realistic tasks that require multiple steps and tool combinations. A central open tension is that tools can improve access to evidence but also add cost and failure points, while current agents still struggle with perception, planning, and knowing when a tool will help.

Latest papers

CardsList

66 papers5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. Omni-Workspace: Benchmarking Omni-Modal AI Assistants on Multimodal Workspace Tasks

    Nov 14, 2026Yuhao Wang, Yucheng Zhao, Yiran Zhong +5Multimodal Agent EvaluationMultimodal Agents

  2. Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

    Oct 28, 2026Yi Wang, Tianzi Wang, Peter BellAudio-Visual QAMultimodal Agents

  3. Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams

    Oct 26, 2026Yinsong Yan, Ling Wang, Yuxuan Wang +11Multimodal Agent EvaluationMLLM Evaluation

  4. MMedAgent-Plus: Scaling Multimodal Medical Agents across Modalities, Tasks and Workflows

    Oct 19, 2026Ruoxin Wang, Yuanting Pan, Binxu Li +13Medical AI AgentsTool-Using Vision-Language Agents

  5. TRACER: A Multimodal Harness for Patient Care in Hospitals

    Oct 15, 2026Zane Durante, Paul Tang, Keshigeyan Chandrasegaran +11Video UnderstandingTemporal Video Understanding

  6. VIEWCRAFT: Evolving What Multimodal Agents See and How They Read It

    Oct 11, 2026Binxiao Xu, Ruichuan An, Bocheng Zou +7Multimodal AgentsLLM Agent Skill Learning

  7. OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

    Oct 1, 2026Haibo Wang, Jiteng Mu, Jialu Li +5Audio-Visual UnderstandingRL for Language Model Reasoning

  8. WeaveData: A Multimodal Data Analysis System with Self-Critiquing and Self-Evolving LLM Plans

    Sep 28, 2026Min Jia, Shihao Zhou, Jun-Peng Zhu +9LLM Self-RefinementLLM Planning

  9. MM-VeriAgent: Learning to Use Extensive Tools to Verify Multimodal Misinformation with Reinforcement Learning

    Sep 25, 2026Peipei Li, Shuhan Xia, Shengyang Liu +2Multimodal Deception DetectionMultimodal Tool Use

  10. MedTRACE: Tool-Augmented Multimodal Clinical Reasoning Agents for Evidence-Grounded Decision-Making

    Sep 13, 2026Ji Lu, Lifei Liu, Haoran Yu +5Medical DiagnosisClinical Decision-Making

  11. AgenticCADedit: A Stateful, Tool-Mediated Agentic Approach to Multimodal 3D CAD Editing

    Aug 28, 2026Saptarshi Neil Sinha, Mika Silvan Goschke, Paul Julius Kühn +23D Shape EditingMultimodal Tool Use

  12. WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

    Aug 28, 2026Zongkai Liu, Hui Zhang, Liqiang Niu +7Multimodal IRMultimodal Search Agents

  13. MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents

    Aug 25, 2026Yu Liu, Wenxiao Zhang, Cheng Hu +5Multimodal AgentsMemory-Augmented Language Model Agents

  14. VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

    Aug 15, 2026Yansong Ning, Jingwen Ye, Zhongkai Wu +5Multimodal AgentsAI Agent Benchmarks

  15. A Cloud-Edge System for Multimodal Clinical Screening in Resource-Constrained Rural Settings

    Aug 13, 2026Hei Ting Chan, Chenwei Wu, Xueshen Liu +7Efficient Multimodal InferenceHealthcare

  16. Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents

    Aug 4, 2026Siqi Fan, Minghao Li, Xiaoqian Ma +6Computer-Use AgentsRL for LLM Agents

  17. See2Think: Do Multimodal Models Really Use Intermediate Visual States?

    Jul 29, 2026Siyu Yan, Zhuoran Yan, Haiying Xu +10Visual ReasoningMultimodal Reasoning

  18. ReDesign: Recovering Editable Design Structures from Images via Agentic Decomposition

    Jul 28, 2026Jooyeol Yun, Jintae Park, Hyesu Lim +3Agentic WorkflowsMultimodal Tool Use

  19. ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning

    Jul 17, 2026Binglin Zhou, Peng Shi, Ryo Kamoi +2Tool Use in VLMsScientific Claim Verification

  20. Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents

    Jul 13, 2026Ming Ma, Yi Zhu, Yiran Zhong +6Multimodal AgentsMultimodal Tool Use

  21. UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

    Jul 12, 2026Xiyu Wei, Qingwei Zong, Zhuocheng Yu +1Multimodal QAAI Agent Benchmarks

  22. Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards

    Jul 6, 2026Tianhao Niu, Ziyu Han, Qiguang Chen +5Data VisualizationCode Generation Evaluation

  23. CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration

    Jul 6, 2026Hairui Zhu, Yiying Yang, Tengjin Weng +5Tool Use in VLMsText-Guided Image Editing

  24. Data Journalist Agent: Transforming Data into Verifiable Multimodal Stories

    Jun 9, 2026Kevin Qinghong Lin, Batu EI, Yuhong Shi +5LLM Agent OrchestrationMultimodal Agents

  25. TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents

    Jun 4, 2026Chengqi Dong, Chuhuai Yue, Hang He +6Multimodal IRToken-Level Credit Assignment