Multimodal Tool Use

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 41

All topics

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
Oct 1, 2026Haibo Wang, Jiteng Mu, Jialu Li +5Audio-Visual Understanding RL for Language Model Reasoning
University of California, Davis · Adobe Research
WeaveData: A Multimodal Data Analysis System with Self-Critiquing and Self-Evolving LLM Plans
Sep 28, 2026Min Jia, Shihao Zhou, Jun-Peng Zhu +9LLM Self-Refinement LLM Planning
Northwest A&F University · PingCAP · East China Normal University +1
MedTRACE: Tool-Augmented Multimodal Clinical Reasoning Agents for Evidence-Grounded Decision-Making
Sep 13, 2026Ji Lu, Lifei Liu, Haoran Yu +5Medical Diagnosis Clinical Decision-Making
Vanderbilt University, United States · Wichita State University, United States · University of Florida, United States +4
WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents
Aug 28, 2026Zongkai Liu, Hui Zhang, Liqiang Niu +7Multimodal IR Multimodal Search Agents
Weixin AI, Tencent · Sun Yat-sen University
A Cloud-Edge System for Multimodal Clinical Screening in Resource-Constrained Rural Settings
Aug 13, 2026Hei Ting, Chan, Chenwei Wu +8Efficient Multimodal Inference Healthcare
Una · University of Michigan, Ann Arbor, MI, USA · Massachusetts Institute of Technology, Cambridge, MA, USA +1
Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents
Aug 4, 2026Siqi Fan, Minghao Li, Xiaoqian Ma +6Computer-Use Agents RL for LLM Agents
University of Electronic Science and Technology of China · AI Platform, Xiaohongshu Inc. · Gaoling School of Artificial Intelligence, Renmin University of China +1
VC-Tooler: Learning Compositional and Adaptive Visual Tool Use
Aug 3, 2026Yizheng Wu, Jiashen Hua, Bing Deng +1Tool Use in VLMs Tool-Using Vision-Language Agents
Alibaba Group
See2Think: Do Multimodal Models Really Use Intermediate Visual States?
Jul 29, 2026Siyu Yan, Zhuoran Yan, Haiying Xu +10Visual Reasoning Multimodal Reasoning
1The Hong Kong University of Science and Technology · 3Shanghai AI Laboratory · 2Central South University +4
ReDesign: Recovering Editable Design Structures from Images via Agentic Decomposition
Jul 28, 2026Jooyeol Yun, Jintae Park, Hyesu Lim +3Agentic Workflows Multimodal Tool Use
KAIST AI, Daejeon, Korea · Helmholtz Munich, Neuherberg, Germany · Korea University, EverEx
Silent Failures in Multimodal Agentic Search:A Diagnostic Taxonomy and Cross-Judge Evaluation
Jul 22, 2026Zhengxian Wu, Junjie Gao, Kai YangMultimodal Grounding Agentic Search
Ant Group · Hangzhou, China
ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning
Jul 17, 2026Binglin Zhou, Peng Shi, Ryo Kamoi +2Tool Use in VLMs Scientific Claim Verification
1The Pennsylvania State University · University of Waterloo
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
Jul 13, 2026Kaixin Ma, Di Feng, Alexander Metz +3Tool-Use Evaluation Tool-Using Vision-Language Agents
Apple
Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents
Jul 13, 2026Ming Ma, Yi Zhu, Yiran Zhong +6Multimodal Agents Multimodal Tool Use
Institute of Neuroscience, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tongyi Lab, Alibaba Group +2
Beyond the Eye: Efficient Multimodal Reasoning via Self-Regulated Implicit Visual Tools
Jul 13, 2026Xiuwei Chen, Quanlin Chen, Wentao Hu +8Efficient Multimodal Inference Tool Use in VLMs
Sun Yat-sen University, China
UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
Jul 12, 2026Xiyu Wei, Qingwei Zong, Zhuocheng Yu +1Multimodal QA AI Agent Benchmarks
Key Laboratory of Computational Linguistics, MOE, Peking University · School of Software and Microelectronics, Peking University · School of Computer Science, Peking University
Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards
Jul 6, 2026Tianhao Niu, Ziyu Han, Qiguang Chen +5Data Visualization Code Generation Evaluation
Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China
CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration
Jul 6, 2026Hairui Zhu, Yiying Yang, Tengjin Weng +5Tool Use in VLMs Text-Guided Image Editing
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) Shenzhen, China
Data Journalist Agent: Transforming Data into Verifiable Multimodal Stories
Jun 9, 2026Kevin Qinghong Lin, Batu EI, Yuhong Shi +5LLM Agent Orchestration Multimodal Agents
University of Oxford · Stanford University · BBC R&D
TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents
Jun 4, 2026Chengqi Dong, Chuhuai Yue, Hang He +6Multimodal IR Token-Level Credit Assignment
University of Science and Technology of China · 2Meituan
VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark
Jun 2, 2026Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari +5Mathematical Reasoning Benchmarks Mathematical Reasoning
University of British Columbia · University of Illinois Chicago · Stony Brook University
Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains
Jun 1, 2026Garvin Guo, Donglei Yu, Yu Chen +6Tool-Use Evaluation LLM Agent Evaluation
University of Chinese Academy of Sciences · Tongyi Lab, Alibaba Group · Amap, Alibaba Group +1
MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand
Jun 1, 2026Shouang Wei, Houcheng Min, Xinpeng Dong +8LLM Agent Skill Learning Multimodal Agents
East China Normal University · Zhejiang University · Tsinghua University +2
Crayotter: Traceable Multi-Agent Workflows for Long-Form Video Editing
May 31, 2026Lecheng Yan, Yichong Zhang, Ben Pan +5Video Editing Multi-Agent Collaboration
Crayotter Project
Sandboxed Coding Agents are Competitive Omni-modal Task Solvers
May 30, 2026Dongping Chen, Xuanao Huang, Zhihan Hu +3Audio-Visual Understanding AI Coding Agents
University of Maryland · MBZUAI
Syll: Open-Source Personal Automation with Cross-Surface Execution
May 28, 2026Bo Zhang, Borui Zhang, Chenghao Jiang +5LLM Agent Skill Learning Human-in-the-Loop AI
Tsinghua University · GigaAI
Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation
May 28, 2026Chenghao Zhang, Guanting Dong, Yufan Liu +3Multimodal Generation Multimodal Agents
Gaoling School of Artificial Intelligence, Renmin University of China
EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
May 27, 2026Yunqi Liu, Tong Niu, Zitong Wang +4AI Agent Evaluation Egocentric Video Understanding
Ant Group
AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture
May 21, 2026Zi Ye, Yibin Wen, Xiaoya Fan +10AI Agent Benchmarks Tool-Using Vision-Language Agents
1Sun Yat-Sen University · 2Southwest University · 3Northeastern University +4
Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning
May 19, 2026Qinghe Ma, Zhen Zhao, Yiming Wu +3Multimodal Large Language Models Multimodal Reasoning
The State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Shanghai Artificial Intelligence Laboratory +2
Hallucination as Exploit: Evidence-Carrying Multimodal Agents
May 18, 2026Guijia Zhang, Hao Zheng, Harry YangAI Agent Security Multimodal Hallucination
1Shenzhen University · 2HKUST

Oct 1, 2026, cs.CVJ/K move · Enter open · S save

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

Haibo Wang, Jiteng Mu, Jialu Li, Jingru Yi+4

University of California, Davis · Adobe Research

Audio-Visual Understanding RL for Language Model Reasoning Multimodal Large Language Models Multimodal CoT Reasoning Multimodal Tool Use

Open paper AskPDF arXiv

We present OmniSeek, an agentic framework that transforms an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use. Rather than passively processing an entire audio-visual sequence in a single forward pass, OmniSeek makes evidence acquisition part of the reasoning process: it dynamically decides whether to look or listen, and over which temporal window, to retrieve sparse but critical evidence across different modalities within long contexts. Through an iterative multi-turn protocol, the retrieved raw audio or visual segments are appended back into the context to support subsequent reasoning. To cold-start this capability, we build a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with interleaved audio and visual evidence. We first supervise the model on these trajectories to instill multi-turn tool-use behavior, and then further optimize the policy via a two-stage reinforcement learning with verifiable rewards. Moreover, we introduce an Audio-Visual Necessity objective that explicitly rewards successful trajectories whose reasoning depends on both modalities, discouraging single-modality shortcuts. Extensive experiments across a wide range of benchmarks demonstrate that OmniSeek learns adaptive cross-modal evidence seeking and consistently improves audio-visual reasoning performance.

Momentum

Related topics

Latest papers 41