Multimodal Tool Use

Multimodal tool-use research asks how AI agents can inspect visual and audio evidence, search for information, and act through tools such as browsers, code, and graphical interfaces. Common approaches include tool routing and planning, iterative perception–action loops, evidence tracking and verification, and supervised or reinforcement learning on tool-use trajectories. Benchmarks such as AgentVista, VTC-Bench, and OSWorld test agents on realistic tasks that require multiple steps and tool combinations. A central open tension is that tools can improve access to evidence but also add cost and failure points, while current agents still struggle with perception, planning, and knowing when a tool will help.