Tool-Using Vision-Language Agents

Overview

These agents use vision-language models to interpret images, video, documents, or live scenes, then call tools such as search, image viewers, code, specialist perception models, and robot actions. Common approaches structure work as plan–act–observe–reflect loops, with learned tool routing, retrieval and memory, and training through supervised trajectories or reinforcement learning. They are evaluated on visual question answering, charts and documents, long-video search, web navigation, and embodied tasks, where evidence traces and successful actions matter alongside answer accuracy. A central challenge is making tool use both useful and reliable: agents still struggle with visual grounding, choosing and recovering from calls, long-horizon coordination, and balancing accuracy against cost, latency, and safety.

Latest papers

CardsList

150 papers10 papers in the last four weeks, down 58% on the four weeks before. 0.1% of all new papers.

  1. InteractVLA: Turning VLAs into Interactive Tools for Agentic Robotic Manipulation

    Nov 17, 2026Chen Zhao, Zipeng Ye, Shifeng Bao +5Agentic AI for RoboticsTool-Using Vision-Language Agents

  2. Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding

    Nov 13, 2026Haolin Yang, Wenshan Wu, Jin Jiang +13Tool-Using Vision-Language AgentsMultimodal Agents

  3. DisasterAgent: An Evidence-Driven Agent for Verifiable Disaster Report Generation From Remote Sensing Imagery

    Nov 10, 2026Zhijian Li, Jiawei Song, Chao Ren +4Disaster Damage AssessmentRemote Sensing

  4. PSGAgent: Enhancing MLLMs' Dynamic Visual Perception for Panoptic Scene Graph Generation via Agentic Reasoning

    Nov 5, 2026Qingyun Liu, Yuankun Liu, Jiaye Feng +6Scene Graph GenerationTool-Using Vision-Language Agents

  5. Route First, See Later: Enabling Vision Agents to Reason Scientifically by Separating Routing Policy from Action Spaces

    Nov 4, 2026Shubhabrata Mukherjee, Jeffrey Donatelli, Daniela UshizimaTool-Using Vision-Language AgentsLLM Agent Routing

  6. GeoGuesser Bench: Benchmarking Vision-Language Agents on Interactive Geolocation

    Nov 2, 2026Huanzhi Mao, Yichuan Wang, Michael Koo +5Geospatial Reasoning BenchmarksVLN Benchmarks

  7. Coupled or Independent? Fault Coupling in Agentic Medical Vision Systems

    Oct 29, 2026Nishi Kanta Paul, Md Shihabul Islam Shovo, Md Mostafijur Rahman +1Tool-Using Vision-Language AgentsMedical VLMs

  8. EditAgent: Rethinking Agentic Image Editing

    Oct 29, 2026Haiyang Xu, Tianyu Wang, Jialu Li +5Image EditingTool-Using Vision-Language Agents

  9. S-Agent 2.0: Spatial Reasoning with a General Coding Agent

    Oct 20, 2026Rui Gong, Ruoyu Wang, Yalun Dai +9Spatial Reasoning in Language Models3D Spatial Reasoning

  10. MMedAgent-Plus: Scaling Multimodal Medical Agents across Modalities, Tasks and Workflows

    Oct 19, 2026Ruoxin Wang, Yuanting Pan, Binxu Li +13Medical AI AgentsTool-Using Vision-Language Agents

  11. LVMem: Image-Augmented Memory for Long-Horizon Question Answering

    Oct 19, 2026Victoria Khoruzhaya, Maksim Kalmykov, Tuong Do Minh +3Embodied QALong-Term Memory for Embodied Agents

  12. Inverse Garment Design with Visual Agents

    Oct 11, 2026Nanxiang Jiang, Yunuo Chen, Zeng Tao +53D Garment ReconstructionInverse Design

  13. Self-Evolving Visual Agent via Margin Reward

    Oct 10, 2026Wenxi Yang, Yuzhong Zhao, Baochang Zhang +2Tool-Using Vision-Language AgentsRL for Tool Use

  14. EviScene: Beyond "Is It AI?" for Agentic Image Factuality Assessment

    Oct 10, 2026ZiAn Wang, Ruiqi Liu, Zhengbo Zhang +10Automated Fact-CheckingTool-Using Vision-Language Agents

  15. CaptchaAgent: Verifiable Visual Reasoning for CAPTCHA Solving

    Oct 9, 2026Shangquan Sun, Ruoyu Chen, Zhirui Liu +4CAPTCHA SolvingTool-Using Vision-Language Agents

  16. SuperNav: An Agentic Navigation System for Any Task in Any Scene

    Oct 8, 2026Jinkai Zhang, Jingyi Xu, Yuanhong Yu +5Robot NavigationVisual Robot Navigation

  17. Visual Orchestration Tax in Agentic VLM Pipelines: Auditing and Certifying Visual Evidence Reuse

    Oct 6, 2026Lingteng ZengVLM EvaluationAI Agent Auditing

  18. IRSTD-Agent: Agentic Infrared Small Target Detection via Zoom-Guided Interaction Learning

    Oct 4, 2026Jiawen Xi, Yu Zhang, Tianyi Zhao +3Tool Use in VLMsInfrared Small Target Detection

  19. EditHero: A Benchmark for Long-Horizon Part-Level 3D Editing and Vibe Modeling

    Oct 1, 2026Ruihan Yu, Yu-Ju Tsai, Muyao Niu +93D Asset Editing3D Shape Editing

  20. Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens

    Oct 1, 2026Ruiyang Si, Jianxin Bi, Shunyu Yang +12Vision-Language-Action ModelsTool-Augmented Language Model Agents

  21. EviRover: Reinforcing Agentic Perception Beyond a Glance

    Sep 30, 2026Kaixuan Fan, Kaituo Feng, Tianshuo Peng +4Tool-Using Vision-Language AgentsActive Visual Perception

  22. A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations): A Visual-Symbolic Framework for Virtual Humans

    Sep 28, 2026Alessandro Emmanuel Pecora, Stefano Calzolari, Francesco Strada +1Language Model-Based ControlWorld Models