Tool Use in VLMs

VLM: Vision-Language Model

Overview

Tool-using VLMs combine image or video understanding with operations such as cropping and zooming, OCR, code execution, geometric solvers, retrieval, and specialist image analysis. Agents learn to plan tool calls, inspect the results, and revise answers; common training approaches include supervised tool-use trajectories and reinforcement learning that rewards useful calls, alongside training-free tool harnesses. Evaluation spans fine-grained and high-resolution visual question answering, long-video understanding, spatial reasoning, charts and documents, and medical imaging. A central challenge is ensuring calls are necessary and their evidence actually grounds the answer, while controlling cost, latency, safety risks, and performance on unfamiliar tasks.

Latest papers

CardsList

123 papers9 papers in the last four weeks, down 44% on the four weeks before. 0.1% of all new papers.

  1. Visual Harness Scales General VLMs to Medical Images

    Nov 17, 2026Shaowen Wan, Hanqi Jiang, Zihao Wu +6Medical VLMsTool Use in VLMs

  2. LogT: Logically Think with Images for Visual Search

    Nov 14, 2026Yanjun Fu, Quanzeng You, Jiadong Guo +6Tool Use in VLMsReasoning for VLMs

  3. Beyond Answering: Learning to Speak First and Act in Streaming Video

    Nov 12, 2026Shuhan Li, Huankang Guan, Xintong Guo +15Streaming Video UnderstandingProactive Video Understanding

  4. SpaceMCP: Vision Language Model Spatial Reasoning via Model Context Protocol

    Nov 11, 2026Ameen Izhac, Nobuko Yoshida, Ronald ClarkSpatial Reasoning in Language ModelsMultimodal Spatial Reasoning

  5. ArabGeoText: Tool-Augmented Vision-Language Models for Arabic Geospatial Text Spotting

    Nov 2, 2026Yakoub Bazi, Sarah Alsaleh, Mohamad M. Al Rahhal +2Scene Text RecognitionTool Use in VLMs

  6. Parse Before You Execute: Efficient Spatial Reasoning with Executable Primitives

    Oct 30, 2026Xiwei Liu, Yulong Li, Xinling Zhuang +5Tool Use in VLMsMultimodal Spatial Reasoning

  7. PerceptRL: Making Visual Perception Rewardable in MLLMs from Answers to Interactions

    Oct 27, 2026Tongkun Guan, Zhibo Yang, Mingkun Yang +3Vision-Language ModelsRL for VLMs

  8. LocalAgent: Collaborative Agentic Verification for Fine-Grained Instance-Level Consistency

    Oct 21, 2026Jiachen Guo, Xinshan Zhu, Boyun Wang +4Image MatchingTool Use in VLMs

  9. VisualEvolve: A Skill Self-Evolution Framework for Visual Reasoning

    Oct 20, 2026Jiawei Wang, Yucun Li, Xueqi Li +9Tool Use in VLMsReasoning for VLMs

  10. OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

    Oct 19, 2026Yu Zhou, Cheng-Fu Yang, Rui Sun +8Tool Use in VLMsMultimodal Reasoning

  11. CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness

    Oct 15, 2026Dongjie Cheng, Ruifeng Yuan, Ran Wei +3VLM ReasoningVideo Reasoning

  12. Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?

    Oct 8, 2026Minhao Fan, Yinyi Liu, Jiayu Zhao +3Tool Use in VLMsVision-Language Models

  13. IRSTD-Agent: Agentic Infrared Small Target Detection via Zoom-Guided Interaction Learning

    Oct 4, 2026Jiawen Xi, Yu Zhang, Tianyi Zhao +3Tool Use in VLMsInfrared Small Target Detection

  14. MLLMs Fail to Refuse when Using Tools Agentically

    Oct 2, 2026Rikiya Takehi, Ryo Hachiuma, Shaona Ghosh +3MLLM SafetyLLM Agent Safety

  15. CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment

    Oct 1, 2026Kunyang Li, Hai Nguyen, Joshua Lowe +6Tool Use in VLMsMedical Visual Question Answering

  16. CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding

    Sep 30, 2026Yiduo Jia, Muzhi Zhu, Jinchuan Shi +4Temporal Video GroundingVideo Understanding

  17. EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

    Sep 30, 2026Shulin Tian, Junsu Kim, Shuai Liu +17Tool-Use EvaluationTool Use in VLMs

  18. Agentic Tool-Augmented Reasoning for Explainable Image Forgery Detection

    Sep 30, 2026Zhiya Tan, Jing Huang, Changtao Miao +5Tool Use in VLMsImage Forgery Detection

  19. Visual Parallel Search: Learning to Search High-Resolution Images with Parallel Tile Inspection and Adaptive Zoom

    Sep 29, 2026Xijia Tao, Yihua Teng, Xinyu Fu +5Visual Question AnsweringTool Use in VLMs

  20. InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning

    Sep 26, 2026Hanqian Li, Sirui Huang, Chen Ling +12Table QAMultimodal CoT Reasoning

  21. CityToolVQA: Tool-Augmented Visual Question Answering for 3D Spatial Cognition in Urban Low-Altitude Environments

    Sep 26, 2026Boao Yu, Yingzhen Nie, Yue Hu +23D Visual Question AnsweringTool Use in VLMs

  22. WeaveAgent: A Two-Stage Tool-Routing Agent for Ultra-High-Resolution Remote Sensing Imagery

    Sep 25, 2026Zhongyu PangRemote Sensing Image UnderstandingTool Use in VLMs

  23. VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoning

    Sep 21, 2026Hexiong Yang, Mingrui Chen, Jie Cao +1VLM ReasoningVision-Language Models

  24. When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Used

    Sep 19, 2026Kunyu Peng, Junming Liu, Ruiqi He +3Tool Use in VLMsRL for VLMs

  25. MintAct: A Unified Visual Agent for Digital Environments

    Sep 18, 2026Mingfei Gao, Rui Tian, Haiming Gang +11GUI AutomationTool Use in VLMs

  26. AgenTeeth: A Model-Agnostic Framework for Suppressing Hallucination in Frozen Vision-Language Models on Dental X-Rays via Tool Evidence Injection

    Sep 15, 2026Ahmed Rafid, Fariya Ahmed, Rumman Adib +3VLM Hallucination MitigationRadiology VLMs

  27. Reasoning with Image Generation

    Sep 14, 2026Nishad Singhi, Hector Garcia Rodriguez, Aditya Arora +2Tool Use in VLMsVLM Reasoning