Tool Use in VLMs

VLM: Vision-Language Model

Overview

Tool-using VLMs combine image or video understanding with operations such as cropping and zooming, OCR, code execution, geometric solvers, retrieval, and specialist image analysis. Agents learn to plan tool calls, inspect the results, and revise answers; common training approaches include supervised tool-use trajectories and reinforcement learning that rewards useful calls, alongside training-free tool harnesses. Evaluation spans fine-grained and high-resolution visual question answering, long-video understanding, spatial reasoning, charts and documents, and medical imaging. A central challenge is ensuring calls are necessary and their evidence actually grounds the answer, while controlling cost, latency, safety risks, and performance on unfamiliar tasks.

Latest papers

CardsList

123 papers9 papers in the last four weeks, down 44% on the four weeks before. 0.1% of all new papers.

  1. Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning

    Dec 16, 2025Yankai Jiang, Yujie Zhang, Peng Zhang +5Image SegmentationTool Use in VLMs

  2. REVEAL: Robust Evolution of Vision-Language Models for Explainable AI-Video Detection

    Feb 20, 2025Yun-Yun Tsai, Qingyuan Liu, Ruijian Zha +4Tool Use in VLMsVision-Language Models

  3. Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner

    Dec 30, 2024Yitong Zhou, Mingyue Cheng, Qingyang Mao +3Table Structure RecognitionVLM Evaluation