Tool Use in VLMs
VLM: Vision-Language Model
Overview
Tool-using VLMs combine image or video understanding with operations such as cropping and zooming, OCR, code execution, geometric solvers, retrieval, and specialist image analysis. Agents learn to plan tool calls, inspect the results, and revise answers; common training approaches include supervised tool-use trajectories and reinforcement learning that rewards useful calls, alongside training-free tool harnesses. Evaluation spans fine-grained and high-resolution visual question answering, long-video understanding, spatial reasoning, charts and documents, and medical imaging. A central challenge is ensuring calls are necessary and their evidence actually grounds the answer, while controlling cost, latency, safety risks, and performance on unfamiliar tasks.
Latest papers
117 papers13 papers in the last four weeks, down 24% on the four weeks before. 0.1% of all new papers.