Tool Use in VLMs

Tool-using VLMs combine image or video understanding with operations such as cropping and zooming, OCR, code execution, geometric solvers, retrieval, and specialist image analysis. Agents learn to plan tool calls, inspect the results, and revise answers; common training approaches include supervised tool-use trajectories and reinforcement learning that rewards useful calls, alongside training-free tool harnesses. Evaluation spans fine-grained and high-resolution visual question answering, long-video understanding, spatial reasoning, charts and documents, and medical imaging. A central challenge is ensuring calls are necessary and their evidence actually grounds the answer, while controlling cost, latency, safety risks, and performance on unfamiliar tasks.