cs.CVNov 19, 2026

Learning to Think with Images Faithfully via Adaptive Data Selection

Authors: Yu Zhao, Jiaxin Zhang, Rohit Saxena, Fazl Barez, Pasquale Minervini, Yiwen Guo

Organizations: University of Edinburgh · LIGHTSPEED · University of Oxford · Independent Researcher

Abstract

Thinking with images (TwI) empowers vision-language models (VLMs) to overcome perceptual bottlenecks by actively cropping and zooming into fine-grained visual details, promising both higher accuracy and interpretable, region-grounded reasoning. However, our analysis shows that the improved accuracy of TwI models does not guarantee faithful tool use; they frequently invoke tools without actually relying on the zoomed region, undermining the reliability of tool-assisted reasoning. We trace this unfaithful behaviour to the training data: when questions are answerable without zooming, the model can answer correctly while ignoring the returned evidence, and this problem is exacerbated during training as the model becomes better at answering without tools. To keep zooming useful throughout training, we adapt the training data to the model's evolving capability and train on questions where correct zoomed evidence can still help the model answer correctly. This selection is supported by a data synthesis pipeline that localises small objects and constructs verifiable questions about their fine details, and complemented by an adaptiveness reward that favours tool-free answers when zooming is unnecessary. The experimental results show that our method substantially improves reasoning faithfulness and achieves state-of-the-art performance on visual search benchmarks (+11.6% on V*, +7.9% on HR-Bench 4K, and +9.4% on HR-Bench 8K). On multimodal hallucination benchmarks, our model also outperforms the compared TwI baselines. These results suggest that improving TwI faithfulness does not sacrifice accuracy and that both can be learnt from the right training data without directly supervising the reasoning process.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LogT: Logically Think with Images for Visual Search

    Nov 14, 2026Yanjun Fu, Quanzeng You, Jiadong Guo +6Tool Use in VLMsReasoning for VLMs

  2. FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Self-Verification

    Jul 30, 2026Haoqing Wang, Xingrun Xing, Ziheng Li +2Tool Use in VLMsVLM Reasoning

  3. Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions

    Mar 31, 2026Xuesong Wang, Harry WangVLM RobustnessReasoning for VLMs