cs.CVNov 18, 2026

Keep the Image in Mind: A Visual Board for Multimodal Agents

Authors: Haiming Zhu, Zhiwei Chen, Ding Cao, Tianxiang Zhao

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Multi-step visual reasoning suffers when early visual misinterpretations persist as incorrect premises in subsequent reasoning. We argue that how visual evidence is represented during reasoning matters as much as how much of it is gathered. Rather than appending observations and inferences to an ever-growing dialogue, we introduce a Visual Board-centered harness that separates visual observations, uncertain interpretations, and derived reasoning within a structured, capacity-bounded, and revisable workspace. This separation makes each premise explicit and auditable, so that conflicts between what is seen and what is derived become detectable even by a text-only critic and can trigger targeted reinspection of the original image. Without task-specific training, our harness consistently outperforms standalone models across 4B, 8B, Qwen3-VL-Flash, and 32B configurations on Euclid30K, SolidGeo, and ChartQA bar charts, with gains of up to 11.68, 18.42, and 24.82 percentage points, respectively. Under identical tools, critic, and interaction limits, the structured Board outperforms shared free-form notes by 6.67 to 13.00 points and no-Board retries by 7.35 to 18.80 points while using 7.9% to 38.2% fewer tokens. Supervised fine-tuning on harness trajectories further improves direct answering and critic-free tool use, with positive transfer to chart reasoning.

Figures & tables

Explore similar work

CardsList
  1. Hierarchical Visual Agent: Managing Contexts in Joint Image-Text Space for Advanced Chart Reasoning

    May 5, 2026Qihua Dong, Ruozhen He, Junwen Chen +4Data VisualizationChart QA

  2. See2Think: Do Multimodal Models Really Use Intermediate Visual States?

    Jul 29, 2026Siyu Yan, Zhuoran Yan, Haiying Xu +10Visual ReasoningMultimodal Reasoning

  3. OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

    Oct 19, 2026Yu Zhou, Cheng-Fu Yang, Rui Sun +8Tool Use in VLMsMultimodal Reasoning