Learning Where to Look: Observation Policy Optimization for Thinking with Images

Junfeng Wang; Jiawei Liu; Yongchao Xu; Tao Jiang; Jiangbo Ai; Jin Zhang; Chong Wang; Qixing Zhang

cs.CVNov 3, 2026

Learning Where to Look: Observation Policy Optimization for Thinking with Images

Authors: Junfeng Wang, Jiawei Liu, Yongchao Xu, Tao Jiang, Jiangbo Ai, Jin Zhang, Chong Wang, Qixing Zhang

Organizations: University of Science and Technology of China · Alibaba Group

Abstract

Complex visual question answering often requires vision-language models to think with images, i.e., actively acquire local visual evidence by zooming into informative regions of high-resolution images. Our error analysis shows that many failures in this setting originate from incorrect observation actions, where the model fails to localize key regions or obtain crops that contain sufficient evidence for answering. We further observe that bounding-box tokens exhibit substantially higher generation entropy than ordinary language tokens, indicating high uncertainty when the model decides where to look. However, existing reinforcement learning methods based on final-answer rewards provide only trajectory-level feedback, making it difficult to assign precise credit to the observation actions. We propose Observation Policy Optimization (OPO), an RL algorithm for optimizing observation actions in tool-augmented thinking with images. OPO consists of two core modules. First, Uncertainty-Guided Observation Branching (UOB) treats each image zoom-in tool call as an observation event and selectively branches at high-uncertainty, low-redundancy events during rollout, enabling efficient exploration of alternative cropping decisions and their resulting visual evidence. Second, Evidence-Localized Advantage Attribution (ELAA) compares sibling branches using local evidence-sufficiency rewards and assigns credit only to the corresponding observation-action tokens. By decoupling observation-action optimization from trajectory-level outcome supervision, OPO improves visual evidence acquisition and further enhances visual reasoning performance. Experiments show that OPO consistently improves Qwen3-VL models across scales and achieves competitive performance against larger models and tool-augmented baselines. Code is publicly available.

Figures & tables

Figure 1: Motivation for observation-action optimization. (a) Error-type breakdown of incorrect V* trajectories for Qwen3-VL-8B and DeepEyes-trained Qwen3-VL-8B, together with an example observation-error trajectory. (b) Token-entropy analysis, where the top panel shows an event-aligned entropy trace over a generated response and the bottom panel compares entropy distributions of reasoning tokens, bounding-box tokens, and answer tokens.

Figure 2: Overview of OPO. UOB concentrates rollout exploration around high-uncertainty and low-redundancy observation decisions by branching from the shared pre-tool prefix. ELAA then compares sibling crops within each branch group using evidence-sufficiency rewards and assigns the resulting local advantage only to the observation-action tokens.

Model

Backbone

Tool

Param.

V* Bench

HR-Bench 4K

HR-Bench 8K

Attr.

Spatial

Overall

FSP

FCP

Overall

FSP

FCP

Overall

Advanced proprietary models

GPT-5 mini [ 44 ]

–

78.6

–

78.6

–

74.4

GPT-5.2 [ 45 ]

–

75.9

–

Claude Opus 4.6 [ 46 ]

–

66.5

–

Gemini-3 Pro [ 47 ]

–

88.0

–

Table 1: Main results on high-resolution perception benchmarks. LLaVA-OV abbreviates LLaVA-OneVision. Tool denotes the inference-time mechanism: Zoom for image zoom-in crop tools, WF for manually designed visual-search workflows, Code for code-execution tools, General for code-execution, image search and text search tools, C+S for image cropping and search,

\times

for no tool. Best and second-best results are bolded and underlined. † denotes results reproduced by us, and – indicates that the corresponding entry is not reported or not applicable.

Model	V* Bench			HR-Bench 4K			HR-Bench 8K
	Attr.	Spatial	Overall	FSP	FCP	Overall	FSP	FCP	Overall
Qwen2.5-VL-7B [ 1 ]	53.0	55.3	53.9	68.5	55.3	61.9	60.8	48.0	54.4
+ DeepEyes-style GRPO [ 7 ]	87.0	81.6	84.8	88.0	55.0	71.5	76.3	59.8	68.0
+ OPO	93.9	80.3	88.5	87.8	60.3	74.0	78.8	57.5	68.1
$\Delta$ (vs GRPO)	+6.9	-1.3	+3.7	-0.2	+5.3	+2.5	+2.5	-2.3	+0.1
Qwen3-VL-2B [ 2 ]	82.6	65.8	75.9	79.3	52.8	66.0	68.0	53.3	60.6

Table 2: Controlled comparison between DeepEyes-style outcome-level GRPO and OPO under the same backbone and zoom-tool training setup. All results are reproduced by us.

Variant	V*	HR-Bench 4K	HR-Bench 8K
Qwen3-VL-2B	75.9	66.0	60.6
OPO full	85.9	78.4	76.4
w/o UOB & ELAA	82.7	71.5	71.1
w/o ELAA	83.8	76.5	73.8
w/o uncert. sel.	83.3	76.9	73.0
w/o redund. filt.	84.8	76.5	75.1

Table 3: Ablation study on Qwen3-VL-2B. We report overall accuracy. The default setting is

M=8,B=2

, where

M

is the rollout budget and

B

is the number of base trajectories. The default full OPO setting is highlighted in bold.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative comparison on a V* direct-attribute example.

Figure 6: Qualitative comparisons on V* and HR-Bench. The top example is a V* spatial-relation question, while the bottom example is from HR-Bench.

Model	Accuracy	Unsuccessful	Localization failures
Qwen3-VL-32B-Instruct	169/191 (88.5%)	22	17/22 (77.3%)
Kimi K3	172/191 (90.1%)	19	15/19 (78.9%)
Qwen3.7-Max	182/191 (95.3%)	9	9/9 (100.0%)
Overall	523/573 (91.3%)	50	41/50 (82.0%)

Appendix

Table 4: Localization failures in stronger models on V*. Percentages in the last column are computed over unsuccessful trajectories.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 260 papers

Jun 7, 2026

Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillation

''Thinking with Images'' has emerged as an effective paradigm for fine-grained visual reasoning: by explicitly zooming into relevant regions and reasoning over crops, models can access local evidence that is difficult to recover from a single global image. However, this benefit comes with redundant tool invocations and longer inference traces. Moreover, when such behaviors are learned mainly from outcome reward, the resulting intermediate crops or visual cues can be noisy or fail to faithfully capture task-relevant visual evidence. In this work, we ask whether the reasoning benefits of ''Thinking with Images'' can be internalized through Thinking with Imagination: an internal process that decides where to look and imagines what visual cues closer inspection would reveal without actually invoking tools. We propose Imagine-OPD, an on-policy self-distillation framework in which a teacher plays the role of a ''Thinking with Images'' reasoner during training: it receives privileged zoomed evidence views derived from annotated regions, and supervises the model's own imagination reasoning trajectories. Imagine-OPD does not require an external teacher or high-quality imagination demonstrations. Experiments on vision-centric benchmarks show that Imagine-OPD achieves the best average performance among compared models while significantly reducing inference overhead compared with ''Thinking with Images'' methods.

Yishuo Cai, Jiahui Liu, Yuanxin Liu +9

1State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 2Central South University · University of Science and Technology of China +2

Tool Use in VLMs Efficient VLM Inference On-Policy Self-Distillation Latent Visual Reasoning VLM Reasoning Visually Grounded Reasoning

Ask PDF

Feb 26, 2026

ConFoThinking: Consolidated Focused Attention Driven Thinking for Visual Question Answering

Thinking with Images improves fine-grained VQA for MLLMs by emphasizing visual cues. However, tool-augmented methods depend on the capacity of grounding, which remains unreliable for MLLMs. In parallel, attention-driven methods to crop the Region of Interest (ROIs) are proposed but they are constrained by (1) fragmented attention signals scattered across layers, leading to suboptimal localization and (2) relying on question- or redundant-text-conditioned attention extraction. Our analysis reveals three patterns: MLLMs may attend to the correct region yet generate incorrect coordinates, where-to-look attention is often fragmented across layers, and attention extraction is query-sensitive. Motivated by these, We propose ConFoThinking, a Consolidated-Focused-Attention-Driven Thinking framework that learns to aggregate attention into a designated intermediate layer, from which we mine and zoom in salient regions for downstream visual understanding. Moreover, we extract attention using concise semantic cues of what to look into, which mitigates the semantic noise introduced by question- or redundant-text-based attention extraction. Experiments across five VQA benchmarks demonstrate ConFoThinking significantly improves perception performance. The code, checkpoints, and dataset will be released after being accepted.

Zhaodong Wu, Haochen Xue, Qi Cao +5

Wangxuan Institute of Computer Technology, Peking University, Beijing, China · School of AI and Advanced Computing, Xi’an Jiaotong-Liverpool University, Suzhou, China

Visual Question Answering VLM Reasoning

Ask PDF

Jun 14, 2026

Self-Questioning Vision-Language Models: Reinforcement Learning for Compositional Visual Reasoning

Vision-Language Models (VLMs) are AI systems that process both images and text, yet they often struggle with compositional visual reasoning questions that require chaining multiple steps together, such as identifying objects, counting them, and comparing the results. Existing approaches improve this reasoning by training models on human-written step-by-step explanations, but creating these annotations is expensive and difficult to scale. We propose a self-questioning framework that trains a VLM to break visual questions into smaller sub-questions and answer each one before producing a final response, using a reinforcement learning algorithm called Group Relative Policy Optimization (GRPO). The model is never shown examples of how to decompose questions, it discovers this behavior on its own, guided by a reward signal that scores whether the output contains sub-questions and whether the final answer is correct. We apply this framework to a 3-billion-parameter model, training on both synthetic scenes of geometric shapes (CLEVR) and real-world photographs (A-OKVQA). On A-OKVQA, both self-questioning and standard reinforcement learning substantially improve accuracy over the untrained model (52.2% and 51.6% vs. 46.8%). We introduce the first self-questioning VLM by rewarding not only the final answer like standard RL but additionally for generating intermediate sub-questions, enabling it to discover compositional decomposition strategies. These results suggest that teaching AI systems to ask themselves intermediate questions is a promising strategy for complex visual reasoning, particularly when the difficulty of a question warrants explicit step-by-step decomposition.

Saraswathy Amjith

Massachusetts Institute of Technology

Visual Question Answering Visual Reasoning Question Decomposition Question Answering RL for VLMs VLM Reasoning+2

Ask PDF

Abstract

Visual Question Answering

Explore similar work

Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillation

ConFoThinking: Consolidated Focused Attention Driven Thinking for Visual Question Answering

Self-Questioning Vision-Language Models: Reinforcement Learning for Compositional Visual Reasoning