Thinking with Images without Coordinates

Meng Cao; Haoze Zhao; Yi Wang; Ian Reid; Xiaodan Liang

cs.CVOct 9, 2026

Thinking with Images without Coordinates

Authors: Meng Cao, Haoze Zhao, Yi Wang, Ian Reid, Xiaodan Liang

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Shanghai AI Lab · Sun Yat-sen University

Abstract

Recent advances in multi-modal large language models have shifted toward the "thinking with images" paradigm to improve reasoning faithfulness, yet existing methods rely heavily on explicit coordinate- based grounding. We identify two fundamental limitations in this approach, i.e., a semantic gap where sparse regressions miss high-dimensional latent structures, and a resolution gap due to the granularity mismatch between patch-level tokens and pixel-level coordinates. To address these, we propose TiCO (Thinking with images without COordinates). It reformulates visual reasoning as a semantic evidence discovery process within the latent space, and evidence is injected as token-level regions without pixel coordinates. Specifically, TiCO utilizes a dedicated probe token to estimate similarity distributions and performs importance sampling over spatially connected components to generate diverse, semantically dominant evidence rollouts. Extensive experiments across multiple high-resolution and general visual reasoning benchmarks demonstrate that our TiCO consistently outperforms state-of-the-art coordinate-based methods with significantly improved inference efficiency.

Figures & tables

Figure 2: Conceptual comparison and performance trade-offs. (a) Coordinate-based methods [ 19 , 56 ] localize evidence via explicit bounding-box prediction, reducing reasoning to geometric regression in image space; (b) Coordinate-free methods instead operate directly in visual token space, inducing a similarity distribution over patches and extracting semantically coherent evidence without explicit coordinates; (c) Performance–efficiency comparison on HR-Bench 8K [ 41 ] . Our TiCO achieves superior accuracy with substantially lower inference latency and fewer tokens compared to coordinate-based approaches such as Mini-o3 [ 19 ] and DeepEyes [ 56 ] .

Figure 3: Overview of TiCO. Given an image–text query pair, the policy model first produces a semantic probe token ( <GROUND> ) whose embedding interacts with visual patch tokens to generate a similarity map. After thresholding the similarity distribution, spatially connected components are extracted as candidate evidence regions. These components are treated as evidence rollouts and are subsequently fed back into the policy model to generate answer rollouts.

Model	Param Size	$V^{*}$ Bench [ 44 ]			HR-Bench 4K [ 41 ]			HR-Bench 8K [ 41 ]
Model	Param Size	Attr	Spatial	Overall	FSP	FCP	Overall	FSP	FCP	Overall
GPT-4o [ 1 ]	-	-	-	66.0	70.0	48.0	59.0	62.0	49.0	55.5
Gemini Pro [ 31 ]	-	40.9	59.2	48.2	-	-	-	-	-	-
GPT-4V [ 26 ]	-	51.3	60.5	55.0	-	-	-	-	-	-
Gemini 1.5 Flash [ 31 ]	-	-	-	-	76.8	56.8	66.8	69.2	56.7	62.8
SEAL [ 44 ]	7B	74.8	76.3	75.4	-	-	-	-	-	-

Table 1: Performance comparisons on high-resolution reasoning benchmarks. ∗ denotes the results sourced from DeepEyes [ 56 ] and † denotes the results sourced from Mini-o3 [ 19 ] . The bold values correspond to the best-performing results in each column, while the underlined values correspond to the second-best.

Model	VisualProbe ∗			MME -RW ∗	POPE †
Model	hard	medium	easy	MME -RW ∗	Adv	Pop	Random	Overall
GPT-4o [ 1 ]	11.2	15.4	47.5	45.2	-	-	-	-
LLaVA-OneVision [ 20 ]	13.4	12.5	36.2	57.4	-	-	-	88.4
Qwen2.5-VL [ 3 ]	23.9	26.0	39.1	57.3	85.9	86.5	87.2	85.9
Pixel Reasoner [ 39 ]	28.8	29.6	58.4	64.4	-	-	-	-
DeepEyes [ 56 ]	35.1	29.8	60.1	64.0	84.0	87.5	91.8	87.7

Table 2: Performance comparisons on VisualProbe and general reasoning benchmarks. For VisualProbe ∗ [ 19 ] and MME-RealWorld ∗ (MME-RW) [ 54 ] , the results of the compared methods are sourced from Mini-o3 [ 19 ] . For POPE † [ 23 ] , the results of the compared methods are sourced from DeepEyes [ 56 ] . The bold values correspond to the best-performing results in each column, while the underlined values correspond to the second-best.

Model	HR-Bench 4K [ 41 ]			HR-Bench 8K [ 41 ]
Model	Overall $\uparrow$	#Token $\downarrow$	Time $\downarrow$	Overall $\uparrow$	#Token $\downarrow$	Time $\downarrow$
Mini-o3 [ 19 ]	77.5	527.7	20.2	73.3	714.5	35.1
DeepEyes [ 56 ]	73.2	398.2	18.3	69.5	419.3	27.4
TiCO (Ours)	78.0	6.2	1.9	76.3	6.1	3.6

Table 3: Efficiency analysis. “Overall" denotes the overall accuracy. “#Token" denotes the average number of tokens contained in the generated answers. “Time" denotes the average inference time per query (in seconds).

Method	$V^{*}$ Bench [ 44 ]			HR-Bench 4K [ 41 ]			HR-Bench 8K [ 41 ]
Method	Attr	Spatial	Overall	FSP	FCP	Overall	FSP	FCP	Overall
Coord-based w/ thinking	85.2	80.3	83.2	90.5	62.0	76.3	88.0	53.8	73.6
Coord-based w/o thinking	85.2	82.9	84.3	89.8	62.5	76.1	86.5	58.0	72.3
Coord-free (Ours)	91.3	89.5	90.6	92.3	63.8	78.0	91.0	61.5	76.3
Probe via text embedding	88.7	82.9	86.4	91.0	63.5	77.3	88.0	58.3	73.1
Probe via <GROUND> token	91.3	89.5	90.6	92.3	63.8	78.0	91.0	61.5	76.3

Table 4: Ablation studies on the architectural design. Top: Comparison between our proposed coordinate-free approach and coordinate-based counterparts; Middle: Comparison of different probing mechanisms, specifically probing via text embedding versus probing via <GROUND> token. Bottom: Comparison of visual input configurations, including evidence-only, full-image-only, and their integrated setting.

Figure 4: Left: Robustness analysis on HR-Bench 4K [ 41 ] ; Right: Hyper-parameter analysis of threshold

\tau

on V ∗ benchmark [ 44 ] .

Method	$V^{*}$ Bench [ 44 ]			HR-Bench 4K [ 41 ]			HR-Bench 8K [ 41 ]
Method	Attr	Spatial	Overall	FSP	FCP	Overall	FSP	FCP	Overall
Top-scoring sampling	88.7	80.3	85.3	91.5	61.5	76.5	87.5	59.5	73.5
Random sampling	87.8	79.0	84.3	93.0	60.8	76.9	86.3	60.0	73.1
Token-wise sampling	81.7	80.3	81.2	87.5	59.0	73.3	76.3	53.8	65.0
Importance sampling	91.3	89.5	90.6	92.3	63.8	78.0	91.0	61.5	76.3

Table 5: Comparisons of sampling strategies including top-scoring (deterministic highest-score selection), random (uniform selection), token-wise (independent patch selection without grouping), and our importance sampling. The bold values denote the best-performing results, while the underlined values denote the second-best results.

Method	$V^{*}$ Bench [ 44 ]			HR-Bench 4K [ 41 ]			HR-Bench 8K [ 41 ]
Method	Attr	Spatial	Overall	FSP	FCP	Overall	FSP	FCP	Overall
SFT Only	87.0	81.6	84.8	90.8	61.5	76.1	86.5	55.8	71.1
RL Only	79.1	73.7	77.0	85.8	54.3	70.0	82.0	55.8	68.9
SFT ( w/o ground loss) + RL	77.4	72.4	75.4	77.8	59.3	68.5	66.3	58.8	62.5
SFT + RL (Ours)	91.3	89.5	90.6	92.3	63.8	78.0	91.0	61.5	76.3

Table 6: Ablations on training configurations including SFT and RL stages. "SFT ( w/o ground loss)" denotes removing the grounding loss (Equ ( 5 )) in SFT.

Figure 5: Training dynamics of TiCO during SFT and RL stages. Left: The convergence of the SFT phase including the grounding loss and the next-token prediction (NTP) loss; Right: The accuracy reward and format reward over training steps.

Method	$V^{*}$ Bench [ 44 ]			HR-Bench 4K [ 41 ]			HR-Bench 8K [ 41 ]
Method	Attr	Spatial	Overall	FSP	FCP	Overall	FSP	FCP	Overall
$N_{1}=1$ , $N_{2}=8$	87.8	80.3	84.8	90.8	62.3	76.5	88.5	58.5	73.5
$N_{1}=2$ , $N_{2}=4$	89.6	86.8	88.5	92.5	62.8	77.6	89.8	58.8	74.3
$N_{1}=4$ , $N_{2}=2$	91.3	86.8	89.5	92.3	63.8	78.0	91.0	61.5	76.3

Table 7: Hyper-parameter analysis including the number of evidence region

N_{1}

and answer rollout

N_{2}

. The bold values correspond to the best-performing results, while the underlined values correspond to the second-best.

Figure 6: Hyper-parameter analysis of threshold

\tau

Panel of Figure 7 — Figure 7: (a) Performance under controlled budget; (b) Illustartions of dense patch-wise label assignment in SFT training.

Figure 8: Exemplar SFT dataset in Mini-o3.

Figure 9: Evolution of attention maps from SFT to RL. (a) Input images with the annotated ground-truth bounding box; (b) Attention map after the SFT stage; (c) Attention map after the RL stage.

Panel of Figure 10 — Figure 10: Qualitative examples of failure cases in the "thinking" mode of coordinate-based methods. (a) Repeated Thinking: The model falls into an infinite loop during the multi-turn reasoning process, repeatedly generating the same crop coordinates and thoughts without progressing toward a final answer. (b) Inconsistent Logic: Although the model correctly identifies the target object (a yellow bicycle) in the final observation, it produces an incorrect final answer (Option B: red) that contradicts its own internal reasoning and visual evidence.

Question: {input}

Your task is to locate the region that corresponds to the question. To indicate the region, we will use some special tokens, which is used to refer to a visual patch later. For example, you can output:

<

your_special_token_here

>

Refer to the region extracted in the previous turn and provide the final answer.

Table 8: Prompts for TiCO. The placeholder {input} is dynamically instantiated with the task-specific query and the corresponding high-resolution image context during both the training and inference phases.

Figure 11: Failure cases of our TiCO. (Top) Spatial relationship error: TiCO incorrectly predicts the left–right relation between two persons, revealing limitations in spatial reasoning. (Bottom) Detailed attribution error: TiCO misidentifies the backpack color (predicting red instead of gray), indicating weaknesses in fine-grained attribute recognition.

Panel of Figure 12 — Figure 12: Comparison between coordinate-based methods and TiCO.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought
Feb 27, 2026Kesen Zhao, Beier Zhu, Junbao Zhou +3Multimodal CoT Reasoning Visual CoT
Nanyang Technological University · Chalmers University of Technology and University of Gothenburg · University of Science and Technology of China
Let's Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughts
Mar 23, 2026Xu Liu, Yongheng Zhang, Qiguang Chen +3Multimodal CoT Reasoning VLM Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen · School of Computer Science and Engineering, Central South University · Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center, Guizhou University +1
Optical Reasoning: Rethinking Images as an Expressive Reasoning Medium Beyond Text
Jun 8, 2026Yutong Bian, Dongjie Cheng, Heming Xia +2Visual Reasoning Multimodal Reasoning
The Hong Kong Polytechnic University

Feb 27, 2026J/K move · Enter open · S save

Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought

Kesen Zhao, Beier Zhu, Junbao Zhou, Xingyu Zhu+2

Nanyang Technological University · Chalmers University of Technology and University of Gothenburg · University of Science and Technology of China

Multimodal CoT Reasoning Visual CoT Visually Grounded Reasoning Vision-Language-Action Models

Open paper AskPDF arXiv

Recent MLLMs increasingly rely on visual chain-of-thought to perform region-grounded reasoning over images. However, existing approaches ground regions via either textified coordinates (causing modality mismatch and semantic fragmentation) or fixed-granularity patches that both limit precise region selection and often require non-trivial architectural changes. In this paper, we propose Numerical Visual Chain-of-Thought (NV-CoT), a framework that enables MLLMs to reason over images using continuous numerical coordinates. NV-CoT expands the MLLM action space from discrete vocabulary tokens to a continuous Euclidean space, allowing models to directly generate bounding-box coordinates as actions with only minimal architectural modification. The framework supports both supervised fine-tuning and reinforcement learning. In particular, we replace categorical token policies with a Gaussian (or Laplace) policy over coordinates and introduce stochasticity via reparameterized sampling, making NV-CoT fully compatible with GRPO-style policy optimization. Extensive experiments on three benchmarks against 11 representative visual reasoning baselines demonstrate that NV-CoT significantly improves localization precision and final answer accuracy, while also accelerating training convergence, validating the effectiveness of continuous-action visual reasoning in MLLMs. Project page: https://kesenzhao.github.io/NV-CoT/

Abstract

VLM Reasoning

Explore similar work

Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought

Let's Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughts

Optical Reasoning: Rethinking Images as an Expressive Reasoning Medium Beyond Text