Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought

Kesen Zhao; Beier Zhu; Junbao Zhou; Xingyu Zhu; Zhongqi Yue; Hanwang Zhang

cs.CVFeb 27, 2026

Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought

Authors: Kesen Zhao, Beier Zhu, Junbao Zhou, Xingyu Zhu, Zhongqi Yue, Hanwang Zhang

Organizations: Nanyang Technological University · Chalmers University of Technology and University of Gothenburg · University of Science and Technology of China

Abstract

Recent multimodal large language models (MLLMs) increasingly rely on visual chain-of-thought to perform region-grounded reasoning over images. However, existing approaches ground regions via either textified coordinates-causing modality mismatch and semantic fragmentation or fixed-granularity patches that both limit precise region selection and often require non-trivial architectural changes. In this paper, we propose Numerical Visual Chain-of-Thought (NV-CoT), a framework that enables MLLMs to reason over images using continuous numerical coordinates. NV-CoT expands the MLLM action space from discrete vocabulary tokens to a continuous Euclidean space, allowing models to directly generate bounding-box coordinates as actions with only minimal architectural modification. The framework supports both supervised fine-tuning and reinforcement learning. In particular, we replace categorical token policies with a Gaussian (or Laplace) policy over coordinates and introduce stochasticity via reparameterized sampling, making NV-CoT fully compatible with GRPO-style policy optimization. Extensive experiments on three benchmarks against eight representative visual reasoning baselines demonstrate that NV-CoT significantly improves localization precision and final answer accuracy, while also accelerating training convergence, validating the effectiveness of continuous-action visual reasoning in MLLMs. The code is available in https://github.com/kesenzhao/NV-CoT.

Figures & tables

Figure 1: Our NV-CoT outperforms text-based visual CoT models (Vis-CoT ( Shao et al., 2024a ) and DeepEyes ( Zheng et al., 2025 ) ) in localization precision , answer accuracy , and convergence speed across both SFT and RL. SFT-based models are evaluated on the Vis-CoT-363K dataset ( Shao et al., 2024a ) , where ground-truth bounding boxes are available, while RL-based models are evaluated on the DeepEyes-47K dataset ( Zheng et al., 2025 ) . We only replace the text-space discrete coordinate objective with our Euclidean-space continuous one, while keeping all other training configurations unchanged for a fair comparison.

Figure 2: Comparison of different paradigms for thinking with images. (a) Text-based approaches represent localized regions as discrete coordinate tokens, leading to modality mismatch and fragmented semantics. (b) Patch-based approaches reason directly over fine-grained visual tokens but are constrained by the fixed spatial granularity of the vision backbone. (c) Our NV-CoT predicts region coordinates in continuous space, enabling flexible and precise localization.

Model	$V^{*}$ Bench			HR Bench 4K			HR Bench 8K
Model	Attr	Spatial	Overall	FSP	FCP	Overall	FSP	FCP	Overall
LLaVA-OneVision-7B	75.7	75.0	75.4	71.7	53.3	62.5	69.1	52.1	60.6
Qwen2.5-VL-7B	73.9	67.1	71.2	84.7	52.9	68.8	78.7	51.8	65.3
Qwen2.5-VL-32B	87.8	88.1	87.9	89.4	58.3	73.9	83.6	56.9	70.3
PaDT-7B	75.7	73.6	74.9	86.0	54.8	70.4	79.0	53.8	66.4
LVR-7B	84.4	76.3	81.2	88.3	56.8	72.5	82.5	54.5	68.5

Table 1: Overall comparison of different models on three benchmarks. The best results are in bold. Our NV-CoT consistently improves performance under both SFT and RL settings. We use NV-CoT (SFT) with an

\ell_{1}

loss and NV-CoT (RL) with a Laplace policy. DeepEyes-7B † and NV-CoT † denote results under multi-step tool use (multiple zoom-in calls). For a fair comparison, all other visual CoT methods are evaluated with a single zoom-in call, since some baselines do not support multiple tool invocations.

Model		$V^{*}$ Bench			HR Bench 4K			HR Bench 8K
Model		Attr	Spatial	Overall	FSP	FCP	Overall	FSP	FCP	Overall
SFT	$\ell_{2}^{2}$ loss	83.5	76.3	80.6	81.5	53.5	67.5	76.0	54.3	65.1
SFT	$\ell_{1}$ loss	84.3	78.9	82.2	82.0	55.3	68.6	77.3	54.5	65.9
RL	Gaussian policy	88.7	85.5	87.4	90.3	58.3	74.3	85.0	58.0	71.5
	+ Independent $\bm{\sigma}$	87.8	86.8	87.4	90.0	58.3	74.1	85.5	57.8	71.6
	Laplace policy	90.4	86.8	89.0	91.0	58.5	74.8	85.8	58.3	72.0

Table 2: Ablation on coordinate policy (Gaussian (

\ell_{2}^{2}

) vs. Laplace (

\ell_{1}

)) and uncertainty parameterization (shared

\sigma/\alpha

vs. per-coordinate

\bm{\sigma}/\bm{\alpha}

). Non-shared

\bm{\sigma}/\bm{\alpha}

: per-coordinate scale (standard deviation) predicted by four heads. Laplace policy consistently performs better than Gaussian policy, while using a shared versus per-coordinate

\bm{\sigma}/\bm{\alpha}

yields comparable results.

	Vis-CoT	NV-CoT ( $\ell_{2}^{2}$ )	NV-CoT ( $\ell_{1}$ )
Bbox IoU	47.3	57.4	59.5

Table 3: Bounding-box IoU on the Vis-CoT-363K dataset. NV-CoT substantially improves localization precision over Vis-CoT, with the

\ell_{1}

loss outperforming the

\ell_{2}^{2}

loss.

Figure 3: Behavior of

\alpha

. Successful trajectories exhibit smaller

\alpha

than failed ones, reflecting higher confidence.

Figure 4: Effect of

\lambda

. Performance peaks at

\lambda=0.3

and NV-CoT consistently outperforms the baseline across all values.

Figure 5: Visualization of bounding boxes. NV-CoT produces more accurate bounding boxes (shown in red ) compared to the backbone model (shown in blue ), demonstrating improved localization capability.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: More visualization results of bounding boxes. NV-CoT produces more accurate bounding boxes (shown in red ) compared to the backbone model (shown in blue ), demonstrating improved localization capability.

Explore similar work

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
Jun 7, 2026Lianyu Hu, Xiaoyu Ma, Zeqin Liao +1Multimodal Large Language Models Multimodal Reasoning
College of Computing and Data Science of Nanyang Technological University, Singapore.
Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Jun 1, 2026Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2Fine-Tuning Multimodal Large Language Models
University of Virginia
Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations
Jun 15, 2026Zhiqiang Zhou, Junliang Dai, Xu lingVLM Interpretability Multimodal CoT Reasoning
Hunan Chemical Industry Vocational and Technical College, Zhuzhou, Hunan 412000, China

Jun 7, 2026J/K move · Enter open · S save

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Lianyu Hu, Xiaoyu Ma, Zeqin Liao, Yang Liu

College of Computing and Data Science of Nanyang Technological University, Singapore.

Multimodal Large Language Models Multimodal Reasoning VLM Reasoning Multimodal CoT Reasoning Visually Grounded Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a `vision-blind reasoning' paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens <THINK>, <LOOK> and <ANSWER>. These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.

Abstract

Explore similar work

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations