Beyond Static Visual Tokens: Structured Sequential Visual Chain-of-Thought Reasoning

Guangfu Guo; Xiaoqian Lu; Yue Feng; Mingming Sun

cs.CVMar 21, 2026

Beyond Static Visual Tokens: Structured Sequential Visual Chain-of-Thought Reasoning

Authors: Guangfu Guo, Xiaoqian Lu, Yue Feng, Mingming Sun

Organizations: Beijing Institute of Mathematical Sciences and Applications · University of Bristol · University of Birmingham

Abstract

Current multimodal LLMs encode images as static visual prefixes and rely on text-based reasoning, lacking goal-driven and adaptive visual access. Inspired by human visual perception-where attention is selectively and sequentially shifted from the most informative regions to secondary cues-we propose Structural Sequential Visual CoT SSV-CoT. First, a question-relevant saliency map identifies and organizes key visual regions, explicitly modeling the spatial distribution of visual importance. Second, reasoning is performed following this discriminative order, inducing a curriculum-like semantic progression from primary to secondary cues. This method is trained end-to-end, using text cot and answer supervision, without relying on region-level annotations or specialized external tools. Experiments on diverse visual reasoning benchmarks show gains, validating structured and sequential visual cognition.

Figures & tables

Figure 1: MLLM lacks visual analysis, while SSV integrates cues through structured sequential reasoning to reach the correct answer.

Figure 2: Overview of the SSV-CoT framework. The model first constructs question-aware structured visual regions, then performs sequential visual access during chain-of-thought reasoning to progressively integrate visual evidence and generate the final answer.

Methods	M3CoT	ScienceQA	LLaVA-W
	ACC.↑	ACC.↑	ROUGE-L↑
Qwen2-VL-7B	43.6	56.3	32.7
MultimodalCoT	40.1	51.3	30.7
CCoT	43.3	56.4	29.4
DDCoT	42.6	55.2	31.2
SCAFFOLD	41.7	53.7	31.8

Table 1: Commonsense visual reasoning results. Comparison of SSV-CoT applied to Qwen2-VL-7B with CoT-based methods (including No-CoT, MultimodalCoT, CCoT, DDCoT, SCAFFOLD, and ICoT) on M3CoT (Accuracy), ScienceQA (Accuracy), and LLaVA-W (ROUGE-L). The best result for each benchmark is highlighted in bold.

Model	MathVista	MathVision	Average
	Acc. $\uparrow$	Acc. $\uparrow$	Acc. $\uparrow$
MiniCPM-V-2.6	60.8	18.4	39.6
VITA-1.5	66.2	19.5	42.9
LLaVA-CoT	52.5	19.9	36.2
Qwen2-VL-7B	60.9	16.3	38.6
InternVL2.5	64.5	17.0	40.8

Table 2: Mathematical visual reasoning results. Comparison of SSV-CoT applied to Qwen2-VL-7B with vision-language models on MathVista (Accuracy) and MathVision (Accuracy). The best results are highlighted in bold.

Methods	M3CoT	ScienceQA	LLaVA-W
	ACC.↑	ACC.↑	ROUGE-L↑
SSV-CoT	44.9	57.3	35.7
w/o SR	43.5 ( -1.4 )	56.1 ( -1.2 )	33.9 ( -1.8 )
w/o SS	43.8 ( -1.1 )	56.4 ( -0.9 )	34.2 ( -1.5 )
w/o AS	44.3 ( -0.6 )	56.9 ( -0.4 )	35.0 ( -0.7 )

Table 3: Ablation study of SSV-CoT on commonsense visual reasoning benchmarks. SR denotes Structured Regions, SS denotes Sequential Selection, and AS denotes Adaptive Stopping. Numbers in parentheses denote performance degradation compared with the full model.

Figure 3: Comparison of Text-CoT and SSV-CoT on Qwen2-VL-7B for visual reasoning. Errors are shown in red. Only selected regions are displayed, and numbers indicate the token injection order.

	Order (Acc@1 %)
Visual repr.	Random	Cognition
Patch subset	63.1	65.4
Saliency regions	68.0	72.2

Table 4: 2

\times

2 ablation of visual structure and cognition order on MathVista. Each cell reports Acc@1 (%). All variants share the same visual token budget and maximum steps

K

Strategy	Acc@1 (%)	Avg #regions
Fixed- $K{=}2$ (Seq)	68.5	2.0
Fixed- $K{=}3$ (Seq)	71.0	3.0
Fixed- $K{=}4$ (Seq)	71.3	4.0
Adaptive- $K$ (Ours)	72.2	2.7

Table 5: Visual budget analysis on MathVista. Fixed-

K

variants always query exactly

K

regions. Our adaptive-

K

policy learns when to stop via a <STOP> action, achieving comparable or better accuracy with fewer region queries on average.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MathVista Acc.	Delta
Vanilla Qwen2-VL GRPO-LoRA	59.5	–
SSV-GRPO-LoRA w/o step state	62.0	+2.5
SSV-GRPO-LoRA	63.5	+4.0

Appendix

Table 6: Controlled same-training ablation on MathVista under the same 4K LoRA-GRPO setting. The results show that SSV visual access improves over vanilla GRPO, and step-aware sequential selection further improves over non-step-aware visual access.

Method	Training / Supervision	MathVista Acc.
TVC-Qwen2-VL-7B	Large-scale SFT, no RL	68.1
R1-VL-7B ( Zhang et al., 2025 )	Mulberry-260K ( Yao et al., 2025 ) + StepGRPO	69.63
MINT-CoT-7B ( Chen et al., 2025a )	54K step-level visual-token annotations + SFT + RL	73.70
SSV-CoT	50K+ SFT + GRPO, no region-level annotations	72.2

Appendix

Table 7: Additional public mathematical visual reasoning comparison on MathVista. Unlike methods relying on step-level visual-token annotations, SSV-CoT does not use region-level or visual-token-level annotations.

Method	Updates	Peak GPU	Mean GPU	Mean Update Wall	Train-loop Wall
		(GiB)	(GiB)	(s)	(h)
Vanilla GRPO-LoRA	300	52.276	44.832	55.086	4.59
SSV-GRPO-LoRA	300	56.481	46.165	62.086	5.18

Appendix

Table 8: Training efficiency comparison under the same GRPO-LoRA configuration. SSV-GRPO-LoRA increases mean GPU memory by about 3.0% and train-loop wall time by about 1.13

\times

(a) Inference efficiency on 200 MathVista samples
Method	Acc.	Latency / Sample	Extra Latency	Output Tokens	Throughput
		(ms)			(tok/s)
Qwen2-VL vLLM	57.0	757.4	–	121.58	160.5
SSV-CoT vLLM	68.0	899.0	+18.7%	143.85	160.0
(b) Adaptive stopping behavior grouped by question complexity
Complexity	Samples	Avg. #Regions	Stop Early $\leq$ 2	Query More $\geq$ 3

Appendix

Table 9: Additional analysis of SSV-CoT. (a) SSV-CoT improves accuracy with a moderate inference-latency increase while maintaining nearly the same token-level throughput. (b) The learned STOP action adapts the visual budget according to question complexity. (c) The global complement region provides modest auxiliary contextual benefit.

Figure 4: A representative failure case caused by imperfect region-bank construction. Although SSV-CoT performs sequential visual access, the policy can only choose from the precomputed candidate region bank. If the bank misses critical fine-grained chart evidence, the model may still produce an incorrect answer.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
Jun 7, 2026Lianyu Hu, Xiaoyu Ma, Zeqin Liao +1Multimodal Large Language Models Multimodal Reasoning
College of Computing and Data Science of Nanyang Technological University, Singapore.
Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Jun 1, 2026Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2Multimodal CoT Reasoning Multimodal Large Language Models
University of Virginia
SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows
Oct 1, 2026Albert Gao, Bing Xue, Andrea ZanetteVisual Reasoning Latent Visual Reasoning
Carnegie Mellon University · Meta

Jun 7, 2026J/K move · Enter open · S save

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Lianyu Hu, Xiaoyu Ma, Zeqin Liao, Yang Liu

College of Computing and Data Science of Nanyang Technological University, Singapore.

Multimodal Large Language Models Multimodal Reasoning VLM Reasoning Multimodal CoT Reasoning Visually Grounded Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a `vision-blind reasoning' paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens <THINK>, <LOOK> and <ANSWER>. These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.

Abstract

VLM Reasoning

Explore similar work

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows