Visual Scaffolding: Toward Native Multimodal Reasoning

Run Shao; Zhaoyang Zhang; Ziyu Li; Linrui Xu; Ji Qi; Xiu Su; Yichao Cao; Chao Tao

cs.CVOct 18, 2026

Visual Scaffolding: Toward Native Multimodal Reasoning

Authors: Run Shao, Zhaoyang Zhang, Ziyu Li, Linrui Xu, Ji Qi, Xiu Su, Yichao Cao, Chao Tao,

Organizations: Central South University

Abstract

The success of chain-of-thought (CoT) in complex reasoning can be attributed to a textual scaffolding effect: explicit intermediate results can be examined, reused, and composed to support subsequent reasoning, much as scaffolding supports a building under construction. However, mainstream multimodal LLMs lack an equivalent scaffold for visual distinctions that are difficult to express in concise verbal descriptions. We therefore ask: Can native visual representations become first-class citizens alongside language, providing a visual scaffold for reasoning? We define native multimodal reasoning as reasoning in which native visual representations and language jointly carry reusable intermediate states. As a concrete implementation, the visual scaffolding mechanism we proposed lets the MLLM autonomously initiate a retrieval action at any reasoning step. A lightweight retriever selects native visual features, which are then incorporated directly into the evolving CoT and remain available to subsequent steps. To evaluate this mechanism under controlled visual demands, we introduce NMR-Bench, comprising Burden Counting and Path Tracing. With 88M added parameters (approximately 2%) on Qwen3-VL-4B-Thinking, our model achieves 66.43% and 72.50% accuracy, respectively, surpassing the evaluated GPT-5.5 and Gemini 3.1 Pro baselines. Controlled interventions show that changing selected visual content affects task accuracy and can redirect subsequent visual selection even when intervening text is held fixed. These findings show that native visual states can function as intermediate reasoning material and point toward reusable scaffolds for additional modalities.

Figures & tables

Figure 1: From textual CoT to visual scaffolding. Textual CoT describes spatial relations in words. Visual scaffolding inserts native visual features as reusable intermediate material. Colored boxes link source regions to illustrative feature views. Right: NMR-Bench accuracy with a 4B backbone and 88M added parameters. Traces are schematic.

Figure 2: Visual scaffolding mechanism. The MLLM generates text until [ref] initiates a visual request. The Visual Retriever selects native features from the dense visual bank, and the Evidence Adaptor inserts them into CoT before generation resumes. The standard visual bank supplies the original image input. Panels detail (a) dual-resolution encoding, (b) retrieval conditioned on reasoning context and local spatial structure, and (c) residual evidence adaptation.

Figure 3: Learning to select and use visual evidence. (a) During training, GT-selected features enter the reasoning sequence. Next-token prediction supervises text and request tokens, while focal and Dice losses supervise feature selection. Phase I emphasizes the new interface; Phase II jointly adapts it with the backbone. (b) During inference, the model initiates each request and selects the features inserted before generation resumes. Training allows up to 64 evidence tokens per request; the main evaluation caps predicted selections at 8.

Frontier Multimodal LLMs
Model	Burden Counting								Path Tracing
	Overall		Easy		Medium		Hard		Overall		Easy		Medium		Hard
	Acc.	MAE	Acc.	MAE	Acc.	MAE	Acc.	MAE	Acc.	CRA	Acc.	CRA	Acc.	CRA	Acc.	CRA
MiniMax-M3	4.76	9.14	10.71	2.84	3.57	7.92	0.00	16.66	18.33	36.77	40.00	53.89	14.17	35.50	0.83	20.91
Claude-Opus-4.8	6.43	9.38	14.29	3.03	2.86	8.27	2.14	16.85	33.06	54.09	66.67	76.11	29.17	52.71	3.33	33.45
Kimi-K2.6	9.29	15.67	20.71	3.19	5.00	11.87	2.14	31.66	58.06	68.71	76.67	81.67	62.50	73.50	35.00	50.97

Table 1: NMR-Bench results. VScaffold-4B is our visual-scaffolding model. Acc. and CRA are percentages. MAE uses integer-answer tasks.

\Delta_{\mathrm{NoRef}}=M_{\mathrm{ours}}-M_{\mathrm{NoRef}}

reports signed gains over the trigger-only No-Ref control trained on the same data without inserted visual features. Higher Acc./CRA and lower MAE are better. Bold marks column-best values. Training and comparison settings are in Appendices A.4 and C .

Figure 4: Visual intermediate states in two benchmark settings. Counting interleaves a running tally with features selected from large, tilted unicorns (left). Path Tracing uses selected curve segments to recover endpoint connections (right). Outlines and insets indicate the source regions of the selected features. Traces are abbreviated and edited for presentation.

Figure 5: Effects of changing visual intermediate content. (a) Replacing features at every request changes task accuracy. (b) Replacing one middle block measures the effect of a local intervention relative to unmodified replay. (c) Immediate next-token choices and subsequent visual selections capture different responses to a changed block. Intervening words are fixed only when comparing subsequent selections. Fractions indicate the share of features replaced; whiskers show 95% bootstrap intervals. The evidence cap is 32.

Figure 6: Selective visual support. (a) Accuracy at evidence caps 8, 16, and 32. (b) Coverage at cap 32. Whiskers show 95% bootstrap intervals; the panel (b) axis starts at 80%.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Path	Counting
Training examples / epochs	40,000 / 4	32,000 / 7
Updates / Phase I updates	5,000 / 1,250	7,000 / 1,000
Global batch / microbatch per GPU	32 / 1	32 / 1
GPUs / gradient accumulation	8 / 4	8 / 4
Phase I peak / constant rates	$10^{-4},10^{-7},10^{-8}$	same
Phase II peak rates	$5\!\times\!10^{-5},10^{-5},5\!\times\!10^{-6}$	same

Appendix

Table 2: Optimization settings for the final models. Learning-rate triplets list the scaffold, language, and vision groups in that order.

Task	Training size	Test	Test stratification
Burden Counting	32,000	420	7 forms $\times$ 3 levels $\times$ 20
Path Tracing	40,000	360	4 marker types $\times$ 3 levels $\times$ 30

Appendix

Table 3: NMR-Bench training and test sets. Training examples include reasoning traces and patch annotations. Test inputs contain only an image and a question.

Form	Query condition	Answer value
T1a	Object category	Integer count
T2a	Category and size	Integer count
T2b	Category and tilt	Integer count
T2c	Category and occlusion	Integer count
T2d	Category, size, and tilt	Integer count
T3a	Compare large and small groups	Group label and count difference

Appendix

Table 4: Burden Counting task forms and canonical answer values.

Table label	Requested model identifier	Counting	Path
MiniMax-M3	MiniMax-M3	0	34
Claude-Opus-4.8	claude-opus-4-8	0	0
Kimi-K2.6	kimi-k2.6	197	75
Qwen-3.7-Plus	qwen3.7-plus	40	34
GPT-5.5	gpt-5.5-ca	0	0
Gemini-3.1-Pro	gemini-3.1-pro-preview	1	77

Appendix

Table 5: API model identifiers and length-limit stops for both tasks. Stopped predictions remain in the accuracy denominator and are scored whenever they contain a parseable answer.

Intervention	Path $\Delta$ Acc.	Path $\Delta$ CRA	Counting $\Delta$ Acc.
Early, all	$-41.7\;[-52.1,-31.2]$	$-30.7\;[-37.1,-24.3]$	$-31.0\;[-41.7,-20.2]$
Middle, all	$-41.7\;[-52.1,-32.3]$	$-30.2\;[-36.1,-24.4]$	$-22.6\;[-34.5,-10.7]$
Late, all	$-21.9\;[-32.3,-11.5]$	$-16.1\;[-22.1,-10.4]$	$-22.6\;[-33.3,-11.9]$
Middle, seed 2	$-38.5\;[-49.0,-28.1]$	$-27.5\;[-34.2,-21.1]$	$-20.2\;[-32.1,-8.3]$
Middle, seed 3	$-42.7\;[-52.1,-33.3]$	$-31.2\;[-37.5,-24.9]$	$-32.1\;[-42.9,-21.4]$
Middle, norm matched	$-41.7\;[-52.1,-31.2]$	$-26.3\;[-31.9,-20.9]$	$-22.6\;[-34.5,-10.7]$

Appendix

Table 6: Paired local effects relative to sham. Values are percentage-point changes with 95% confidence intervals. The 96 Path and 84 Counting images have sham accuracies of 60.42% and 58.33%, respectively. “Half” uses the same donors and prefix as complete replacement.

Figure 7: Identical words, different visual decisions. (a) Selected content and (b) same-image replacement. Both branches emit the displayed text, but their next selected regions (c,d) are disjoint. Normal recovers all four pairs, while replacement recovers two. The case is closest to median immediate JSD ( $2.97\times 10^{-5}$ nats) among the 13 correct-to-incorrect Path cases with identical nonempty intervening text and changed selections. Overlays show source cells. Panel labels report actual block sizes, subject to a cap of 32.

Task	$K=8$	$K=16$	$K=32$
Burden Counting	171.8	201.5	204.3
Path Tracing	346.6	528.3	764.9

Appendix

Table 7: Mean evidence tokens per image on the standard test sets. A smaller cap reduces the visual material inserted into the reasoning trace.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
Jun 7, 2026Lianyu Hu, Xiaoyu Ma, Zeqin Liao +1Multimodal Large Language Models Multimodal Reasoning
College of Computing and Data Science of Nanyang Technological University, Singapore.
Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Jun 1, 2026Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2Multimodal CoT Reasoning Multimodal Large Language Models
University of Virginia
SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows
Oct 1, 2026Albert Gao, Bing Xue, Andrea ZanetteVisual Reasoning Latent Visual Reasoning
Carnegie Mellon University · Meta

Jun 7, 2026J/K move · Enter open · S save

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Lianyu Hu, Xiaoyu Ma, Zeqin Liao, Yang Liu

College of Computing and Data Science of Nanyang Technological University, Singapore.

Multimodal Large Language Models Multimodal Reasoning VLM Reasoning Multimodal CoT Reasoning Visually Grounded Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a `vision-blind reasoning' paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens <THINK>, <LOOK> and <ANSWER>. These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.

Abstract

Multimodal Reasoning

Explore similar work

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows