Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning

Haoqiang Kang; Yinpeng Chen; Luyang Liu; Jesper Sparre Andersen; Abhijit Ogale; Baochen Sun; Lichan Hong; Ed H. Chi

cs.CVAug 20, 2026

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning

Authors: Haoqiang Kang, Yinpeng Chen, Luyang Liu, Jesper Sparre Andersen, Abhijit Ogale, Baochen Sun, Lichan Hong, Ed H. Chi

Organizations: Google DeepMind · UC San Diego · Work done at Google DeepMind

Abstract

Latent reasoning has advanced multimodal reasoning through a two-stage training paradigm: (1) a helper image is encoded into latent tokens to teach visual chain-of-thought during a supervised fine-tuning (SFT) stage, and (2) these latent tokens are further refined with reward feedback during a reinforcement learning (RL) stage. In this paper, we identify two key limitations of this framework, one in each stage. First, the SFT stage typically relies on an off-the-shelf vision encoder to encode the helper image, yielding suboptimal latent representations that may not be well aligned with the downstream reasoning task. Second, existing RL methods treat the latent component only through deterministic regularization, which constrains policy drift but does not create alternative latent trajectories for exploration. To address these limitations, we propose Scaffolding Minds. Our approach learns a dedicated scaffolding encoder that provides an optimized target in latent space, and learns both the mean and variance of the RL sampler. We further show that these two improvements are complementary, together yielding substantial gains over strong baselines. Empirically, our method improves over the strongest latent reasoning baseline by +9.5 points on FrozenLake spatial planning, with the gain widening to +19 points on the 32x32 grids, and by +5.6 points on average across nine visual-centric reasoning benchmarks.

Figures & tables

Figure 1 : Scaffolding Minds. Left: Prior latent-reasoning methods derive latent targets by encoding a training-time helper image with a frozen off-the-shelf vision encoder. Our method instead learns a dedicated scaffolding encoder that optimizes the latent target for downstream reasoning, then uses a learned Gaussian policy to sample reward-guided residual latent actions for direct exploration. Right: Scaffolding Minds improves the strongest prior latent baseline by $+9.5$ points on average across FrozenLake levels, with the gain widening to $+19$ points on the hardest $32{\times}32$ grids, and by $+5.6$ points on average across nine visual-centric reasoning benchmarks.

Figure 2 : RL performance comparison on the FrozenLake benchmark.

Figure 3 : SFT stage of Scaffolding Minds. Scaffolding Phase (top): the scaffolding encoder $\psi$ is trained to compress the helper image into target latent tokens $\mathbf{z}^{*}$ that drive the frozen base VLM to produce the correct answer. Latent-Prediction Phase (bottom): the VLM ( $\theta$ ) is then trained to predict latent tokens $\mathbf{z}$ from the input $\mathbf{x}$ alone, supervised by $\ell_{2}$ matching $\mathcal{L}=\|\mathbf{z}-\mathbf{z}^{*}\|_{2}^{2}$ to the (now frozen) scaffolding encoder’s targets, while the helper image is no longer used at inference.

Figure 4 : Three Stage 2 RL paradigms in latent space. Left: text-only RL provides no direct gradients on latent actions. Middle: VLPO applies fixed-variance Gaussian regularization to deterministic latent outputs but does not sample latent actions. Right: Scaffolding RL samples residual latent actions from a learned distribution and uses reward to optimize the latent tokens with exploration.

Base model & supervised fine-tuning
Method	Lv. 8 ( $8{\times}8$ )	Lv. 16 ( $16{\times}16$ )	Lv. 24 ( $24{\times}24$ )	Lv. 32 ( $32{\times}32$ )	Avg.
SFT	84.0	65.0	41.0	18.0	52.0
SFT+GRPO	85.0	67.0	43.0	20.0	53.8
Image-generation methods
VPRL ⋄ ( Xu et al., 2025 )	88.0	62.0	48.0	35.0	58.3
DiffThinker ⋄ ( He et al., 2025 )	92.0	68.0	53.0	44.0	64.3

Table 1 : FrozenLake results (accuracy %) on four representative levels; Avg. is their mean. Best in bold , second best underlined . Methods marked ⋄ use image-generation architectures.

\Delta

: gain of Scaffolding Encoder+RL over SFT.

Figure 5 : Per-level FrozenLake accuracy. Scaffolding Minds (red) degrades gracefully as grid size grows, while baselines collapse faster. The shaded band highlights the steadily widening advantage of Ours + RL over the strongest latent baseline (VaLR), reaching +19 points at Level 32.

Closed-source models
Method	V ⋆	BLINK	MMVP	MMStar	CVB.	HRB-4K	HRB-8K	MME	Jig.	Avg.
GPT-4o	66.0	60.0	70.7	61.6	80.1	59.0	55.5	52.0	53.0	62.0
GPT-4V	58.0	58.3	51.0	56.0	69.5	56.5	52.0	45.0	47.0	54.8
GPT-4o-mini	57.0	53.6	56.0	54.8	68.0	56.0	51.0	37.4	46.0	53.3
Claude 3.7-Sonnet	72.0	56.6	64.0	65.1	80.0	68.0	64.0	49.0	52.0	63.4
Base model & supervised fine-tuning

Table 2 : Visual-centric reasoning benchmark results (accuracy %). Bold : best; underlined : second best. † : reproduced via original checkpoint/code; ‡ : our re-implementation. § : retrained by us under our setup. Unmarked baseline cells are taken from the original papers. CVB.=CVBench, HRB=HRBench, MME=MME-RealWorld-Lite, Jig.=Jigsaw.

Figure 6 : Stage 1 visual target ablation on FrozenLake.

	Latent sampling		Stage 1 target
Stage 2 recipe	Mean	Variance	Scaffolding enc.	Frozen target
Stage 1 checkpoint (no RL)	–		72.0 $\pm 0.6$	57.8 $\pm 0.6$
Text-only GRPO	no latent sampling		72.6 $\pm 0.5$ ( $+0.6$ )	58.4 $\pm 0.5$ ( $+0.6$ )
VLPO ( Wang et al., 2026 )	latent regularization		72.9 $\pm 0.6$ ( $+0.9$ )	59.0 $\pm 0.6$ ( $+1.2$ )
Learned mean only	learned	fixed	73.6 $\pm 0.5$ ( $+1.6$ )	60.4 $\pm 0.7$ ( $+2.6$ )
Learned variance only	zero	learned	73.2 $\pm 0.6$ ( $+1.2$ )	59.8 $\pm 0.5$ ( $+2.0$ )

Table 3 : Stage 2 RL ablation on FrozenLake. We report the mean and standard deviation of accuracy over 3 seeds, averaged over the four representative Levels 8, 16, 24, and 32.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$K$	L8	L16	L24	L32	Avg.
1	85.0	70.0	54.0	45.0	63.5
2	88.0	74.0	57.0	48.0	66.8
4	94.0	79.0	63.0	52.0	72.0
8	93.0	78.0	61.0	51.0	70.8
16	91.0	76.0	59.0	50.0	69.0

Appendix

Table 4 : Number of latent tokens

K

on FrozenLake (Stage 1 only, accuracy %).

$K$	L8	L16	L24	L32	Avg.
1	85.0	70.0	54.0	45.0	63.5
2	88.0	74.0	57.0	48.0	66.8
4	94.0	79.0	63.0	52.0	72.0
8	93.0	78.0	61.0	51.0	70.8
16	91.0	76.0	59.0	50.0	69.0

Appendix

Table 4 : Number of latent tokens

K

on FrozenLake (Stage 1 only, accuracy %).

Helper image	Scaffold. enc.	L8	L16	L24	L32	Avg.
Red-arrow overlay	$\times$	85.0	67.0	52.0	40.0	61.0
Red-arrow overlay	$\checkmark$	92.0	76.0	58.0	46.0	68.0
Value-function heatmap	$\times$	88.0	71.0	47.0	25.0	57.8
Value-function heatmap	$\checkmark$	94.0	79.0	63.0	52.0	72.0

Appendix

Table 5 : Helper image choice on FrozenLake (Stage 1 only, accuracy %). Scaffolding encoder

\checkmark

: our scaffolding encoder is used;

\times

: a frozen off-the-shelf vision encoder is used as in prior work.

	Variant	V ⋆	BLINK	MMVP	MMStar	CVB.	HRB-4K	HRB-8K	MME	Jig.	Avg.
Stage 1	Frozen target	82.4	57.5	71.3	67.8	79.0	72.0	68.3	45.2	52.8	66.3
Stage 1	Scaffolding encoder	87.4	64.8	73.3	72.1	86.4	73.6	71.1	53.4	56.0	70.9
Stage 2	VLPO	89.3	66.2	75.3	72.9	87.6	75.9	72.9	55.7	58.2	72.7
Stage 2	Scaffolding RL	90.6	67.2	76.7	73.5	88.4	77.5	74.1	57.3	59.7	73.9

Appendix

Table 6 : Ablation on the visual-centric reasoning benchmarks (accuracy %). All rows use the same backbone and Zebra-CoT training data. Stage 1 rows differ only in the latent target; Stage 2 rows start from the same Scaffolding Encoder checkpoint and differ only in the RL recipe.

Preceding latent context	V ⋆	MMVP	CVB.	HRB-4K	Mean	Forward passes	Training time
Target $\mathbf{z}^{*}$ (teacher forcing, ours)	87.4	73.3	86.4	73.6	80.2	$1{+}1$	$1.0\times$
Predicted $\mathrm{sg}(\hat{\mathbf{z}})$ (self-predicted)	87.1	72.9	86.2	73.3	79.9	$B{+}1$	$3.2\times$

Appendix

Table 7 : Teacher-forced vs. self-predicted latent context (Stage 1, accuracy %). Forward passes : predictor passes

+

answer pass per training trace with

B

latent blocks. Training time : Stage 1 training time relative to teacher forcing.

Metric	Text-only GRPO	VLPO	Scaffolding RL
Reward s.d. within a rollout group	0.15	0.32	0.42
Pass@4 / @8 / @16 (%)	74 / 76 / 80	75 / 79 / 82	79 / 82 / 85
Latent Gaussian scale $\sigma$	N/A	0.05 ∗ (fixed)	0.04–0.15 (learned)

Appendix

Table 8 : RL exploration metrics on FrozenLake, all from the same Stage 1 checkpoint (

72.0\%

). ∗ VLPO does not sample latents;

0.05

is the fixed scale

\sigma_{0}

of its Gaussian regularizer (Section 2 ).

$\lambda_{\text{latent}}$	$\lambda_{\text{task}}$	L8	L16	L24	L32	Avg.
1.0	0.0	88.0	72.0	54.0	45.0	64.8
1.0	0.5	94.0	79.0	63.0	52.0	72.0
1.0	1.0	92.0	77.0	60.0	51.0	70.0
0.5	1.0	93.0	78.0	62.0	52.0	71.3

Appendix

Table 9 : Latent generation loss weights on FrozenLake (accuracy %; Avg. over the four levels shown).

\lambda_{\text{latent}}

and

\lambda_{\text{task}}

weight the latent matching term and the task loss term, respectively.

	Qwen2.5-VL-3B (full FT)					Qwen2.5-VL-7B (LoRA $r{=}32$ )
Method	Lv. 8	Lv. 16	Lv. 24	Lv. 32	Avg.	Lv. 8	Lv. 16	Lv. 24	Lv. 32	Avg.
SFT	75.0	60.0	38.0	14.0	46.8	70.0	52.0	32.0	13.0	41.8
SFT+GRPO	77.0	62.0	40.0	16.0	48.8	71.0	54.0	34.0	14.0	43.3
LVR ( Li et al., 2025b )	78.0	63.0	41.0	18.0	50.0	73.0	56.0	36.0	16.0	45.3
Mirage ( Yang et al., 2025b )	79.0	65.0	43.0	20.0	51.8	74.0	58.0	38.0	18.0	47.0
VaLR ( Jeon et al., 2026 )	83.0	71.0	51.0	28.0	58.3	80.0	65.0	44.0	24.0	53.3

Appendix

Table 10 : Effect of model scale and LoRA fine-tuning on FrozenLake (accuracy %). Left: Qwen2.5-VL-3B with full fine-tuning. Right: Qwen2.5-VL-7B with LoRA

r{=}32

Latent reasoning methods
		FrozenLake (single block)		Visual-centric (interleaved)
Method	$\boldsymbol{K}$	Latency (ms)	Overhead	Latency (ms)	Overhead
Qwen2.5-VL-7B (base)	—	260	—	740	—
Mirage ( Yang et al., 2025b )	4	266	$+2.3\%$	–	–
LVR ( Li et al., 2025b )	4	267	$+2.7\%$	794	$+7.3\%$
VaLR ( Jeon et al., 2026 )	4	268	$+3.1\%$	802	$+8.4\%$

Appendix

Table 11 : Single-query latency (single A100, BF16, averaged over 100 queries).

K

: latent tokens per block (— for methods that process or generate images). –: not measured.

Explore similar work

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
May 4, 2026Xin Zhang, Qiqi Tao, Jiawei Du +2Latent Visual Reasoning Efficient Latent Reasoning
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design +1
Leveraging Latent Visual Reasoning in Silence
May 18, 2026Dongyao Zhu, Zhen Wang, Xi Xiao +7Latent Visual Reasoning Visual Reasoning
North Carolina State University · UC, San Diego · University of Alabama at Birmingham +4
OPLD: On-Policy Latent Distillation for Multimodal Reasoning
Jul 30, 2026Shoutai Zhu, Tianyang Xu, Bin Sun +3Multimodal Reasoning Latent Visual Reasoning
ByteDance

May 4, 2026, cs.LGJ/K move · Enter open · S save

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs

Xin Zhang, Qiqi Tao, Jiawei Du, Moyun Liu+1

Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design+1

Latent Visual Reasoning Efficient Latent Reasoning Recent Vision-Language Models Multimodal Large Language Models Inference-Time

Open paper AskPDF arXiv

Continuous latent-space reasoning offers a compact alternative to textual chain-of-thought for multimodal models, enabling high-dimensional visual evidence to be integrated without explicit reasoning tokens. However, we identify a previously overlooked optimization pathology in existing latent visual reasoning methods: although visual latents become semantically enriched during training, their contribution to final answer prediction is systematically suppressed. Within the shared parameter space, the autoregressive objective favors shortcut reliance on direct visual input, driving latent tokens toward transition-like states rather than informative reasoning content. We term this phenomenon Silenced Visual Latents. To address it, we disentangle the two conflicting objectives by directly optimizing the latent reasoning at inference time, keeping backbone parameters frozen. In Stage I, visual latents are warmed up via query-guided contrastive latent--visual alignment, improving semantic quality while preventing latent collapse. In Stage II, the latent reasoning is further optimized via a confidence-progression reward, which incentivizes predicted token distributions along the latent span to become progressively more concentrated, routing predictions through the latent reasoning rather than bypassing it. Experiments across eight benchmarks and four model backbones show that inference-time latent optimization, without any parameter updates, effectively unleashes the suppressed reasoning capacity of visual latents.

Abstract

Explore similar work

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs

Leveraging Latent Visual Reasoning in Silence

OPLD: On-Policy Latent Distillation for Multimodal Reasoning