LatentWeave: Test-Time Latent Optimization for Multimodal Reasoning

Fanci Meng; Kun Wang; Guibin Zhang; Xinlei Yu; Jiaming Zhang; Ming Li; Yang Wang

cs.LGOct 29, 2026

LatentWeave: Test-Time Latent Optimization for Multimodal Reasoning

Authors: Fanci Meng, Kun Wang, Guibin Zhang, Xinlei Yu, Jiaming Zhang, Ming Li, Yang Wang

Organizations: University of Science and Technology of China · Nanyang Technological University · National University of Singapore · The Chinese University of Hong Kong ·

Abstract

Recent advances in test-time optimization have shown that the multimodal reasoning performance of large multimodal language models (MLLMs) can be improved during inference without modifying backbone model parameters. However, existing approaches primarily operate at the prompt or output level, leaving the latent reasoning process largely uncontrolled and insufficient to prevent error accumulation during autoregressive generation. To address this limitation, we propose LatentWeave, a test-time framework that treats multimodal reasoning as a controllable latent trajectory. LatentWeave performs structured Latent Weaving, progressively refining intermediate hidden states segment-wise to correct emerging deviations, and further introduces Reasoning Anchors via lightweight WEAVE tokens to stabilize reasoning structure and guide answer-oriented generation. These mechanisms are designed to enable fine-grained, step-aware intervention in the latent reasoning process while keeping the main MLLM backbone parameters fixed. Extensive experiments across diverse multimodal reasoning benchmarks report improvements over the evaluated baselines; the contribution of latent optimization, anchor training, and additional test-time computation is analyzed separately in the ablation and efficiency studies. Our code is available at https://github.com/jins7/LatentWeave.

Figures & tables

Figure 1: LatentWeave ’s Latent Weaving functions much like rectifying faulty knitting, preventing accumulation of errors and subsequent failure, whereas Reasoning Anchors mirror adding buttons at appropriate positions to a finished sweater body, enhancing its structural integrity.

Figure 2: The overall framework of our proposed LatentWeave .

InternVL2.5-4B
Method	Mathematical Reasoning			General Visual QA			Diagram
Method	Math Vision	Math Verse	Math Vista	MMMU	RealWorld QA	MME	AI2D
Vanilla	$19.58$	$34.49$	$60.17$	$46.82$	$61.04$	$2316$	$79.63$
CoT	$20.93_{{\color[rgb]{0,0.7,0}\uparrow 1.35}}$	$35.95_{{\color[rgb]{0,0.7,0}\uparrow 1.46}}$	$60.81_{{\color[rgb]{0,0.7,0}\uparrow 0.64}}$	$48.22_{{\color[rgb]{0,0.7,0}\uparrow 1.40}}$	$62.19_{{\color[rgb]{0,0.7,0}\uparrow 1.15}}$	$2331_{{\color[rgb]{0,0.7,0}\uparrow 15.0}}$	$80.10_{{\color[rgb]{0,0.7,0}\uparrow 0.47}}$
SFT	$19.28_{{\color[rgb]{0.8,0,0}\downarrow 0.30}}$	$35.13_{{\color[rgb]{0,0.7,0}\uparrow 0.64}}$	$60.82_{{\color[rgb]{0,0.7,0}\uparrow 0.65}}$	$49.51_{{\color[rgb]{0,0.7,0}\uparrow 2.69}}$	$63.29_{{\color[rgb]{0,0.7,0}\uparrow 2.25}}$	$2309_{{\color[rgb]{0.8,0,0}\downarrow 7.0}}$	$80.87_{{\color[rgb]{0,0.7,0}\uparrow 1.24}}$
GRPO	$20.07_{{\color[rgb]{0,0.7,0}\uparrow 0.49}}$	$35.66_{{\color[rgb]{0,0.7,0}\uparrow 1.17}}$	$61.34_{{\color[rgb]{0,0.7,0}\uparrow 1.17}}$	$48.86_{{\color[rgb]{0,0.7,0}\uparrow 2.04}}$	$61.35_{{\color[rgb]{0,0.7,0}\uparrow 0.31}}$	$2302_{{\color[rgb]{0.8,0,0}\downarrow 14.0}}$	$78.41_{{\color[rgb]{0.8,0,0}\downarrow 1.22}}$

Table 1: Main results under two MLLM backbones across mathematical reasoning, visual QA, and diagram understanding. The best and second-best results per benchmark are shown in bold and underline , respectively.

Backbone	Method	MathVision	MathVerse	MathVista	MMMU	RealWorldQA	MME	AI2D
Qwen2.5-VL-7B	Vanilla	23.65	47.98	66.15	56.96	66.02	2328	80.58
	+ LatentWeave	30.11	53.80	72.03	63.15	70.89	2388	88.93
	$\Delta$	+6.46	+5.82	+5.88	+6.19	+4.87	+60.0	+8.35
Qwen3.5-4B	Vanilla	66.12	66.37	75.80	68.32	73.33	2372	85.91
	+ LatentWeave	72.69	71.44	79.30	72.81	76.71	2404	89.25
	$\Delta$	+6.57	+5.07	+3.50	+4.49	+3.38	+32.0	+3.34

Table 2: Performance gains of LatentWeave over vanilla decoding across five MLLM backbones on diverse multimodal reasoning benchmarks.

\Delta

denotes the absolute improvement on each benchmark.

Figure 3: Qualitative case study of reasoning error accumulation and correction. Left : Incorrect early reasoning leads to error propagation. Middle : Latent Weaving corrects the reasoning trajectory. Right : <WEAVE> tokens enhance the clarity and structure of the corrected reasoning trajectory.

Figure 5: Sensitivity to segment ratio

\rho

, anchor-step count

J

(one <WEAVE> opening delimiter per step), and update count

K

on MathVista and RealWorldQA.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$\mathbf{x}$	Textual input (question or instruction)
$\mathbf{v}$	Visual input (image or diagram)
$\mathbf{y}=(y_{1},\dots,y_{T})$	Generated output token sequence
$\mathbf{h}_{t}$	Hidden state at decoding step $t$
$\mathbf{H}=(\mathbf{h}_{1},\dots,\mathbf{h}_{T})$	Latent reasoning trajectory composed of hidden states
$f_{\theta}$	Pretrained vision–language model with frozen parameters $\theta$

Appendix

Table 4: Notation summary for latent trajectories, reward signals, and reasoning anchors.

Phase / split	$\beta$	Verifier weights $(w_{\mathrm{rc}},w_{\mathrm{ac}},w_{\mathrm{cp}},w_{\mathrm{uc}})$	Prompt / config ID	Status
Offline collection	0.3	$(0.30,0.40,0.15,0.15)$	rw-v1/offline	Verified
Development	0	$(0.30,0.40,0.15,0.15)$	rw-v1/dev	Verified
Test evaluation	0	$(0.30,0.40,0.15,0.15)$	rw-v1/test	Verified

Appendix

Table 5: Default reward configuration for offline trajectory collection and evaluation. The verifier weight vector is shared across these settings; the ground-truth coefficient

\beta

is zero for every evaluation example. The weight-sensitivity study is reported separately in Table 15 .

Backbone / split	Segmentation $(\rho,N_{\mathrm{seg}})$	Rollouts $(M,\tau)$	Updates $(K,\eta,\mathrm{clip})$	Baseline / regularization $(b,\lambda)$	Seed / ID
Qwen2.5-VL-7B / test	$(0.25,\,4)$	$(4,\,0.8)$	$(6,\,1{\times}10^{-2},\,1.0)$	LOO / $0.05$	$\{0,1,2\}$ / opt-v1
InternVL2.5-4B / test	$(0.25,\,4)$	$(4,\,0.8)$	$(6,\,1{\times}10^{-2},\,1.0)$	LOO / $0.05$	$\{0,1,2\}$ / opt-v1

Appendix

Table 6: Inner-loop optimization protocol for the four-segment configurations.

\mathrm{clip}

bounds the latent gradient norm,

b^{(m)}

is the leave-one-out baseline, and

\lambda

is the anchoring coefficient in Eq. ( 5 ). The realized segment count follows Eq. ( 4 ) and can differ for other generation lengths.

Stage	Input $\rightarrow$ output	Serialization / insertion	Aux call / cache	Leakage / config
Anchor training	$(\mathbf{x},\mathbf{v})\rightarrow$ anchor text	Six spans: <WEAVE> $\,\mathrm{step}_{j}\,$ </WEAVE> , $\leq 80$ tokens each	No / N/A	Train split only
Evaluation	$(\mathbf{x},\mathbf{v})\rightarrow$ anchor text	Identical string; inserted after visual tokens, before the CoT trigger	Once / no cross-example reuse	No labels / anchor-v1

Appendix

Table 7: Reasoning-anchor training–inference interface. The same serialization is used in training and inference.

Question Type	Required Output Format
Multiple-Choice Questions	You must answer this multiple-choice question. Think step by step, then at the very end of your response, output: Final Answer: [X] where X is the chosen option label (A, B, C, D, E, etc.). Do not include any text after this line.
Open-Ended Questions	You must answer this open-ended question. Think step by step, then at the very end of your response, output: \boxed{your final answer} where your final answer is the concise answer to the question. Do not include any text after this line.

Appendix

Table 8: Evaluation-time answer formatting prompts used for different question types.

Variant	Latent	Anchors	MathVista	MMMU	AI2D
Vanilla	Off	Off	$66.15_{\pm 0.00}$	$56.96_{\pm 0.00}$	$80.58_{\pm 0.00}$
Anchors only	Off	On	$68.42_{\pm 0.51}$	$59.18_{\pm 0.76}$	$84.10_{\pm 0.44}$
Latent Weaving only	On	Off	$71.52$	$60.87$	$86.20$
LatentWeave (Full)	On	On	$\mathbf{72.03}_{\pm 0.55}$	$\mathbf{63.15}_{\pm 0.80}$	$\mathbf{88.93}_{\pm 0.46}$

Appendix

Table 9: Component ablation on Qwen2.5-VL-7B . Entries with three-seed summaries show mean

\pm

standard deviation; the latent-only row reports point estimates aligned with Figure 4 .

Method	Rollouts / updates	Main FWD / BWD	Aux / verifier	Tokens	Time / peak memory
Self-Consistency	$8/0$	$8$ /0	$0$ / $0$	2.8k	14 s / 21 GB
Self-Refine	$3/0$	$3$ /0	$0$ / $2$	1.3k	7 s / 20 GB
TTRV	–/–	–/–	$0$ / $0$	3.0k	18 s / 22 GB
Same-verifier Best-of- $N$	$96/0$	–/0	$0$ / $96$	5.6k	31 s / 21 GB
LatentWeave	$96/24$	–/ $24$	$1$ / $96$	9.2k	52 s / 41 GB

Appendix

Table 10: Inference cost per example on the MathVista-200 cost subset with Qwen2.5-VL-7B and one A100-80GB GPU. The latent method uses four segments. Rollouts accumulate over updates; verifier evaluations count external candidate-level rewards. Dashes denote unspecified operation counts. TTRV updates model parameters using its intrinsic frequency- and entropy-based rewards.

Condition	Latent / anchors	Rollouts / evaluations	Comparison	Accuracy	Gain (pp)
Vanilla	Off / Off	$1/0$	Reference	$66.15$	–
Best-of- $N$	Off / Off	$96/96$	Same verifier	$69.44$	$+3.29$
Latent Weaving only	On / Off	$96/96$	Same verifier	$71.52$	$+2.08$
Full	On / On	$96/96$ ( $+1$ aux)	Same verifier	$72.03$	$+0.51$

Appendix

Table 11: Same-verifier comparison on MathVista with Qwen2.5-VL-7B . Best-of-

N

and both four-segment latent variants each use 96 candidate-level verifier evaluations. Entries report point estimates; gains are percentage-point differences from the preceding row. Vanilla provides the unoptimized accuracy reference.

Verifier	Audit $N$	Agreement (CI)	Discrimination	Calibration	Bias checks
Consistency ( $\mathrm{rc}$ )	$400$	$0.77$ $[0.73,0.81]$	AUROC $0.73$	ECE $0.12$	len. $+0.21$ ; self $+2.4$
Correctness ( $\mathrm{ac}$ )	$400$	$0.91$ $[0.88,0.94]$	AUROC $0.89$	ECE $0.06$	len. $+0.07$ ; self $+1.1$
Completeness ( $\mathrm{cp}$ )	$400$	$0.75$ $[0.71,0.79]$	AUROC $0.70$	ECE $0.13$	len. $+0.34$ ; self $+2.9$
Understanding ( $\mathrm{uc}$ )	$400$	$0.80$ $[0.76,0.84]$	AUROC $0.75$	ECE $0.11$	len. $+0.16$ ; self $+3.6$

Appendix

Table 12: Verifier reliability and bias, audited against a disjoint gold set. Agreement is the fraction of items matching the gold judgment, with a 95% interval; discrimination is AUROC against final answer correctness; calibration is expected calibration error. Bias checks report the Spearman correlation between the verifier score and output length, and the score gain given to same-family generations.

Method	Backbone / checkpoint	Decoding / samples	Train data / budget	Verifier	Seeds / config
CoT	Qwen2.5-VL-7B-Instruct	Greedy, $N{=}1$ , 1024 tok.	–	–	1 / base-cot
Self-Consistency	Qwen2.5-VL-7B-Instruct	$\tau{=}0.7$ , top- $p$ $0.95$ , $N{=}8$	–	Majority vote	3 / base-sc
Self-Refine	Qwen2.5-VL-7B-Instruct	Greedy, 2 refine rounds	–	Self-critique	3 / base-sr
TTRV	Qwen2.5-VL-7B-Instruct	$\tau{=}0.7$ , $N{=}8$	Unlabeled test inputs; online GRPO	Frequency + entropy	3 / base-ttrv
SFT / GRPO	Qwen2.5-VL-7B-Instruct	Greedy, $N{=}1$	18k CoT traces; 2 ep. / 1.0k steps	– / same reward	3 / base-sft , base-grpo
IVT-LR / LaCoT	Released checkpoints	Per original paper	Same 18k pool	Per original paper	3 / base-ivtlr , base-lacot

Appendix

Table 13: Baseline reproduction protocol. All rows use the same split, formatting prompt, and answer extractor.

Parameter sweeps from Figure 5
Factor	Values	MathVista	RealWorldQA	Spread
$\rho$	0.05/0.10/0.15/0.20/0.25/0.30	66.5/67.2/67.8/68.2/ 68.5 /67.5	66.3/66.8/67.1/67.3/ 67.5 /67.2	2.0/1.2
$J$	2/4/6/8	66.8/67.5/ 68.2 /67.6	66.5/ 67.4 /67.0/66.3	1.4/1.1
$K$	3/4/5/6/7	67.2/67.6/67.9/ 68.3 /68.0	66.5/67.0/67.3/ 67.6 /67.1	1.1/1.1
Separate anchor-delimiter control
Factor	Values	MathVista	MMMU	Spread

Appendix

Table 14: Parameter sensitivity and anchor-delimiter control. The upper panel matches Figure 5 ; the lower panel reports the separate delimiter experiment. Accuracies follow the order of the values, and spread is

\max-\min

for the two listed benchmarks. Bold marks each benchmark’s highest accuracy within a sweep.

Configuration	Weight vector	MathVista	MMMU
Default	$(0.30,0.40,0.15,0.15)$	$\mathbf{72.03}$	$\mathbf{63.15}$
Uniform	$(0.25,0.25,0.25,0.25)$	$71.38$	$62.47$
Correctness-heavy	$(0.10,0.70,0.10,0.10)$	$71.26$	$62.31$
Structure-heavy	$(0.40,0.15,0.30,0.15)$	$70.54$	$61.42$

Appendix

Table 15: Reward-weight sensitivity on Qwen2.5-VL-7B with the online budget held fixed. The vector lists weights for reasoning consistency, answer correctness, completeness, and question understanding, in that order.

Reward setting	MathVision	RealWorldQA	$\Delta$ vs. full
Full composite reward	$\mathbf{30.11}_{\pm 0.38}$	$\mathbf{70.89}_{\pm 0.52}$	–
Without consistency	$29.04_{\pm 0.42}$	$69.77_{\pm 0.55}$	$-1.07$ / $-1.12$
Without correctness	$28.30_{\pm 0.47}$	$69.08_{\pm 0.61}$	$-1.81$ / $-1.81$
Without completeness	$29.63_{\pm 0.35}$	$70.46_{\pm 0.49}$	$-0.48$ / $-0.43$
Without understanding	$29.72_{\pm 0.40}$	$70.31_{\pm 0.53}$	$-0.39$ / $-0.58$

Appendix

Table 16: Reward-component ablation on Qwen2.5-VL-7B , mean

\pm

standard deviation over three seeds. The last column is the change relative to the full composite reward on MathVision / RealWorldQA.

Location	Task / backbone / split	Prompt / config	Raw artifact	Metric / value	Status
Table 1	MathVista / Qwen2.5-VL-7B / testmini	cot-v1 / lw-v1	qwen7b_mathvista	Acc. $72.03$	Verified
Table 2	MathVision / STEP3-VL-10B / test	cot-v1 / lw-v1	step3_mathvision	Acc. $70.29$	Verified
Table 9	AI2D / Qwen2.5-VL-7B / test	cot-v1 / lw-anchor-v1	qwen7b_ai2d_s{0,1,2}	Acc. $84.10_{\pm 0.44}$	Verified
Table 10	Cost / Qwen2.5-VL-7B / MathVista-200	lw-v1	cost_qwen7b	$52$ s / item	Verified

Appendix

Table 17: Result-reconciliation ledger with representative entries from selected result tables. Artifact names are the stems of the corresponding .jsonl / .json files; full command lines and hashes remain in the run manifest.

MLLM Reasoning

See all 34 papers

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
May 4, 2026Xin Zhang, Qiqi Tao, Jiawei Du +2Inference-Time Optimization Test-Time Optimization
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design +1
DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
Jun 13, 2026David Huang, Lianlei ShanLatent World Models Multimodal Reasoning
University of Toronto · Tsinghua University
AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution
May 25, 2026Zehao Wang, Yihan Zeng, Zidong Gong +5RL for Language Model Reasoning Multimodal Large Language Models
Harbin Institute of Technology · Huawei Noah’s Ark Lab · Independent Researcher

May 4, 2026J/K move · Enter open · S save

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs

Xin Zhang, Qiqi Tao, Jiawei Du, Moyun Liu+1

Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design+1

Inference-Time Optimization Test-Time Optimization Multimodal Large Language Models Latent Visual Reasoning VLM Reasoning Continuous Latent Reasoning+4

Open paper AskPDF arXiv

Continuous latent-space reasoning offers a compact alternative to textual chain-of-thought for multimodal models, enabling high-dimensional visual evidence to be integrated without explicit reasoning tokens. However, we identify a previously overlooked optimization pathology in existing latent visual reasoning methods: although visual latents become semantically enriched during training, their contribution to final answer prediction is systematically suppressed. Within the shared parameter space, the autoregressive objective favors shortcut reliance on direct visual input, driving latent tokens toward transition-like states rather than informative reasoning content. We term this phenomenon Silenced Visual Latents. To address it, we disentangle the two conflicting objectives by directly optimizing the latent reasoning at inference time, keeping backbone parameters frozen. In Stage I, visual latents are warmed up via query-guided contrastive latent--visual alignment, improving semantic quality while preventing latent collapse. In Stage II, the latent reasoning is further optimized via a confidence-progression reward, which incentivizes predicted token distributions along the latent span to become progressively more concentrated, routing predictions through the latent reasoning rather than bypassing it. Experiments across eight benchmarks and four model backbones show that inference-time latent optimization, without any parameter updates, effectively unleashes the suppressed reasoning capacity of visual latents.

Abstract

MLLM Reasoning

Explore similar work

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution