Visual Enhanced Depth Scaling for Multimodal Latent Reasoning

Yudong Han; Yong Wang; Zaiquan Yang; Zhen Qu; Liyuan Pan; Xiangxiang Chu

cs.CVApr 12, 2026

Visual Enhanced Depth Scaling for Multimodal Latent Reasoning

Authors: Yudong Han, Yong Wang, Zaiquan Yang, Zhen Qu, Liyuan Pan, Xiangxiang Chu

Organizations: Beijing Institute of Technology · AMAP, Alibaba Group · City University of Hong Kong · Institute of Automation, Chinese Academy of Sciences ·

Abstract

Multimodal latent reasoning has emerged as a promising paradigm that replaces explicit Chain-of-Thought (CoT) decoding with implicit feature propagation, simultaneously enhancing representation informativeness and reducing inference latency. By analyzing token-level gradient dynamics during latent training, we reveal two critical observations: (1) visual tokens exhibit significantly smaller gradient norms than their textual counterparts due to inherent language bias, resulting in systematic visual under-optimization; and (2) semantically simple tokens converge rapidly, whereas complex tokens exhibit persistent gradient instability constrained by fixed architectural depths. To address these limitations, we propose a visual replay module and routing depth scaling to collaboratively enhance visual perception and refine complicated latents for deeper contextual reasoning. The former module leverages causal self-attention to estimate token saliency, reinforcing fine-grained grounding through spatially-coherent constraints. Complementarily, the latter mechanism adaptively allocates additional reasoning steps to complex tokens, enabling deeper contextual refinement. Guided by a curriculum strategy that progressively internalizes explicit CoT into compact latent representations, our framework achieves state-of-the-art performance across diverse benchmarks while delivering substantial inference speedups over explicit CoT baselines.

Figures & tables

Panel of Figure 1 — Figure 1: Panel (a) depicts the token-wise Frobenius norm of gradients across different layers throughout the entire training process, revealing a clear gradient imbalance between visual tokens and text tokens/latent token. Panels (b) and (c) further reveal distinct evolution patterns of the Nuclear norm in the QKVO projection matrix across layers and training epochs: gradients for easy samples decay rapidly and converge smoothly, whereas hard samples maintain elevated gradient norms with persistent oscillations. Here, $\mathbf{W}_{A}$ and $\mathbf{W}_{B}$ denote the low-rank matrix in LoRA [ 19 ] .

Figure 2: Left Panel: Schematic illustration of our framework. Input images are encoded into visual tokens by a pretrained visual encoder and projected into a text-centric semantic space aligned with the LLM, while questions are tokenized by the text tokenizer. Our method enhances a standard latent MLLM with two synergistic components during the implicit reasoning phase: Spatially-Coherent Finer Visual Replay (SCF-VR) and Routing Depth Scaling (RDS). The overall objective is jointly optimized via standard cross-entropy loss and self-distillation loss. Right Panel: Detailed architecture of the SCF-VR module and token-wise depth scaling mechanism. TokenSR denotes the token super-resolution module, and Pool is the average pooling.

		M 3 CoT			ScienceQA			GQA
Method	Model	Acc.(%) $\uparrow$	# AR Steps $\downarrow$	Avg. Time $\downarrow$	Acc.(%) $\uparrow$	# AR Steps $\downarrow$	Avg. Time $\downarrow$	Acc.(%) $\uparrow$	# AR Steps $\downarrow$	Avg. Time $\downarrow$
No-CoT		45.4	–	–	64.4	–	–	–	–	–
Multimodal CoT [ 80 ]		42.5	106.3	3.10	58.3	83.9	2.44	–	–	–
CCoT [ 39 ]		44.1	177.2	5.31	63.8	164.0	5.23	51.2	76.4	7.21
ICoT [ 13 ]		46.0	96.5	2.86	65.4	77.4	2.28	–	–	–
SCAFFOLD [ 26 ]		44.9	170.8	5.14	62.5	162.3	4.91	48.7	72.8	6.72

Table 1: Comparison of various multimodal reasoning baselines across three benchmarks. We selected three datasets featuring detailed reasoning chains for training and evaluate on their test split, respectively. Three metrics are reported: Accuracy (%), Average number of Autoregressive Steps (# AR steps), and Average Generation Time (AVG. Time). † denotes the reimplementation for the methods with the same configuration with ours. No-CoT notes that directly predicts answers without generating intermediate steps.

Model	Data Size	Vision-centric Reasoning				Compositional		Mathematics Reasoning
Model	Data Size	MMVP	SeedBench-2-Plus	HallusionBench	HRBench	BLINK	MMStar	MathVista	MathVision	MM-Math
Zero-Shot VLMs
GPT-4o [ 41 ]	–	68.70	72.00	–	–	68.00	64.70	63.80	30.39	31.80
Qwen2.5-VL-7B [ 2 ]	–	65.67	65.31	56.57	68.25	53.60	59.70	68.20	25.60	37.50
LLaVA-OneVision [ 29 ]	9M	74.00	61.22	51.10	63.00	49.34	59.13	58.60	-	-
InternVL3.5-8B [ 60 ]	70K	57.67	69.78	56.15	59.38	54.81	53.33	71.60	28.30	-

Table 2: Performance comparison of our method against baselines across four paradigms. Benchmarks are categorized into three domains: Visual Perception , Compositional Reasoning , and Mathematical Reasoning . Among latent reasoning approaches, the best results are highlighted in bold , and the second-best are underlined . Our method is built upon the Qwen2.5-VL-7B backbone.

Table 5

Panel of Figure 3 — Figure 3: Details of sampled training distribution. Panel (a) depicts the sample distribution across different CoT lengths. The distribution exhibits a boundary at 125 tokens, with sample counts gradually decreasing toward both shorter and longer extremes. Panel (b) illustrates the distribution of reasoning steps in the original data. Specifically, we segment reasoning steps using \n\n as delimiters; during training, we retain up to 4 delimiters to evenly partition sequence length, while samples containing fewer than 4 delimiters preserve their original segmentation. Panel (c) summarizes the topic-wise distribution of samples in the dataset.

		M 3 CoT (Acc.%)
Method	Grid Size	$W=2$	$W=3$	$W=5$
Qwen2-VL-2B	10 $\times$ 10	50.8	51.2	50.3
Qwen2-VL-7B		65.7	66.4	65.2
Qwen2.5-VL-3B		65.0	65.3	64.5
Qwen2.5-VL-7B		73.4	73.9	72.1

Table 5: Performance comparison of different sub-grid window sizes on the M 3 CoT benchmark. Here,

W

denotes the sub-grid window size;

W=10

is the window size of extracted visual features. We set

\lambda=1.0

for the 2B/3B models and

\lambda=0.2

for the 7B model.

Figure 4: Sensitivity analysis of $\lambda$ .

Figure 5: Visualization of learned router scores and selected top- $K$ tokens for depth scaling, revealing which textual tokens, CoT latent tokens, and visual regions exhibit greater optimization difficulty.

Figure 6: Visualization of cropped region in each latent reasoning step.

Latent Visual Reasoning

Latent visual reasoning studies how multimodal models use compact internal visual states to solve perception, spatial, and temporal tasks.

See all 90 papers

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
May 4, 2026Xin Zhang, Qiqi Tao, Jiawei Du +2Inference-Time Optimization Test-Time Optimization
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design +1
Multimodal Latent Reasoning via Hierarchical Visual Cues Injection
Feb 5, 2026Yiming Zhang, Qiangyu Yan, Borui Jiang +1VLM Reasoning Multimodal Large Language Models
Nanyang Technological University · Huawei Noah’s Ark Lab
Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Oct 8, 2026Yicheng Xue, Han Wu, Jufeng Yang +4VLM Reasoning Efficient Multimodal Inference
City University of Hong Kong · Zhejiang University · Peking University +2

May 4, 2026J/K move · Enter open · S save

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs

Xin Zhang, Qiqi Tao, Jiawei Du, Moyun Liu+1

Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Singapore University of Technology and Design+1

Inference-Time Optimization Test-Time Optimization Multimodal Large Language Models Latent Visual Reasoning VLM Reasoning Continuous Latent Reasoning+4

Open paper AskPDF arXiv

Continuous latent-space reasoning offers a compact alternative to textual chain-of-thought for multimodal models, enabling high-dimensional visual evidence to be integrated without explicit reasoning tokens. However, we identify a previously overlooked optimization pathology in existing latent visual reasoning methods: although visual latents become semantically enriched during training, their contribution to final answer prediction is systematically suppressed. Within the shared parameter space, the autoregressive objective favors shortcut reliance on direct visual input, driving latent tokens toward transition-like states rather than informative reasoning content. We term this phenomenon Silenced Visual Latents. To address it, we disentangle the two conflicting objectives by directly optimizing the latent reasoning at inference time, keeping backbone parameters frozen. In Stage I, visual latents are warmed up via query-guided contrastive latent--visual alignment, improving semantic quality while preventing latent collapse. In Stage II, the latent reasoning is further optimized via a confidence-progression reward, which incentivizes predicted token distributions along the latent span to become progressively more concentrated, routing predictions through the latent reasoning rather than bypassing it. Experiments across eight benchmarks and four model backbones show that inference-time latent optimization, without any parameter updates, effectively unleashes the suppressed reasoning capacity of visual latents.

Abstract

Latent Visual Reasoning

Explore similar work

Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs

Multimodal Latent Reasoning via Hierarchical Visual Cues Injection

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning