Multimodal Latent Reasoning via Hierarchical Visual Cues Injection

Yiming Zhang; Qiangyu Yan; Borui Jiang; Kai Han

cs.CVFeb 5, 2026

Multimodal Latent Reasoning via Hierarchical Visual Cues Injection

Authors: Yiming Zhang, Qiangyu Yan, Borui Jiang, Kai Han

Organizations: Nanyang Technological University · Huawei Noah’s Ark Lab

Abstract

The advancement of multimodal large language models (MLLMs) has enabled impressive perception capabilities. However, their reasoning process often remains a "fast thinking" paradigm, reliant on end-to-end generation or explicit, language-centric chains of thought (CoT), which can be inefficient, verbose, and prone to hallucination. This work posits that robust reasoning should evolve within a latent space, integrating multimodal signals seamlessly. We propose multimodal latent reasoning via HIerarchical Visual cuEs injection (HIVE), a novel framework that instills deliberate, "slow thinking" without depending on superficial textual rationales. Our method recursively extends transformer blocks, creating an internal loop for iterative reasoning refinement. Crucially, it injectively grounds this process with hierarchical visual cues from global scene context to fine-grained regional details directly into the model's latent representations. This enables the model to perform grounded, multi-step inference entirely in the aligned latent space. Extensive evaluations demonstrate that test-time scaling is effective when incorporating vision knowledge, and that integrating hierarchical information significantly enhances the model's understanding of complex scenes.

Figures & tables

Figure 1: Visualization of traditional MLLMs, visual features extracted from a vision tower are projected into the language space and directly concatenated with text tokens. This combined sequence is then fed into a stack of transformer decoder blocks. HIVE is built upon Huginn, a recursive architecture that iteratively processes token representations through a unified set of layers to enhance feature depth. We have extended this by incorporating the visual modality and, for the first time, introducing hierarchical visual information into latent space reasoning.

Figure 2: Our framework incorporates a pre-trained vision encoder, with a group of lightweight patch merger that maps visual features into the LLM embedding space. During multimodal alignment, the [CLS] token is removed. represents Embedding , Recurrent , Head blocks respectively.

Method	Visual	Text	Hierarchical	CoT Data Requirement
Training-induced Recurrence
Coconut	✗	✓	-	High
Heima	✓	✓	✗	High
Loop Transformer Recurrence
Huginn	✗	✓	-	Low
HIVE (Ours)	✓	✓	✓	Low

Table 1: Comparison of Latent Space Reasoning Approaches. Coconut/Heima progressively abstract CoT text into learnable tokens without altering model architecture; Heima is the first to include multimodal input. Huginn/ HIVE utilize a Loop Transformer architecture to reason via iterations, reducing dependency on CoT tokens. HIVE further introduces hierarchical visual features.

	Gemma3	LLaMA2	Phi-3-mini	Huginn
Param	4B	7B	3.8B	3.5B
Train Tokens	4T	2.0T	3.3T	0.8T
GSM8K	38.4	-	-	28.20
GSM8K-CoT (8-shot)	-	-	82.5	34.57
HumanEval	36.0	29.9	58.5	23.17

Table 2: Comparison of Various Backbone LMMs, including Gemma3 ( Team, 2025 ) , LLaMA2 ( Touvron et al., 2023 ) , Phi-3-mini ( Abdin et al., 2024 ) and Huginn ( Geiping et al., 2025 ) . We detail model sizes, training token counts, and benchmark performances. The results indicate that the inherent language capabilities of our backbone LLM are constrained, which may, in turn, impact the overall performance of the resulting MLLMs. The missing entries are unavailable results.

Parameters	Stage 1	Stage 2	Stage 3
Learning Rate	$1\times 10^{-3}$	$1\times 10^{-5}$	$1\times 10^{-6}$
Max Dynamic Patches (#)	4	2	2
Max Tokens	1536	2048	2048

Table 3: Training configurations across different stages.

		Gemma3-4B-PT	MobileVLM V2 7B	Bunny-v1.1-4B	Imp-4B	Emu3	HIVE
Backbone		Gemma3	Vicuna-7B-v1.5	Phi-3-3.8B	Phi-3-3.8B	-	Huginn
Param.		4B	7B	4B	4B	8B	4B
Train Data		4T*	3.6M	2.7M	1.6M	-	6.5M
General VQA	MMB	-	69.2	74.2	73.3	58.5	69.6
	SQA ${}_{\text{img}}$	-	74.8	78.3	78.3	89.2	91.6
	SEED ${}_{\text{img}}$	-	-	72.5	-	68.2	70.5

Table 4: Main performance comparison with MLLMs. We report the LLM backbone, model size, and training data scales (4T* denotes 4T tokens, otherwise samples). Note that backbones are detailed in Table 2 . MMB: MMBench (En) dev split; SQA

{}_{\text{img}}

: ScienceQA-Img; SEED

{}_{\text{img}}

: SEED-Bench-Img; RWQA: RealWorldQA; TVQA: TextVQA; CQA: ChartQA; DVQA: DocVQA; POPE; Missing entries indicate unavailable results.

Benchmark	Baseline (r=1)	Ours w/o Hier (r=32)	Ours w/ Hier (r=32)
MMStar	33.28	48.44	49.79
SEED ${}_{\text{img}}$	42.37	70.46	70.48
MMB ${}_{\text{dev}}$	21.74	68.04	69.59
RWQA	41.44	57.52	57.91
SQA ${}_{\text{img}}$	60.09	89.39	91.57
Avg.	39.78	66.77	67.87

Table 5: Results on benchmarks for three variants. Bold text indicates the best performance among these models. MMB: MMBench (En) dev split; SQA

{}_{\text{img}}

: ScienceQA-Img; SEED

{}_{\text{img}}

: SEED-Bench-Img; RWQA: RealWorldQA; TVQA: TextVQA; DVQA: DocVQA; MathV: MathVista; POPE

Figure 3: Building upon Huginn, we integrate a Vision Transformer (ViT) and propose a hierarchical latent-space reasoning framework. Specifically, we argue that latent-space reasoning with visual information should be hierarchical rather than merely iterative. The figure shows our comparison results on ScienceQA

{}_{\text{img}}

Figure 4: MMBench detailed results. LR denotes logic reasoning. FC denotes fine-grained perception (cross-instance). AR denotes attribute reasoning. RR denotes relation reasoning. FI denotes fine-grained perception (instance-level). CP denotes coarse perception.

Figure 5: Distribution of inference steps for the first token generation across multiple-choice benchmarks. We evaluate the impact of hierarchical cue injection on the recurrent steps. The results demonstrate that incorporating these cues causes a distinct leftward shift in the distribution, indicating a reduction in computation during inference.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The details of our training datasets.

Figure 7: A case from ScienceQA, where the question has been modified as a QA.

Figure 8: The visualization shows the evolution of latent states as a function of token position (vertical axis) and iteration depth (horizontal axis) in the model variant (r=32, w/o Hier,). Each cell represents the distance between a given iterate and its corresponding steady state, approximated at $r=32$ .

Figure 9: The visualization shows the evolution of latent states as a function of token position (vertical axis) and iteration depth (horizontal axis) in the HIVE . Each cell represents the distance between a given iterate and its corresponding steady state, approximated at $r=32$ .

Figure 10: Visualization of hidden state trajectories in the model variant (r=32 w/o Hier.) during inference.

Figure 11: Visualization of hidden state trajectories in HIVE during inference.

Figure 12:

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
Jun 13, 2026David Huang, Lianlei ShanLatent World Models Multimodal Reasoning
University of Toronto · Tsinghua University
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
Feb 25, 2026Yongchang Zhang, Oliver Ma, Tianyi Liu +2Multimodal Reasoning VLM Reasoning
Southeast University · AIIA, Ministry of Education, China · University of Oxford
LanteRn: Latent Visual Structured Reasoning
Mar 26, 2026André G. Viveiros, Nuno Gonçalves, Matthias Lindemann +1Latent Visual Reasoning VLM Reasoning
Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa · Carnegie Mellon University

Jun 13, 2026J/K move · Enter open · S save

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

David Huang, Lianlei Shan

University of Toronto · Tsinghua University

Latent World Models Multimodal Reasoning Latent Visual Reasoning Efficient Multimodal Inference Multimodal Understanding RL for VLMs+1

Open paper AskPDF arXiv

Reasoning capabilities of multimodal large language models (MLLMs) have improved considerably in recent years. Existing approaches typically rely on explicit chain-of-thought or continuous latent-space trajectories to enhance multi-step reasoning. However, these methods generally assume that an input admits a single latent interpretation and unfold reasoning along a fixed path or under a uniform computation budget. In real-world multimodal settings, visual observations are often subject to occlusion, blur, viewpoint variation, or semantic ambiguity, giving rise to multiple plausible interpretations. A uniform reasoning strategy not only limits the model's ability to explore multiple hypotheses but also incurs high memory usage and rollout cost. We present DLWM (Diverse Latent World Models), a multimodal reasoning framework that combines latent-space reasoning with reinforcement learning. First, we construct a set of diverse latent world hypotheses in continuous latent space, each capturing a different plausible interpretation of the visual input, and unfold latent reasoning independently on each hypothesis. An orthogonality-based diversity regularizer explicitly prevents hypothesis collapse. Second, we formulate the latent reasoning process as a resource-constrained sequential decision problem and introduce a resource-aware reinforcement learning policy that adaptively allocates computation across hypotheses, dynamically deciding whether to expand, terminate, or merge reasoning paths, thereby substantially reducing memory footprint and improving rollout efficiency. Experiments on multiple multimodal reasoning benchmarks demonstrate that DLWM outperforms existing methods by 2-5 points in accuracy while reducing memory usage by 24%.

Abstract

VLM Reasoning

Explore similar work

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

LanteRn: Latent Visual Structured Reasoning