The advancement of multimodal large language models (MLLMs) has enabled impressive perception capabilities. However, their reasoning process often remains a "fast thinking" paradigm, reliant on end-to-end generation or explicit, language-centric chains of thought (CoT), which can be inefficient, verbose, and prone to hallucination. This work posits that robust reasoning should evolve within a latent space, integrating multimodal signals seamlessly. We propose multimodal latent reasoning via HIerarchical Visual cuEs injection (HIVE), a novel framework that instills deliberate, "slow thinking" without depending on superficial textual rationales. Our method recursively extends transformer blocks, creating an internal loop for iterative reasoning refinement. Crucially, it injectively grounds this process with hierarchical visual cues from global scene context to fine-grained regional details directly into the model's latent representations. This enables the model to perform grounded, multi-step inference entirely in the aligned latent space. Extensive evaluations demonstrate that test-time scaling is effective when incorporating vision knowledge, and that integrating hierarchical information significantly enhances the model's understanding of complex scenes.
Figures & tables
Figure 1: Visualization of traditional MLLMs, visual features extracted from a vision tower are projected into the language space and directly concatenated with text tokens. This combined sequence is then fed into a stack of transformer decoder blocks. HIVE is built upon Huginn, a recursive architecture that iteratively processes token representations through a unified set of layers to enhance feature depth. We have extended this by incorporating the visual modality and, for the first time, introducing hierarchical visual information into latent space reasoning.
Figure 2: Our framework incorporates a pre-trained vision encoder, with a group of lightweight patch merger that maps visual features into the LLM embedding space. During multimodal alignment, the [CLS] token is removed. represents Embedding , Recurrent , Head blocks respectively.
Method
Visual
Text
Hierarchical
CoT Data Requirement
Training-induced Recurrence
Coconut
✗
✓
-
High
Heima
✓
✓
✗
High
Loop Transformer Recurrence
Huginn
✗
✓
-
Low
HIVE (Ours)
✓
✓
✓
Low
Table 1: Comparison of Latent Space Reasoning Approaches. Coconut/Heima progressively abstract CoT text into learnable tokens without altering model architecture; Heima is the first to include multimodal input. Huginn/ HIVE utilize a Loop Transformer architecture to reason via iterations, reducing dependency on CoT tokens. HIVE further introduces hierarchical visual features.
Gemma3
LLaMA2
Phi-3-mini
Huginn
Param
4B
7B
3.8B
3.5B
Train Tokens
4T
2.0T
3.3T
0.8T
GSM8K
38.4
-
-
28.20
GSM8K-CoT (8-shot)
-
-
82.5
34.57
HumanEval
36.0
29.9
58.5
23.17
Table 2: Comparison of Various Backbone LMMs, including Gemma3 ( Team, 2025 ) , LLaMA2 ( Touvron et al., 2023 ) , Phi-3-mini ( Abdin et al., 2024 ) and Huginn ( Geiping et al., 2025 ) . We detail model sizes, training token counts, and benchmark performances. The results indicate that the inherent language capabilities of our backbone LLM are constrained, which may, in turn, impact the overall performance of the resulting MLLMs. The missing entries are unavailable results.
Parameters
Stage 1
Stage 2
Stage 3
Learning Rate
1×10−3
1×10−5
1×10−6
Max Dynamic Patches (#)
4
2
2
Max Tokens
1536
2048
2048
Table 3: Training configurations across different stages.
Gemma3-4B-PT
MobileVLM V2 7B
Bunny-v1.1-4B
Imp-4B
Emu3
HIVE
Backbone
Gemma3
Vicuna-7B-v1.5
Phi-3-3.8B
Phi-3-3.8B
-
Huginn
Param.
4B
7B
4B
4B
8B
4B
Train Data
4T*
3.6M
2.7M
1.6M
-
6.5M
General VQA
MMB
-
69.2
74.2
73.3
58.5
69.6
SQA img
-
74.8
78.3
78.3
89.2
91.6
SEED img
-
-
72.5
-
68.2
70.5
Table 4: Main performance comparison with MLLMs. We report the LLM backbone, model size, and training data scales (4T* denotes 4T tokens, otherwise samples). Note that backbones are detailed in Table 2 . MMB: MMBench (En) dev split; SQA img : ScienceQA-Img; SEED img : SEED-Bench-Img; RWQA: RealWorldQA; TVQA: TextVQA; CQA: ChartQA; DVQA: DocVQA; POPE; Missing entries indicate unavailable results.
Benchmark
Baseline (r=1)
Ours w/o Hier (r=32)
Ours w/ Hier (r=32)
MMStar
33.28
48.44
49.79
SEED img
42.37
70.46
70.48
MMB dev
21.74
68.04
69.59
RWQA
41.44
57.52
57.91
SQA img
60.09
89.39
91.57
Avg.
39.78
66.77
67.87
Table 5: Results on benchmarks for three variants. Bold text indicates the best performance among these models. MMB: MMBench (En) dev split; SQA img : ScienceQA-Img; SEED img : SEED-Bench-Img; RWQA: RealWorldQA; TVQA: TextVQA; DVQA: DocVQA; MathV: MathVista; POPE
Figure 3: Building upon Huginn, we integrate a Vision Transformer (ViT) and propose a hierarchical latent-space reasoning framework. Specifically, we argue that latent-space reasoning with visual information should be hierarchical rather than merely iterative. The figure shows our comparison results on ScienceQA img .
Figure 4: MMBench detailed results. LR denotes logic reasoning. FC denotes fine-grained perception (cross-instance). AR denotes attribute reasoning. RR denotes relation reasoning. FI denotes fine-grained perception (instance-level). CP denotes coarse perception.
Figure 5: Distribution of inference steps for the first token generation across multiple-choice benchmarks. We evaluate the impact of hierarchical cue injection on the recurrent steps. The results demonstrate that incorporating these cues causes a distinct leftward shift in the distribution, indicating a reduction in computation during inference.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The details of our training datasets.
Figure 7: A case from ScienceQA, where the question has been modified as a QA.
Figure 8: The visualization shows the evolution of latent states as a function of token position (vertical axis) and iteration depth (horizontal axis) in the model variant (r=32, w/o Hier,). Each cell represents the distance between a given iterate and its corresponding steady state, approximated at r=32 .
Figure 9: The visualization shows the evolution of latent states as a function of token position (vertical axis) and iteration depth (horizontal axis) in the HIVE . Each cell represents the distance between a given iterate and its corresponding steady state, approximated at r=32 .
Figure 10: Visualization of hidden state trajectories in the model variant (r=32 w/o Hier.) during inference.
Figure 11: Visualization of hidden state trajectories in HIVE during inference.
Reasoning capabilities of multimodal large language models (MLLMs) have improved considerably in recent years. Existing approaches typically rely on explicit chain-of-thought or continuous latent-space trajectories to enhance multi-step reasoning. However, these methods generally assume that an input admits a single latent interpretation and unfold reasoning along a fixed path or under a uniform computation budget. In real-world multimodal settings, visual observations are often subject to occlusion, blur, viewpoint variation, or semantic ambiguity, giving rise to multiple plausible interpretations. A uniform reasoning strategy not only limits the model's ability to explore multiple hypotheses but also incurs high memory usage and rollout cost. We present DLWM (Diverse Latent World Models), a multimodal reasoning framework that combines latent-space reasoning with reinforcement learning. First, we construct a set of diverse latent world hypotheses in continuous latent space, each capturing a different plausible interpretation of the visual input, and unfold latent reasoning independently on each hypothesis. An orthogonality-based diversity regularizer explicitly prevents hypothesis collapse. Second, we formulate the latent reasoning process as a resource-constrained sequential decision problem and introduce a resource-aware reinforcement learning policy that adaptively allocates computation across hypotheses, dynamically deciding whether to expand, terminate, or merge reasoning paths, thereby substantially reducing memory footprint and improving rollout efficiency. Experiments on multiple multimodal reasoning benchmarks demonstrate that DLWM outperforms existing methods by 2-5 points in accuracy while reducing memory usage by 24%.
Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.
Yongchang Zhang, Oliver Ma, Tianyi Liu +2
Southeast University · AIIA, Ministry of Education, China · University of Oxford
While language reasoning models excel in many tasks, visual reasoning remains challenging for current large multimodal models (LMMs). As a result, most LMMs default to verbalizing perceptual content into text, a strong limitation for tasks requiring fine-grained spatial and visual understanding. While recent approaches take steps toward thinking with images by invoking tools or generating intermediate images, they either rely on external modules, or incur unnecessary computation by reasoning directly in pixel space. In this paper, we introduce LanteRn, a framework that enables LMMs to interleave language with compact latent visual representations, allowing visual reasoning to occur directly in latent space. LanteRn augments a vision-language transformer with the ability to generate and attend to continuous visual thought embeddings during inference. We train the model in two stages: supervised fine-tuning to ground visual features in latent states, followed by reinforcement learning to align latent reasoning with task-level utility. We evaluate LanteRn on three perception-centric benchmarks (VisCoT, V*, and Blink), observing consistent improvements in visual grounding and fine-grained reasoning. These results suggest that internal latent representations provide a promising direction for more efficient multimodal reasoning.
André G. Viveiros, Nuno Gonçalves, Matthias Lindemann +1
Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa · Carnegie Mellon University