CrystaL: Spontaneous Emergence of Visual Latents in MLLMs

Yang Zhang; Danyang Li; Yuxuan Li; Xin Zhang; Tianyu Xie; Mingming Cheng; Xiang Li

cs.CVFeb 24, 2026

CrystaL: Spontaneous Emergence of Visual Latents in MLLMs

Authors: Yang Zhang, Danyang Li, Yuxuan Li, Xin Zhang, Tianyu Xie, Mingming Cheng, Xiang Li

Organizations: VCIP, School of Computer Science, Nankai University · Xiamen University · Nankai International Advanced Research Institute (Shenzhen Futian)

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable performance by integrating powerful language backbones with large-scale visual encoders. Among these, latent Chain-of-Thought (CoT) methods enable implicit reasoning in continuous hidden states, facilitating seamless vision-language integration and faster inference. However, existing heuristically predefined supervision signals in latent CoT provide limited guidance for preserving critical visual information in intermediate latent states. To address this limitation, we propose CrystaL (Crystallized Latent Reasoning), a single-stage framework with two paths to process intact and corrupted images, respectively. By explicitly aligning the attention patterns and prediction distributions across the two paths, CrystaL crystallizes latent representations into task-relevant visual semantics, without relying on auxiliary annotations or external modules. Extensive experiments on perception-intensive benchmarks demonstrate that CrystaL consistently outperforms state-of-the-art baselines, achieving substantial gains in fine-grained visual understanding while maintaining robust reasoning capabilities.

Figures & tables

Figure 1: Previous Paradigms vs. Our Paradigm (CrystaL). (a) describes the paradigm of supervising visual latent tokens via predefined features from specific models, such as SAM ( Kirillov et al., 2023 ) , DINO ( Caron et al., 2021 ) . (b) denotes the process of modifying original image to guide the reasoning steps. (c) While other methods use auxiliary modules or data to train the visual latent token, CrystaL can do it by self-supervising in a single-stage.

Desired Properties	Aurora	SKILA	LVR	LIVR	CoVT	Ours
No extra module	✗	✗	✓	✓	✗	✓
No extra images needed for training	✓	✗	✓	✓	✓	✓
One-stage training	✗	✓	✗	✗	✗	✓
Reason in the continuous space	✗	✓	✓	✓	✓	✓

Table 1: Key properties of visual latent reasoning methods. In contrast to prior works like Aurora ( Bigverdi et al., 2025 ) , SKILA ( Tong et al., 2025 ) , LVR ( Li et al., 2025a ) , LIVR, and CoVT, our CrystaL is the only framework that simultaneously satisfies all criteria. Specifically, it eliminates the need for auxiliary modules and supplementary image labels while enabling efficient one-stage training. Symbols in green denote desired properties, while those in red denote undesired ones.

Figure 2: An overview of CrystaL. (a) Starting by corrupting the raw image, we construct two types of visual tokens from vision encoder. Then both the $S_{int}$ and $S_{cor}$ in Sec. 3.3 are fed into the model to compute the $\mathbf{P}_{int}$ and $\mathbf{P}_{cor}$ . But the hidden states of corrput forward (dashed arrow) indeed come from the the intact forward (solid arrow). For the objective function, we adopt a combination of cross entropy loss and alignment loss, the attention map alignment is illustrated in (b).

Figure 3: Detailed illustration of visual latent token copying. For the forward process, the autoregressive hidden states of visual latent tokens in the corrupted path is copied from the intact path.

	CVBench		HRBench
Method	2D	3D	4K	8K	BLINK	RWQA	V*	POPE	Average
Closed-source Methods
Claude-4-Sonnet	-	-	32.3	22.7	39.6	63.7	15.2	-	-
GPT-4o	-	-	50.6	46.7	63.0	69.7	42.9	-	-
Open-source Methods
Qwen2.5-VL-7B	75.0	73.3	68.6	64.9	55.7	68.6	76.4	86.5	71.1

Table 2: Comparison of model performance across diverse visual benchmarks. The results highlight CrystaL’s superior capabilities comparing with baselines including Qwen2.5-VL-7B ( Bai et al., 2025 ) .

Figure 4: Comparison of performance and training data size. Our method achieves superior performance while utilizing significantly fewer samples than baselines, demonstrating exceptional data efficiency.

Quantity	$\text{CV}_{2D}$	$\text{CV}_{3D}$	BLINK	$\text{HR}_{4K}$	$\text{HR}_{8K}$	RWQA	Avg
Baseline
Qwen2.5-VL	75.0	73.3	55.7	68.6	64.9	68.6	67.7
Identical latent token
4 tokens	74.7	74.7	52.9	73.8	71.0	68.0	69.2
8 tokens	75.7	83.7	54.4	74.4	72.0	69.3	71.6
16 tokens	72.3	80.6	52.3	71.5	68.1	69.7	69.1

Table 3: Ablation on the quantity and content of latent reasoning tokens. The upper section denotes configurations using same tokens, while the lower section utilizes diverse tokens.

Strategy	$\text{CV}_{2D}$	$\text{CV}_{3D}$	BLINK	$\text{HR}_{4K}$	$\text{HR}_{8K}$	RWQA	Avg
How does the intact path assist corrupted path?
Qwen2.5-VL	75.0	73.3	55.7	68.6	64.9	68.6	67.7
+ Cor. CE	74.7	83.3	54.6	73.0	70.3	69.9	71.0
+ Cor. CE + KL	75.2	83.6	54.1	75.0	71.1	71.1	71.7
+ Cor. CE + Att. Align	76.2	84.0	57.1	72.6	70.4	70.3	71.8
+ All	76.6	84.4	55.9	73.4	71.1	70.6	72.0

Table 4: Ablation on alignment strategy. Both the

\mathcal{L}_{kl}

and

\mathcal{L}_{attn}

play a critical role in our method. And the type of

\mathcal{L}_{attn}

is also important. Here, “Cor.” means “corrupted”.

Strategy	$\text{CV}_{2D}$	$\text{CV}_{3D}$	BLINK	$\text{HR}_{4K}$	$\text{HR}_{8K}$	RWQA	Avg
Gaussian Blur	76.6	84.4	55.9	73.4	71.1	70.6	72.0
Random Mask	76.3	84.7	55.2	71.9	69.4	68.4	71.0
Jigsaw Shuffling	73.6	82.5	54.0	71.4	69.0	65.2	69.3
Gaussian Noise	74.9	81.8	54.1	72.0	69.8	69.9	70.4
Colour Distortion	74.8	80.3	54.6	72.3	70.4	70.6	70.5

Table 5: Ablation on corruption strategy. The Gaussian blur behaves best in all the corruption strategies, while random mask is also useful for learning visual latents.

Figure 5: Direct Finetune vs CrystaL. CrystaL outperforms direct finetuning’capabilities across all the tasks.

Figure 6: Qualitative comparison on CVBench-2D. We illustrate the responses from CoVT and CrystaL to a question regarding spatial-relational reasoning between objects.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Results on relation and distance.

Figure 9: Gaussian blur variations. We illustrate the multi-scale blurring responses applied to the input samples, showing the original image and three levels of Gaussian smoothing ( $r=2,5,10$ ).

Figure 10: Jigsaw. We apply Jigsaw augmentation to the input samples to challenge the model’s ability to capture global structural consistency. Three levels of shuffling ratios ( $\rho\in\{20\%,50\%,80\%\}$ ) are illustrated to demonstrate increasing spatial complexity.

Figure 11: Colour Distort. We illustrate the visual variations generated by the color jittering module by modulating brightness, contrast, saturation, and hue with increasing intensities ( $s\in\{0.5,1.0,2.0\}$ ).

Figure 12: Gaussian Noise. We illustrate the original image and its corrupted versions subjected to additive Gaussian noise with varying standard deviations ( $\sigma\in\{15,30,50\}$ ).

Figure 13: Random mask. We demonstrate the original image alongside three variants of Random Erasing at different scales ( $s\in\{0.2,0.5,0.8\}$ ).

Figure 14: Attention differences. The intact path attends to detroit while the corrupted path attends to all the labels.

Figure 15: Attention differences. The intact path attends to random positions while the corrupted path attends to the text behind the man.

Figure 16: Attention differences. The intact path attends to random positions while the corrupted path attends to the man.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning
May 8, 2026Jin Cui, Xinyue Long, Xunyong Zhang +5Visual Spatial Reasoning Visual Reasoning
1State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Software Engineering, State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization
Apr 22, 2026Tao Cheng, Shi-Zhe Chen, Hao Zhang +3Multimodal Large Language Models Multimodal Reasoning
Tencent PCG, China · Tencent CSIG, China
ProLaViT: Learning Progressive Latent Visual Thoughts in Structured Latent Space
Jul 3, 2026Peiming Li, Yifan Wang, Xiaotian Zhang +4Multimodal Large Language Models Latent Visual Reasoning
Basic Algorithm Center, PCG, Tencent · Zhejiang University · Peking University

May 8, 2026J/K move · Enter open · S save

Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning

Jin Cui, Xinyue Long, Xunyong Zhang, Yadong Zhang+4

1State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Software Engineering, State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

Visual Spatial Reasoning Visual Reasoning Multimodal Large Language Models Latent Visual Reasoning Vision-Language Grounding VLM Reasoning+2

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have made remarkable progress on vision-language reasoning, yet most methods still compress visual evidence into discrete textual thoughts, creating an information bottleneck for fine-grained perception. Recent latent visual reasoning methods attempt to reason in continuous hidden states, but we find that they suffer from insufficient manifold compatibility: latent trajectories drift away from pretrained reasoning circuits, collapse into instance-agnostic patterns, and are often bypassed during answer generation. To address these issues, we propose RIS (Retrieve, Integrate, and Synthesize), a spatial-semantic grounded framework that develops latent reasoning as a compatible extension of pretrained MLLM computation. We first construct a step-wise grounded reasoning dataset with bounding boxes and region-specific semantic descriptions. Built on this supervision, RIS anchors latent tokens to both spatial and semantic evidence, enforces their causal role through a progressive attention bottleneck, and introduces short language transition tokens to bridge synthesized latent states back to vocabulary-aligned decoding. Experiments on V*, HRBench4K, HRBench8K, MMVP, and BLINK show consistent improvements over closed/open-source and latent reasoning baselines. Further analyses demonstrate that RIS learns diverse, interpretable, and progressively integrated latent trajectories, offering a practical path toward faithful internal visual reasoning in MLLMs.

Abstract

Vision-Language Models

Explore similar work

Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning

HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization

ProLaViT: Learning Progressive Latent Visual Thoughts in Structured Latent Space