LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model

Jiachun Jin; Zetong Zhou; Xiao Yang; Hao Zhang; Pengfei Liu; Jun Zhu; Zhijie Deng

cs.CVApr 2, 2026

LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model

Authors: Jiachun Jin, Zetong Zhou, Xiao Yang, Hao Zhang, Pengfei Liu, Jun Zhu, Zhijie Deng

Organizations: Shanghai Jiao Tong University · Tsinghua University · UCSD

Abstract

Unified models (UMs) hold promise for their ability to understand and generate content across heterogeneous modalities. Compared to merely generating visual content, the use of UMs for interleaved cross-modal reasoning is more promising and valuable, e.g., for solving understanding problems that require dense visual thinking, improving visual generation through self-reflection, or modeling visual dynamics of the physical world guided by stepwise action interventions. However, existing UMs necessitate pixel decoding as a bridge due to their disjoint visual representations for understanding and generation, which is both ineffective and inefficient. In this paper, we introduce LatentUM, a novel unified model that represents all modalities within a shared semantic latent space, eliminating the need for pixel-space mediation between visual understanding and generation. This design naturally enables flexible interleaved cross-modal reasoning and generation. Beyond improved computational efficiency, the shared representation substantially alleviates codec bias and strengthens cross-modal alignment, allowing LatentUM to achieve state-of-the-art performance on the Visual Spatial Planning benchmark, push the limits of visual generation through self-reflection, and support world modeling by predicting future visual states within the shared semantic latent space.

Figures & tables

Figure 1: Latent-space unified models enable interleaved cross-modal reasoning through shared semantic visual representations. Left: LatentUM improves text-to-image generation via self-reflection over its own generated semantic visual tokens. Middle: LatentUM interleaves textual reasoning with latent visual state updates for visual spatial planning. Right: LatentUM supports world modeling by predicting future visual states as semantic tokens conditioned on actions.

Figure 2: Overview of LatentUM. LatentUM unifies visual understanding and generation within a shared semantic latent space, enabling cross-modal reasoning without pixel-space mediation. (a) The visual tokenizer $\mathcal{Q}_{\phi}$ is trained via model behavior aligned quantization (MBAQ): it minimizes the KL divergence between the VLM’s output distributions on original features $\mathbf{V}$ versus quantized features $\mathcal{Q}_{\phi}(\mathbf{V})$ , preserving understanding-oriented semantics rather than pixel details. (b) Mixture-of-Modal Experts (MoME): the transformer maintains two parallel branches, where $\psi$ handles understanding and $\theta$ handles generation. The two branches share one self-attention mechanism. Generated visual codes are de-quantized and recached in the same context, allowing the model to reason over its own outputs. (c) Decoupled pixel decoder: a diffusion decoder $\bm{\epsilon}_{\eta}$ optionally maps quantized semantic features to pixels for visualization. It is trained independently, so the core model never optimizes for pixel fidelity.

Model	MME-P $\uparrow$	POPE $\uparrow$	SEED $\uparrow$	MMBench $\uparrow$	MMMU val $\uparrow$
Tokenflow XL ( Qu et al., 2025 )	1551	87.8	72.6	76.8	43.2
Bagel ( Deng et al., 2025 )	1687	-	-	85.0	55.3
JanusFlow ( Ma et al., 2025b )	1333	88.0	70.5	-	-
Harmo ( Wu et al., 2025d )	1155	87.6	67.1	65.5	-
Chameleon ( Team, 2024 )	1057	77.8	-	-	26.7
Orthus ( Kou et al., 2024 )	1266	79.6	-	-	28.2

Table 1: Evaluation on multimodal understanding benchmarks. LatentUM Base achieves competitive performance while preserving the pretrained VLM’s visual understanding capability. † denotes the quantized setting where LatentUM Base processes benchmark images through quantized visual representations.

Figure 5: Text-to-image gallery of LatentUM. The last column highlights LatentUM’s text rendering capability, which emerges as visual and language tokens share a unified semantic space, enabling legible in-image text.

Model	Single Obj.	Two Obj.	Counting	Colors	Position	Color Attr.	Overall $\uparrow$
Specialized Text-to-Image Models
FLUX.1 [Dev] ( Labs, 2024 )	0.98	0.81	0.74	0.79	0.22	0.45	0.66
SD3.5 Large ( Esser et al., 2024 )	0.98	0.89	0.73	0.83	0.34	0.47	0.71
Qwen-Image ( Wu et al., 2025a )	0.99	0.92	0.89	0.88	0.76	0.77	0.87
Unified Models
TokenFlow-XL ( Qu et al., 2025 )	0.95	0.60	0.41	0.81	0.16	0.24	0.55

Table 2: Evaluation on GenEval ( Ghosh et al., 2023 ) . LatentUM achieves the best performance among unified models on compositional instruction-following for visual generation. LatentUM Vis-Gen is post-trained from LatentUM Base via GRPO with self-derived reward; pixel-reward denotes further replacing the reward signal with the external GenEval pixel-reward model.

	TIFA GM	TIFA AM
Bagel + CoT ( Deng et al., 2025 )	23.1	70.9
Janus Pro ( Chen et al., 2025b )	14.5	58.3
BLIP3o 8B ( Chen et al., 2025a )	13.3	59.4
LatentUM Base	21.4	68.1
LatentUM Vis-Gen	31.3	72.9

Table 3: Evaluation on GenEval2 ( Kamath et al., 2025 ) . LatentUM Vis-Gen outperforms other unified models by a large margin.

Figure 6: Two interleaved reasoning paradigms for visual spatial planning. (a) Coarse-grained: LatentUM Vis-Plan first analyzes the maze layout, generates a complete visual plan, then derives the solution. (b) Fine-grained: LatentUM Vis-Plan alternates between textual reasonings and visual state updates, where each generated visual state serves as context for the next action.

Model	Lv.3	Lv.4	Lv.5	Lv.6	Avg.
Anole ( Chern et al., 2024 )	0.02	0.01	0.00	0.00	0.01
MVoT ( Li et al., 2025a )	0.21	0.11	0.08	0.03	0.11
Mirage ( Yang et al., 2025 )	0.93	0.83	0.76	0.51	0.76
Chameleon ( Team, 2024 )	-	-	-	-	0.01
InternVL3.5 8B ( Wang et al., 2025a )	-	-	-	-	0.08
InternVL3.5 38B ( Wang et al., 2025a )	-	-	-	-	0.20

Table 4: Evaluation on the VSP benchmark ( Wu et al., 2024a ) . LatentUM Vis-Plan outperforms all prior visual reasoning models under both planning paradigms, with the fine-grained variant achieving near-perfect accuracy.

Figure 7: World Modeling for Visual Navigation. (a) Open-loop future frame prediction. Given initial context frames (red box) and a predefined action trajectory, LatentUM WM rolls out temporally consistent future observations that preserve the spatial layout of the initial scene. (b) Zero-shot language-conditioned navigation. Without language supervision during fine-tuning, LatentUM WM translates free-form textual instructions (e.g., “get closer to the left house”) into accurate future visual states, enabled by the inherited language-vision alignment. (c) Quantitative comparison on the NWM benchmark ( Bar et al., 2025 ) . LatentUM WM surpasses the unified baseline Transfusion-RAE on both ATE and RPE, and remains competitive with the specialized NWM model.

Figure 8: Ablation study. (a) Text-to-image generation by the VQVAE baseline using the same prompts as Figure 5 (leftmost and rightmost of the first two rows). (b) Performance comparison on GenEval and GenEval2. Semantically aligned features induce stronger instruction-following capability during visual generation. (c) Comparison of semantic feature quantizers trained with different objectives on visual understanding benchmarks. MBAQ consistently outperforms feature-reconstruction baselines.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Generation results under different CFG scales ( $1.0$ to $5.0$ ) for identical prompts. LatentUM produces coherent images even at CFG $=1.0$ .

Figure 10: Reward curve during GRPO training for self-reflection visual generation.

Figure 11: Qualitative comparison before and after self-reward GRPO. Top: LatentUM Base before GRPO. Bottom: the model after self-reward GRPO. The post-trained model follows counting prompts more accurately.

Figure 12: Examples where the generated images are qualitatively correct, but the detection-based metrics mark them as failures.

Figure 13: Top: independent keyframe decoding with the original decoupled decoder. Bottom: reference-conditioned decoding with the first-frame VAE latent. The bottom row shows much better cross-frame consistency.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 290 papers

LatentUMM: Dual Latent Alignment for Unified Multimodal Models
May 18, 2026Yinyi Luo, Wenwen Wang, Hayes Bai +2Cross-Modal Learning Unified Multimodal Models
Carnegie Mellon University · William & Mary
Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Unified Multimodal Models Multimodal Generation
1S-Lab, Nanyang Technological University · 2SenseTime Research
InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing
Mar 10, 2026Changyao Tian, Danni Yang, Guanzhou Chen +26Unified Multimodal Models Multimodal Generation
Shanghai AI Laboratory · CUHK MMLab · Fudan University +6

May 18, 2026J/K move · Enter open · S save

LatentUMM: Dual Latent Alignment for Unified Multimodal Models

Yinyi Luo, Wenwen Wang, Hayes Bai, Marios Savvides+1

Carnegie Mellon University · William & Mary

Cross-Modal Learning Unified Multimodal Models Cross-Modal Alignment Multimodal Foundation Models Latent Space Alignment Multimodal Understanding

Open paper AskPDF arXiv

Unified multimodal models (UMMs) achieve strong performance in both understanding and generation by learning a shared latent space, yet they often exhibit functional inconsistency between these two capabilities. We observe that this issue does not stem from a lack of shared representations, but from the absence of explicit alignment between the transformations that map into and out of the latent space. As a result, generation and re-encoding can follow inconsistent trajectories, leading to semantic drift under modality transitions. In this work, we propose LatentUMM, a framework that constructs an enhanced shared latent space to explicitly align these transformations and improve cross-modal consistency. LatentUMM consists of two stages. First, dual latent alignment enforces consistency at both the modality and capacity levels: cross-modal alignment uses a stronger embedding model to impose structured cross-modal semantics, while dual capacity alignment enforces bidirectional consistency under generation and re-encoding. Second, latent dynamics stabilization improves robustness via stochastic latent rollouts and preference optimization, favoring trajectories that better preserve semantic consistency. Experiments show that LatentUMM consistently improves multimodal consistency across diverse architectures. Code is available at: https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/LatentUMM.

Abstract

Unified Multimodal Models

Explore similar work

LatentUMM: Dual Latent Alignment for Unified Multimodal Models

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing