Unified Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches typically compromise by employing decoupled encoders, stacking representation encoder atop VAEs, or utilizing discrete quantization. However, these methods often disrupt information coherence and lead to optimization conflicts. To this end, we introduce HYDRA-TOK, a representation-harmonized pure ViT in the insight that visual modeling should evolve from generation to understanding. HYDRA-TOK reformulates the standard backbone into a progressive learner that transitions from a Gen-ViT, which captures structure-preserving primitives, to a Sem-ViT for semantic encoding. Crucially, this transition is mediated by a Generation-Semantic Bottleneck (GSB), which compresses features into a low-dimensional space to filter noise for robust synthesis, then restores dimensionality to empower complex semantic comprehension. Built upon this foundation, we present HYDRA, a native unified framework integrating perception and generation within a single parameter space. Extensive experiments establish HYDRA as a new state-of-the-art. It sets a benchmark in visual reconstruction (rFID 0.08) and achieves top-tier generation performance on GenEval (0.86), DPG-Bench (86.4), and WISE (0.53), while simultaneously outperforming previous native UMMs by an average of ~10.0 points across eight challenging understanding benchmarks.
Figures & tables
Figure 1: Multimodal understanding and image generation results. (a) HYDRA outperforms unified and task-specific models across diverse benchmarks. (b) HYDRA generates high-fidelity images with strong semantic alignment.
Figure 2: Representation schemes in native unified multimodal models. (a) Decoupled Encoder [ 1 , 2 ] : Uses separate encoders (a VAE and a representation encoder) for generation and understanding. (b) Sequential Encoder [ 3 ] : Feeds the outputs of the VAE directly into the representation encoder in a cascaded manner. (c) Single Encoder [ 4 , 5 ] : Unifies both tasks within a standalone representation encoder. (d) Our Proposed Representation-Harmonized ViT Design: Employs a single ViT backbone with a bottleneck module to harmonize feature learning across both tasks.
Figure 3: Training process illustration for HYDRA-TOK and HYDRA . (a) HYDRA-TOK functions as a progressive learner, bridging the gap between reconstruction and understanding. It employs a Generation-Semantic Bottleneck (GSB) to execute a unique compress-and-reconstruct operation, effectively filtering noise to transition from structure-preserving primitives (Gen-ViT) to semantic abstractions (Sem-ViT). (b) HYDRA unifies these representations using a dual-head architecture, effectively combining autoregressive text prediction and rectified flow matching.
Figure 4
Figure 6: Analysis of representation-harmonized co-promotion. (a) For generation, joint training enhances generation efficiency by stabilizing the latent space. (b) For understanding, joint training surpasses single-task baselines after a crossover point, showing that generative constraints refine perceptual precision. Details of each understanding benchmark and experiment setting is shown in Fig. 8 and Appendix D.2
Figure 7: Ablation of HYDRA-TOK loss. Baseline utilizes only Lrec . “1”: Teacher init; “2”: Lreg ; “3”: Ldist .
Stage I
Stage II
Stage III
Avg
GenEval
✗
✓
✓
0.719
0.82
✓
✗
✓
0.683
0.77
✓
✓
✗
-
0.84
✓
✓
✓
0.744
0.86
Table 4: Ablation study on the effects of the three training stages.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Asset Name
Asset Type
License
Qwen2.5 [ 32 ]
Pre-trained Model
Apache License 2.0
InternViT [ 16 ]
Pre-trained Model
Apache License 2.0
ImageNet-1K [ 65 ]
Dataset
Custom (Non-commercial research)
SA-1B [ 63 ]
Dataset
Custom (Research use only)
Appendix
Table 9: Summary of licenses for existing assets used in this work.
Figure 8: Performance comparison on various benchmarks across different training steps. The benchmarks include general perception (SEED-Bench), reasoning (MMStar, MMMU, MMBench, AI2D), and OCR-related tasks (OCRBench).
Figure 9: Qualitative comparison of image reconstruction. We compare visual results from our HYDRA between RAE [ 42 ] , SD1.5 [ 74 ] , VTP [ 56 ] , and FLUX [ 35 ] .
Figure 10: Qualitative comparison of text-to-image generation. We compare visual generation between Harmon [ 5 ] , Show-o2 [ 3 ] , Bagel [ 1 ] , Janus-pro [ 2 ] , and Blip-3o [ 11 ] .
Figure 11: Qualitative comparison of text-to-image generation. We compare visual generation results from our HYDRA between Harmon [ 5 ] , Show-o2 [ 3 ] , Bagel [ 1 ] , Janus-pro [ 2 ] , and Blip-3o [ 11 ] .
Figure 12: Qualitative results on multimodal image Understanding. First, HYDRA exhibits superior proficiency in transforming complex visual data into structured text. From accurately generating LaTeX and Python code for formulas and charts to extracting fine-grained details from concept maps, the results highlight its powerful comprehension of diverse and abstract visual inputs.
Holistic visual tokenizers are fundamental to unified multimodal models (UMMs) as they map diverse visual inputs into a unified representation space. In this paper, we present HYDRA-X, the first UMM that unifies image and video tokenization within a single Vision Transformer (ViT). Our design is driven by two core challenges: efficiently injecting spatiotemporal reconstruction capability into a native ViT, and embedding image- and video-level semantic awareness into the latent space. To address the first, comprehensive ablations reveal two key findings: (1) frame-level causal temporal attention suffices for visual reconstruction, whereas full spatiotemporal attention degrades it; and (2) hierarchical temporal compression substantially outperforms single-step alternatives. To tackle the second, we propose a lightweight decompressor that upsamples temporally compressed features under joint image-video teacher supervision, thereby enforcing complementary semantic structures within the compact latent space. Building on this holistic tokenizer, we further propose a principled improvement of the editing pipeline: source-target interaction should occur at the latent level inside the tokenizer rather than at the semantic level inside the LLM, substantially improving editing consistency and accelerating convergence. Instantiated at the 7B dense model, HYDRA-X achieves strong performance across image and video understanding and generation tasks, paving the way for future unified-tokenizer UMMs.
Guozhen Zhang, Xuerui Qiu, Yutao Cui +11
Nanjing University · Work done during internship at Tencent Hunyuan. · CASIA +3
A recent cutting-edge topic in multimodal modeling is to unify visual comprehension and generation within a single model. However, the two tasks demand mismatched decoding regimes and visual representations, making it non-trivial to jointly optimize within a shared feature space. In this work, we present Cheers, a unified multimodal model that decouples patch-level details from semantic representations, thereby stabilizing semantics for multimodal understanding and improving fidelity for image generation via gated detail residuals. Cheers includes three key components: (i) a unified vision tokenizer that encodes and compresses image latent states into semantic tokens for efficient LLM conditioning, (ii) an LLM-based Transformer that unifies autoregressive decoding for text generation and diffusion decoding for image generation, and (iii) a cascaded flow matching head that decodes visual semantics first and then injects semantically gated detail residuals from the vision tokenizer to refine high-frequency content. Experiments on popular benchmarks demonstrate that Cheers matches or surpasses advanced UMMs in both visual understanding and generation. Cheers also achieves 4x token compression, enabling more efficient high-resolution image encoding and generation. Notably, Cheers outperforms the Tar-1.5B on the popular benchmarks GenEval and MMBench, while requiring only 20% of the training cost, indicating effective and efficient (i.e., 4x token compression) unified multimodal modeling. We will release all code and data for future research.
Yichen Zhang, Da Peng, Zonghao Guo +19
Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences
Unified Multimodal Modeling aims to integrate visual understanding and generation within a single system. However, existing approaches typically rely on two disparate visual tokenizers, which splits the representation space and hinders truly unified modeling. We propose UniAR, a unified autoregressive framework where a single discrete visual tokenizer serves as the key bridge between understanding and generation, enabling a shared context in which the model can directly interpret its own generated visual tokens without additional re-encoding. UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme, preserving both high-level semantics and low-level details while scaling the effective visual vocabulary at minimal cost. Building on this, the unified autoregressive model adopts parallel-bitwise-prediction to jointly predict spatially grouped, multi-level visual codes, substantially reducing visual sequence length and accelerating generation. Finally, a diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images. Through large-scale pre-training, followed by supervised fine-tuning and reinforcement learning, UniAR achieves state-of-the-art performance on image generation and image editing while remaining competitive on multimodal understanding benchmarks. The project page is available at https://sharelab-sii.github.io/uniar-web.
Wujian Peng, Lingchen Meng, Yuxuan Cai +7
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Qwen Team, Alibaba Inc.