The development of unified multimodal large language models (MLLMs) is fundamentally challenged by the granularity gap between visual understanding and generation: understanding requires high-level semantic abstractions, while image generation demands fine-grained pixel-level representations. Existing approaches usually enforce the two supervision on the same set of representation or decouple these two supervision on separate feature spaces, leading to interference and inconsistency, respectively. In this work, we propose EvoTok, a unified image tokenizer that reconciles these requirements through a residual evolution process within a shared latent space. Instead of maintaining separate token spaces for pixels and semantics, EvoTok encodes an image into a cascaded sequence of residual tokens via residual vector quantization. This residual sequence forms an evolution trajectory where earlier stages capture low-level details and deeper stages progressively transition toward high-level semantic representations. Despite being trained on a relatively modest dataset of 13M images, far smaller than the billion-scale datasets used by many previous unified tokenizers, EvoTok achieves a strong reconstruction quality of 0.43 rFID on ImageNet-1K at 256x256 resolution. When integrated with a large language model, EvoTok shows promising performance across 7 out of 9 visual understanding benchmarks, and remarkable results on image generation benchmarks such as GenEval and GenAI-Bench. These results demonstrate that modeling visual representations as an evolving trajectory provides an effective and principled solution for unifying visual understanding and generation.
Figures & tables
Figure 1: Different paradigms of current MLLMs and unified models. (a) Entangled unified MLLMs share semantic and pixel features, enabling both understanding and generation but tightly coupling the two objectives. (b) Decoupled unified MLLMs separate semantic and pixel encoders or feature layers to disentangle tasks. (c) Our unified evolution MLLM evolves pixel features into semantic representations within a shared space while keeping decoupled latents, achieving aligned understanding and generation while preserving decoupled representations.
Figure 2: The overview of the proposed EvoTok. An image is encoded into a sequence of residual tokens within a shared latent space, forming a residual evolution trajectory. Tokens from earlier stages preserve pixel-level details for reconstruction, while tokens from deeper stages, formed by progressively accumulating tokens from full stages, capture semantic-level features aligned with those extracted by the pre-trained SigLIP2 [ 49 ] . Within this residual evolution trajectory, low-level pixel features and high-level semantic representations co-evolve , enabling a decoupled and consistent representations to support both visual understanding and generation.
Figure 3: Images reconstruction at a resolution of 256 × 256 on ImageNet-1K.
Model
Resolution
Code Shape
Codebook Size
rFID ↓
Reconstruction Only
LLaMAGen [ 45 ]
256
16 × 16
16,384
2.19
RQVAE [ 25 ]
256
16 × 16 × 4
16,384
3.20
VQGAN-LC [ 60 ]
256
16 × 16
16,384
2.62
IBQ [ 43 ]
256
16 × 16
252,144
1.00
Unified Tokenizer
Table 1: Comparison of the reconstruction FID on ImageNet-1K. rFID is measured at 256 × 256 resolution. Bold and underline indicate the best and the second best performance, respectively. † : indicates model trained on billion-scale datasets such as COYO-700M [ 4 ] or DataComp-1B [ 14 ] .
Model
LLM
Res.
SEEDB
MM-Vet
GQA
AI2D
RWQA
MMMU
MMB
MME
MME-P
Understanding Only (Continuous)
Qwen-VL-Chat [ 2 ]
Qwen-7B
448
57.7
-
57.5
-
-
-
-
1848.3
1478.5
VILA [ 29 ]
LLaMA-2-7B
336
61.1
34.9
62.3
-
-
-
68.9
1533.0
-
ShareGPT4V [ 9 ]
Vicuna-7B
336
69.7
37.6
63.3
58.0
54.9
37.2
68.8
1943.8
1567.4
LLaVA-v1.5 [ 31 ]
Vicuna-1.5-13B
336
68.1
36.1
63.3
61.1
55.3
36.4
67.7
1826.7
1531.3
Unified Understanding and Generation (Continuous)
Table 2: Quantitative results on visual understanding benchmarks. We collect benchmarks including: SEEDBench [ 27 ] , MM-Vet [ 57 ] , GQA [ 21 ] , AI2D [ 24 ] , RealWorldQA [ 55 ] , MMMU [ 58 ] , MMBench [ 34 ] , MME [ 13 ] and MME-P (MME-Perception). Three main types of methods are compared: continuous understanding methods ( top ), continuous unified methods ( middle ) and discrete unified methods ( bottom ). The best and the second best results among discrete unified methods are highlighted in bold and underline respectively.
Figure 4: Images generated with our unified EvoTok at a resolution of 256 × 256. Our model generates high-quality images spanning diverse visual domains, including portraits, landscapes, objects, artistic paintings, etc.
Model
Type
GenAI-Bench ↑
GenEval ↑
Basic
Advanced
Overall
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
Geneartion Only
SD v2.1 [ 42 ]
Diffusion
0.78
0.62
0.50
0.98
0.51
0.44
0.85
0.07
0.17
SD v1.5 [ 42 ]
Diffusion
-
-
0.43
0.97
0.38
0.35
0.76
0.04
0.06
SDXL [ 39 ]
Diffusion
0.83
0.63
0.55
0.98
0.74
0.39
0.85
0.15
0.23
DALL-E 3 [ 41 ]
Diffusion
-
-
0.67
0.96
0.87
0.47
0.83
0.43
0.45
Table 3: Quantitative results on visual generation benchmarks. We compare two types of baselines: generation-only specialist models ( top ) and unified MLLMs ( bottom ). EvoTok achieves strong performance on both widely used benchmarks, GenAI-Bench [ 22 ] and GenEval [ 17 ] .
Lpix
Lsem
Reconstruction
Understanding
Generation
rFID ↓
PSNR ↑
SSIM ↑
SEEDB ↑
GQA ↑
MME ↑
GenAI ( Bas. ) ↑
GenAI ( Adv. ) ↑
GenEval ↑
4
4
0.66
22.96
0.76
62.7
58.0
1668.9
0.70
0 .59
0 .64
16
4
0.44
24.39
0.79
6 4.6
6 0.9
1 731.6
0 .71
0.58
0.60
4
16
0 .55
2 3.60
0 .77
67.1
61.3
1793.5
0.75
0.62
0.67
Table 4: Ablation on unified feature trajectories. Three different unified strategies are compared: 1) entangled pixel and semantic modeling, 2) semantic-to-pixel evolution, and 3) our proposed pixel-to-semantic evolution. Our method achieves the most balanced performance across all tasks.
Figure 5: Analysis of the Unified Latent Space.
Figure 6: Visualization of the feature evolution trajectory across different residual depths. We visualize image clusters grouped by K-means centroids at specific depth stages L∈{1,4,8,16} . (Top Rows) At shallow depths ( L=1,4 ), the latent features primarily capture perceptual primitives such as textures ( Mesh , Grid ), color palettes ( Yellow , Blue ), and low-level structures ( Stride ). (Bottom Rows) As depth increases ( L=8,16 ), the representation naturally evolves toward high-level semantic abstractions , grouping diverse instances into coherent conceptual categories (e.g., Appliances , Food , Crawl ).
Decouple Method
rFID ↓
1) No decouple
0.66
2) Different feature layer
0.59
3) Different branch
0.68
4) Ours
0.55
Table 5: Ablation on different feature decoupling method.
Building a unified visual tokenizer is essential for bridging the gap between visual understanding and generation. Yet existing approaches struggle with the inherent conflict between these tasks, as a single token space is forced to support both high-level semantic abstraction and low-level pixel reconstruction. We propose WinTok, a concise hybrid tokenizer that achieves a win-win performance by explicitly decoupling the two objectives. WinTok supplements pixel tokens with a set of learnable semantic tokens, effectively mitigating cross-task interference without incurring the computational overhead of dual tokenizers. To further enhance understanding capability, we introduce an asymmetric token distillation mechanism: the semantic tokens are guided by pretrained semantic embeddings from any visual foundation model, enabling them to inherit strong discriminative power while maintaining flexibility. Across 10 challenging benchmarks, WinTok delivers consistent improvements in reconstruction, understanding, and generation. Trained on only 50M open-source data, WinTok surpasses the strong baseline UniTok by 11.2% in classification accuracy and achieves a competitive reconstruction rFID of 0.41, despite using substantially less training data. Code is released at https://github.com/markywg/WinTok.
Yiwei Guo, Shaobin Zhuang, Zhipeng Huang +4
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · WeChat Vision, Tencent Inc. · Shanghai Jiao Tong University
Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.
Yunpeng Qu, Kaidong Zhang, Yukang Ding +2
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China
Unified Multimodal Modeling aims to integrate visual understanding and generation within a single system. However, existing approaches typically rely on two disparate visual tokenizers, which splits the representation space and hinders truly unified modeling. We propose UniAR, a unified autoregressive framework where a single discrete visual tokenizer serves as the key bridge between understanding and generation, enabling a shared context in which the model can directly interpret its own generated visual tokens without additional re-encoding. UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme, preserving both high-level semantics and low-level details while scaling the effective visual vocabulary at minimal cost. Building on this, the unified autoregressive model adopts parallel-bitwise-prediction to jointly predict spatially grouped, multi-level visual codes, substantially reducing visual sequence length and accelerating generation. Finally, a diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images. Through large-scale pre-training, followed by supervised fine-tuning and reinforcement learning, UniAR achieves state-of-the-art performance on image generation and image editing while remaining competitive on multimodal understanding benchmarks. The project page is available at https://sharelab-sii.github.io/uniar-web.
Wujian Peng, Lingchen Meng, Yuxuan Cai +7
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Qwen Team, Alibaba Inc.