The development of unified multimodal large language models (MLLMs) is fundamentally challenged by the granularity gap between visual understanding and generation: understanding requires high-level semantic abstractions, while image generation demands fine-grained pixel-level representations. Existing approaches usually enforce the two supervision on the same set of representation or decouple these two supervision on separate feature spaces, leading to interference and inconsistency, respectively. In this work, we propose EvoTok, a unified image tokenizer that reconciles these requirements through a residual evolution process within a shared latent space. Instead of maintaining separate token spaces for pixels and semantics, EvoTok encodes an image into a cascaded sequence of residual tokens via residual vector quantization. This residual sequence forms an evolution trajectory where earlier stages capture low-level details and deeper stages progressively transition toward high-level semantic representations. Despite being trained on a relatively modest dataset of 13M images, far smaller than the billion-scale datasets used by many previous unified tokenizers, EvoTok achieves a strong reconstruction quality of 0.43 rFID on ImageNet-1K at 256x256 resolution. When integrated with a large language model, EvoTok shows promising performance across 7 out of 9 visual understanding benchmarks, and remarkable results on image generation benchmarks such as GenEval and GenAI-Bench. These results demonstrate that modeling visual representations as an evolving trajectory provides an effective and principled solution for unifying visual understanding and generation.
Figures & tables
Figure 1: Different paradigms of current MLLMs and unified models. (a) Entangled unified MLLMs share semantic and pixel features, enabling both understanding and generation but tightly coupling the two objectives. (b) Decoupled unified MLLMs separate semantic and pixel encoders or feature layers to disentangle tasks. (c) Our unified evolution MLLM evolves pixel features into semantic representations within a shared space while keeping decoupled latents, achieving aligned understanding and generation while preserving decoupled representations.
Figure 2: The overview of the proposed EvoTok. An image is encoded into a sequence of residual tokens within a shared latent space, forming a residual evolution trajectory. Tokens from earlier stages preserve pixel-level details for reconstruction, while tokens from deeper stages, formed by progressively accumulating tokens from full stages, capture semantic-level features aligned with those extracted by the pre-trained SigLIP2 [ 49 ] . Within this residual evolution trajectory, low-level pixel features and high-level semantic representations co-evolve , enabling a decoupled and consistent representations to support both visual understanding and generation.
Figure 3: Images reconstruction at a resolution of 256 × 256 on ImageNet-1K.
Model
Resolution
Code Shape
Codebook Size
rFID ↓
Reconstruction Only
LLaMAGen [ 45 ]
256
16 × 16
16,384
2.19
RQVAE [ 25 ]
256
16 × 16 × 4
16,384
3.20
VQGAN-LC [ 60 ]
256
16 × 16
16,384
2.62
IBQ [ 43 ]
256
16 × 16
252,144
1.00
Unified Tokenizer
Table 1: Comparison of the reconstruction FID on ImageNet-1K. rFID is measured at 256 × 256 resolution. Bold and underline indicate the best and the second best performance, respectively. † : indicates model trained on billion-scale datasets such as COYO-700M [ 4 ] or DataComp-1B [ 14 ] .
Model
LLM
Res.
SEEDB
MM-Vet
GQA
AI2D
RWQA
MMMU
MMB
MME
MME-P
Understanding Only (Continuous)
Qwen-VL-Chat [ 2 ]
Qwen-7B
448
57.7
-
57.5
-
-
-
-
1848.3
1478.5
VILA [ 29 ]
LLaMA-2-7B
336
61.1
34.9
62.3
-
-
-
68.9
1533.0
-
ShareGPT4V [ 9 ]
Vicuna-7B
336
69.7
37.6
63.3
58.0
54.9
37.2
68.8
1943.8
1567.4
LLaVA-v1.5 [ 31 ]
Vicuna-1.5-13B
336
68.1
36.1
63.3
61.1
55.3
36.4
67.7
1826.7
1531.3
Unified Understanding and Generation (Continuous)
Table 2: Quantitative results on visual understanding benchmarks. We collect benchmarks including: SEEDBench [ 27 ] , MM-Vet [ 57 ] , GQA [ 21 ] , AI2D [ 24 ] , RealWorldQA [ 55 ] , MMMU [ 58 ] , MMBench [ 34 ] , MME [ 13 ] and MME-P (MME-Perception). Three main types of methods are compared: continuous understanding methods ( top ), continuous unified methods ( middle ) and discrete unified methods ( bottom ). The best and the second best results among discrete unified methods are highlighted in bold and underline respectively.
Figure 4: Images generated with our unified EvoTok at a resolution of 256 × 256. Our model generates high-quality images spanning diverse visual domains, including portraits, landscapes, objects, artistic paintings, etc.
Model
Type
GenAI-Bench ↑
GenEval ↑
Basic
Advanced
Overall
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
Geneartion Only
SD v2.1 [ 42 ]
Diffusion
0.78
0.62
0.50
0.98
0.51
0.44
0.85
0.07
0.17
SD v1.5 [ 42 ]
Diffusion
-
-
0.43
0.97
0.38
0.35
0.76
0.04
0.06
SDXL [ 39 ]
Diffusion
0.83
0.63
0.55
0.98
0.74
0.39
0.85
0.15
0.23
DALL-E 3 [ 41 ]
Diffusion
-
-
0.67
0.96
0.87
0.47
0.83
0.43
0.45
Table 3: Quantitative results on visual generation benchmarks. We compare two types of baselines: generation-only specialist models ( top ) and unified MLLMs ( bottom ). EvoTok achieves strong performance on both widely used benchmarks, GenAI-Bench [ 22 ] and GenEval [ 17 ] .
Lpix
Lsem
Reconstruction
Understanding
Generation
rFID ↓
PSNR ↑
SSIM ↑
SEEDB ↑
GQA ↑
MME ↑
GenAI ( Bas. ) ↑
GenAI ( Adv. ) ↑
GenEval ↑
4
4
0.66
22.96
0.76
62.7
58.0
1668.9
0.70
0 .59
0 .64
16
4
0.44
24.39
0.79
6 4.6
6 0.9
1 731.6
0 .71
0.58
0.60
4
16
0 .55
2 3.60
0 .77
67.1
61.3
1793.5
0.75
0.62
0.67
Table 4: Ablation on unified feature trajectories. Three different unified strategies are compared: 1) entangled pixel and semantic modeling, 2) semantic-to-pixel evolution, and 3) our proposed pixel-to-semantic evolution. Our method achieves the most balanced performance across all tasks.
Figure 5: Analysis of the Unified Latent Space.
Figure 6: Visualization of the feature evolution trajectory across different residual depths. We visualize image clusters grouped by K-means centroids at specific depth stages L∈{1,4,8,16} . (Top Rows) At shallow depths ( L=1,4 ), the latent features primarily capture perceptual primitives such as textures ( Mesh , Grid ), color palettes ( Yellow , Blue ), and low-level structures ( Stride ). (Bottom Rows) As depth increases ( L=8,16 ), the representation naturally evolves toward high-level semantic abstractions , grouping diverse instances into coherent conceptual categories (e.g., Appliances , Food , Crawl ).
Decouple Method
rFID ↓
1) No decouple
0.66
2) Different feature layer
0.59
3) Different branch
0.68
4) Ours
0.55
Table 5: Ablation on different feature decoupling method.