Unified models aim to support both understanding and generation by encoding images into discrete tokens and processing them alongside text within a single autoregressive framework. This unified design offers architectural simplicity and cross-modal synergy, which facilitates shared parameterization, consistent training objectives, and seamless transfer between modalities. However, the large number of visual tokens required by such models introduces substantial computation and memory overhead, and this inefficiency directly hinders deployment in resource constrained scenarios such as embodied AI systems. In this work, we propose a unified token compression algorithm UniCompress that significantly reduces visual token count while preserving performance on both image understanding and generation tasks. Our method introduces a plug-in compression and decompression mechanism guided with learnable global meta tokens. The framework is lightweight and modular, enabling efficient integration into existing models without full retraining. Experimental results show that our approach reduces image tokens by up to 4 times, achieves substantial gains in inference latency and training cost, and incurs only minimal performance degradation, which demonstrates the promise of token-efficient unified modeling for real world multimodal applications.
Figures & tables
Figure 1: We propose UniCompress , a plug-in-and-play token compression algorithm for unified models. The samples are from UniTok [ 24 ] .
Figure 2: Overview of UniCompress . The tokenizer is augmented with three modules: a global token extractor, a token compressor, and an autoregressive decompressor. The language model consumes a compact visual sequence for understanding and produces compressed-domain targets for generation.
Methods
GQA
MME Cog.
MME
POPE
Seed-bench
TextVQA
MMMU
MM-Bench
UniTok
55.71
251.79
1162.50
82.66
49.22
24.90
26.22
40.34
UniTok-Compressed
53.07
235.00
1036.05
79.36
48.07
24.66
26.22
42.14
Vila-U
53.43
247.65
1130.20
81.45
47.88
24.11
25.72
39.55
Vila-U-Compressed
51.90
241.24
1105.35
80.75
47.30
23.85
25.55
39.94
VARGPT
58.12
269.30
1290.65
88.04
50.54
26.18
27.13
44.22
VARGPT-Compressed
55.90
265.83
1272.80
84.99
48.41
25.34
27.40
41.15
Table 1: Unified model performance on visual understanding benchmarks (higher is better). XXX-Compressed denotes the same backbone with our plug-in token compression ( s=2 , Ng=4 ). ’MME Cog.’ refers to the score of MME Cognition.
Methods
FID( ↓ )
CLIP( ↑ )
UniTok
16.14
30.5
UniTok-Compressed
16.33
25.0
Vila-U
14.80
29.8
Vila-U-Compressed
16.37
28.9
VARGPT
14.77
24.2
VARGPT-Compressed
15.02
21.6
Table 2: Performance of the original and compressed unified models on image generation benchmarks. XXX-Compressed inserts UniCompress without changing the LM interface. Lower FID and higher CLIP indicate better quality.
Methods
Und. Train (h)
Gen. Train (h)
Und. Inference (min)
Gen. Inference (min)
UniTok
0.94
4.60
5.41
32.25
UniTok-Compressed
0.81
3.89
5.25
18.96
Vila-U
0.89
4.25
5.12
30.15
Vila-U-Compressed
0.77
3.65
5.09
17.55
VARGPT
1.13
5.52
5.45
38.74
VARGPT-Compressed
0.92
4.67
5.43
22.71
Table 3: Wall-clock time with/without plug-in token compression. Understanding: ShareGPT4V_PT (train), GQA (inference). Generation: JDB (train), MJHQ-30K (inference). Lower is better. Although the model is trained on the two datasets jointly in other experiments, the training times in this table were measured by training on each dataset separately.
Figure 3: Understanding task examples: generating the texts that describe the image.
Figure 4: Ablation on global token type. Results use Ng=4 . Our global meta token yields competitive understanding and notably stronger generation quality (lower FID, higher CLIP).
Figure 5: UniCompress preserves the most visual information under compression by using global meta tokens and autoregressive decompressor.
Figure 6: Effect of token keep ratio on accuracy. GQA (understanding) vs. MJHQ-30K CLIP (generation).
Compressor
GQA
TextVQA
MM-Bench
Und. Avg
FID( ↓ )
AvgPool (Ours)
53.07
24.66
42.14
39.29
20.01
MaxPool
50.80
24.40
41.90
39.03
20.15
Strided Conv ( 2×2 )
51.20
24.70
42.10
39.33
19.95
Top- k Token Selection
49.90
24.00
41.20
38.37
21.00
Learned Pooling (Gated)
51.30
24.75
42.20
39.42
19.90
Table 4: Ablation on local token compression (pooling/selection). All rows target the same token budget ( ×4.0 ) as our default setting with s=2 . Results use Ng=4 .
Figure 7: Effect of the number of global tokens Ng . Left: vision–language understanding (GQA, TextVQA, MMMU, MMBench; higher is better). Right: image generation quality (FID; lower is better).
Current unified multimodal models typically rely on discrete visual tokenizers to bridge the modality gap. However, discretization inevitably discards fine-grained semantic information, leading to suboptimal performance in visual understanding tasks. Conversely, directly modeling continuous semantic representations (e.g., CLIP, SigLIP) poses significant challenges in high-dimensional generative modeling, resulting in slow convergence and training instability. To resolve this dilemma, we introduce UniCom, a unified framework that harmonizes multimodal understanding and generation via compressed continuous representation. We empirically demonstrate that reducing channel dimension is significantly more effective than spatial downsampling for both reconstruction and generation. Accordingly, we design an attention-based semantic compressor to distill dense features into a compact unified representation. Furthermore, we validate that the transfusion architecture surpasses query-based designs in convergence and consistency. Experiments demonstrate that UniCom achieves state-of-the-art generation performance among unified models. Notably, by preserving rich semantic priors, it delivers exceptional controllability in image editing and maintains image consistency even without relying on VAE.
Yaqi Zhao, Wang Lin, Zijian Zhang +5
Peking University · Tencent Hunyuan · Zhejiang University
Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone, but their joint training is computationally expensive and largely overlooked from an efficiency perspective. In this work, we study the feasibility and limits of token-reduction-based acceleration for unified VLM training. Through a systematic analysis of layerwise attention allocation, we uncover a fundamental asymmetry: visual understanding exhibits substantial late-layer visual redundancy, whereas visual generation maintains persistent dependence on image tokens across depth. Guided by this observation, we design task-specific accelerators that selectively reduce image-token computation for each objective. While these methods achieve significant efficiency gains in isolated settings, we observe a consistent synergy loss under unified training -- task-specific token dropping necessitates divergent parameter pathways and eliminates the mutual performance gains typically observed in joint optimization. Our findings suggest that efficient unified modeling requires preserving shared cross-task structures, highlighting the need for synergy-aware acceleration strategies. Project page: https://chicychen.github.io/TokenReductionUnifiedVLM/.
Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.