UniCompress: Token Compression for Unified Vision-Language Understanding and Generation

Ziyao Wang; Chen Chen; Jingtao Li; Weiming Zhuang; Jiabo Huang; Ang Li; Lingjuan Lyu

cs.CVMar 11, 2026

UniCompress: Token Compression for Unified Vision-Language Understanding and Generation

Authors: Ziyao Wang, Chen Chen, Jingtao Li, Weiming Zhuang, Jiabo Huang, Ang Li, Lingjuan Lyu

Organizations: Sony AI · University of Maryland, College Park

Abstract

Unified models aim to support both understanding and generation by encoding images into discrete tokens and processing them alongside text within a single autoregressive framework. This unified design offers architectural simplicity and cross-modal synergy, which facilitates shared parameterization, consistent training objectives, and seamless transfer between modalities. However, the large number of visual tokens required by such models introduces substantial computation and memory overhead, and this inefficiency directly hinders deployment in resource constrained scenarios such as embodied AI systems. In this work, we propose a unified token compression algorithm UniCompress that significantly reduces visual token count while preserving performance on both image understanding and generation tasks. Our method introduces a plug-in compression and decompression mechanism guided with learnable global meta tokens. The framework is lightweight and modular, enabling efficient integration into existing models without full retraining. Experimental results show that our approach reduces image tokens by up to 4 times, achieves substantial gains in inference latency and training cost, and incurs only minimal performance degradation, which demonstrates the promise of token-efficient unified modeling for real world multimodal applications.

Figures & tables

Figure 1: We propose UniCompress , a plug-in-and-play token compression algorithm for unified models. The samples are from UniTok [ 24 ] .

Figure 2: Overview of UniCompress . The tokenizer is augmented with three modules: a global token extractor, a token compressor, and an autoregressive decompressor. The language model consumes a compact visual sequence for understanding and produces compressed-domain targets for generation.

Methods	GQA	MME Cog.	MME	POPE	Seed-bench	TextVQA	MMMU	MM-Bench
UniTok	55.71	251.79	1162.50	82.66	49.22	24.90	26.22	40.34
UniTok-Compressed	53.07	235.00	1036.05	79.36	48.07	24.66	26.22	42.14
Vila-U	53.43	247.65	1130.20	81.45	47.88	24.11	25.72	39.55
Vila-U-Compressed	51.90	241.24	1105.35	80.75	47.30	23.85	25.55	39.94
VARGPT	58.12	269.30	1290.65	88.04	50.54	26.18	27.13	44.22
VARGPT-Compressed	55.90	265.83	1272.80	84.99	48.41	25.34	27.40	41.15

Table 1: Unified model performance on visual understanding benchmarks (higher is better). XXX-Compressed denotes the same backbone with our plug-in token compression (

s{=}2

N_{g}{=}4

). ’MME Cog.’ refers to the score of MME Cognition.

Methods	FID( $\downarrow$ )	CLIP( $\uparrow$ )
UniTok	16.14	30.5
UniTok-Compressed	16.33	25.0
Vila-U	14.80	29.8
Vila-U-Compressed	16.37	28.9
VARGPT	14.77	24.2
VARGPT-Compressed	15.02	21.6

Table 2: Performance of the original and compressed unified models on image generation benchmarks. XXX-Compressed inserts UniCompress without changing the LM interface. Lower FID and higher CLIP indicate better quality.

Methods	Und. Train (h)	Gen. Train (h)	Und. Inference (min)	Gen. Inference (min)
UniTok	0.94	4.60	5.41	32.25
UniTok-Compressed	0.81	3.89	5.25	18.96
Vila-U	0.89	4.25	5.12	30.15
Vila-U-Compressed	0.77	3.65	5.09	17.55
VARGPT	1.13	5.52	5.45	38.74
VARGPT-Compressed	0.92	4.67	5.43	22.71

Table 3: Wall-clock time with/without plug-in token compression. Understanding: ShareGPT4V_PT (train), GQA (inference). Generation: JDB (train), MJHQ-30K (inference). Lower is better. Although the model is trained on the two datasets jointly in other experiments, the training times in this table were measured by training on each dataset separately.

Figure 3: Understanding task examples: generating the texts that describe the image.

Figure 4: Ablation on global token type. Results use

N_{g}{=}4

. Our global meta token yields competitive understanding and notably stronger generation quality (lower FID, higher CLIP).

Figure 5: UniCompress preserves the most visual information under compression by using global meta tokens and autoregressive decompressor.

Figure 6: Effect of token keep ratio on accuracy. GQA (understanding) vs. MJHQ-30K CLIP (generation).

Compressor	GQA	TextVQA	MM-Bench	Und. Avg	FID( $\downarrow$ )
AvgPool (Ours)	53.07	24.66	42.14	39.29	20.01
MaxPool	50.80	24.40	41.90	39.03	20.15
Strided Conv ( $2{\times}2$ )	51.20	24.70	42.10	39.33	19.95
Top- $k$ Token Selection	49.90	24.00	41.20	38.37	21.00
Learned Pooling (Gated)	51.30	24.75	42.20	39.42	19.90

Table 4: Ablation on local token compression (pooling/selection). All rows target the same token budget (

\times 4.0

) as our default setting with

s=2

. Results use

N_{g}=4

Figure 7: Effect of the number of global tokens $N_{g}$ . Left: vision–language understanding (GQA, TextVQA, MMMU, MMBench; higher is better). Right: image generation quality (FID; lower is better).

Explore similar work

UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations
Mar 11, 2026Yaqi Zhao, Wang Lin, Zijian Zhang +5Unified Multimodal Models Multimodal Generation
Peking University · Tencent Hunyuan · Zhejiang University
On the Limits of Token Reduction for Efficient Unified Vision Language Training
May 31, 2026Siyi Chen, Weiming Zhuang, Jingtao Li +1Vision-Language Models Unified Multimodal Models
University of Michigan · Sony AI
DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
Feb 21, 2026Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma +2Efficient VLM Inference Multimodal Token Compression
Advanced Micro Devices, Inc. (AMD)

Mar 11, 2026J/K move · Enter open · S save

UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations

Yaqi Zhao, Wang Lin, Zijian Zhang, Miles Yang+4

Peking University · Tencent Hunyuan · Zhejiang University

Unified Multimodal Models Multimodal Generation Multimodal Token Compression Multimodal Understanding

Open paper AskPDF arXiv

Current unified multimodal models typically rely on discrete visual tokenizers to bridge the modality gap. However, discretization inevitably discards fine-grained semantic information, leading to suboptimal performance in visual understanding tasks. Conversely, directly modeling continuous semantic representations (e.g., CLIP, SigLIP) poses significant challenges in high-dimensional generative modeling, resulting in slow convergence and training instability. To resolve this dilemma, we introduce UniCom, a unified framework that harmonizes multimodal understanding and generation via compressed continuous representation. We empirically demonstrate that reducing channel dimension is significantly more effective than spatial downsampling for both reconstruction and generation. Accordingly, we design an attention-based semantic compressor to distill dense features into a compact unified representation. Furthermore, we validate that the transfusion architecture surpasses query-based designs in convergence and consistency. Experiments demonstrate that UniCom achieves state-of-the-art generation performance among unified models. Notably, by preserving rich semantic priors, it delivers exceptional controllability in image editing and maintains image consistency even without relying on VAE.

Abstract

Explore similar work

UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations

On the Limits of Token Reduction for Efficient Unified Vision Language Training

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference