UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations

Yaqi Zhao; Wang Lin; Zijian Zhang; Miles Yang; Jingyuan Chen; Wentao Zhang; Zhao Zhong; Liefeng Bo

cs.CVMar 11, 2026

UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations

Authors: Yaqi Zhao, Wang Lin, Zijian Zhang, Miles Yang, Jingyuan Chen, Wentao Zhang, Zhao Zhong, Liefeng Bo

Organizations: Peking University · Tencent Hunyuan · Zhejiang University

Abstract

Current unified multimodal models typically rely on discrete visual tokenizers to bridge the modality gap. However, discretization inevitably discards fine-grained semantic information, leading to suboptimal performance in visual understanding tasks. Conversely, directly modeling continuous semantic representations (e.g., CLIP, SigLIP) poses significant challenges in high-dimensional generative modeling, resulting in slow convergence and training instability. To resolve this dilemma, we introduce UniCom, a unified framework that harmonizes multimodal understanding and generation via compressed continuous representation. We empirically demonstrate that reducing channel dimension is significantly more effective than spatial downsampling for both reconstruction and generation. Accordingly, we design an attention-based semantic compressor to distill dense features into a compact unified representation. Furthermore, we validate that the transfusion architecture surpasses query-based designs in convergence and consistency. Experiments demonstrate that UniCom achieves state-of-the-art generation performance among unified models. Notably, by preserving rich semantic priors, it delivers exceptional controllability in image editing and maintains image consistency even without relying on VAE.

Figures & tables

Figure 1: High-quality samples generated by UniCom. Built on compressed continuous representations, our unified multimodal model demonstrates exceptional capabilities in text-to-image generation, precise image editing, and fine-grained controllable generation.

Figure 2: Overview of the proposed framework. For a controlled comparison, both pathways are built upon the same compressed representations and jointly optimized with cross-entropy loss ( $\mathcal{L}_{ce}$ ) and flow matching loss ( $\mathcal{L}_{fm}$ ).

Specialized tokenizers
Tokenizer	Res.	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
SD-VAE Esser et al. (2024b)	256	1.06	28.62	0.86
GigaTok Xiong et al. (2025)	256	0.51	21.32	0.69
VA-VAE Yao et al. (2025)	256	0.26	28.59	0.80
DC-AE Chen et al. (2024)	512	0.22	26.15	0.71
MAE-Tok Chen et al. (2025a)	512	0.62	-	-

Table 1: Evaluation on image reconstruction ability on the ImageNet validation set .

d

denotes the feature dimension of compressed representations

\tilde{\mathbf{z}}

. Models marked with † are re-evaluated using official checkpoints.

Figure 3: Comparison of the results for image editing, highlighting UniCom performance in tasks such as image manipulation, object swapping, and color adjustment. See more visualization in the Appendix.

Figure 4: Overview of the proposed diffusion decoder and reconstruction analysis. Given an input image, the vision encoder extracts semantic features which are then processed by compressor and decompressor to condition the diffusion model for image reconstruction. During training, the compression modules and the diffusion backbone are optimized, while the encoder remains frozen. We investigate the bottleneck capacity by varying the token number $n$ and feature dimension $d$ . As illustrated in the bottom row, compressing the dimension $d$ yields superior reconstruction fidelity, whereas reducing the token number $n$ leads to noticeable blurring in fine details.

Figure 5: Visual comparison of image reconstruction results . Notably, our method (d64) maintains high fidelity in high-frequency details (e.g., text characters) and preserves facial identity better than these semantic-based baselines, achieving quality comparable to the specialized Flux.1-dev VAE.

Generation-only Models
	GenEval							DPG	WISE
Models	Single	Two	Count	Colors	Pos	Col-Attr	Overall	Overall	Cult.	Time	Space	Bio.	Phy.	Chem.	Overall
SD3-Medium ( Esser et al., 2024a )	0.99	0.94	0.72	0.89	0.33	0.60	0.74	-	-	-	-	-	-	-	-
FLUX.1 [Dev] ( Labs, 2024 )	0.98	0.93	0.75	0.93	0.68	0.65	0.82	84.00	0.48	0.58	0.62	0.42	0.51	0.35	0.50
Unified Multimodal Models
MetaQuery-XL † ( Pan et al., 2025b )	-	-	-	-	-	-	0.80	-	0.56	0.55	0.62	0.49	0.63	0.41	0.55

Table 2: Image generation results on GenEval, DPG-Bench, and WISE . † refers to methods using LLM rewriters on GenEval. Abbreviations for WISE: Cult. (Cultural), Bio. (Biology), Phy. (Physics), Chem. (Chemistry).

	ImgEdit-Bench										GEdit-Bench			KRIS-Bench				WorldEdit
Models	Add	Adj.	Ext.	Rep.	Rm.	Bg.	Sty.	Hyb.	Act.	Overall	G-SC	G-PQ	G-Overall	Fact.	Conc.	Proc.	Overall	Overall
Generation-only Models
FLUX.1 Kontext [Pro] ( Batifol et al., 2025 )	4.25	4.15	2.35	4.56	3.57	4.26	4.57	3.68	4.63	4.00	7.02	7.60	6.56	57.22	55.06	46.69	54.17	3.21
Qwen-Image ( Wu et al., 2025a )	4.38	4.16	3.43	4.66	4.14	4.38	4.81	3.82	4.69	4.27	8.00	7.86	7.56	-	-	-	-	-
Specialized Editing Models
Instruct-Pix2Pix ( Brooks et al., 2023 )	2.45	1.83	1.44	2.01	1.50	1.44	3.55	1.20	1.46	1.88	3.58	5.49	3.68	23.33	25.59	17.28	22.82	2.44

Table 3: Comparison of image editing capabilities. We evaluate on ImgEdit-Bench, GEdit-Bench, KRIS-Bench and WorldEdit. For ImgEdit-Bench, performance is evaluated across nine distinct operation categories (e.g., ‘Add’, ‘Adjust’, ‘Extract’, ‘Replace’, ‘Remove’, ‘Background’, ‘Style’, ‘Hybrid’, and ‘Action’. ). For GEdit-Bench, metrics include ‘G-Semantic Consistency’ (G-SC) and ‘G-Perceptual Quality’ (G-PQ). For KRIS-Bench, we report Factual (Fact.), Conceptual (Conc.), and Procedural (Proc.) knowledge scores. Bold : best results. Underline : second-best.

Projector	Tokens ( $n$ )	Dim ( $d$ )	Reconstruction
Projector	Tokens ( $n$ )	Dim ( $d$ )	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
–	1024	1152	0.40	23.26	0.69
–	256	1152	0.72	20.29	0.56
MLP	1024	256	0.62	21.73	0.66
MLP	1024	64	0.55	22.17	0.66
MHA	1024	64	0.56	22.61	0.69

Table 4: Reconstruction performance of different compressed feature shapes on ImageNet50k ( Deng et al., 2009 ) .

Figure 8: T-SNE visualization of feature distributions. We randomly select 6 distinct classes from ImageNet, sampling 150 images per class. MHA compression to 64-d preserves the structure, MLP projection yields scattered 64-d features, and FLUX-VAE has no semantic separability due to pixel-reconstruction focus.

Compression Strategies
Model Config.	General VQA			Knowledge	Text-Centric
Model Config.	GQA	RealWorldQA	SEED	MMMU	ChartQA	OCRBench
Baseline	65.25	64.31	74.63	44.56	69.04	55.40
MLP Proj.	62.80	60.39	69.92	43.00	56.80	31.70
MHA Proj.	64.01	63.14	71.75	44.11	62.12	36.00
Concatenation Strategies

Table 5: Visual understanding performance on six benchmarks using compressed representations as visual inputs.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Reducing dimensions maintains higher reconstruction fidelity than sequence reduction. We compare the reconstruction training dynamics (Loss and PSNR) of different compressed representations ( $\tilde{\mathbf{z}}$ ).

Figure 10: Additional visual comparison of image reconstruction results.

Figure 11: Qualitative reconstruction samples using Decoder (d64). Left: Reconstructed image. Right: Original image. Even with an $18\times$ channel compression ratio, our model accurately recovers complex high-frequency details.

Cause Category	Metric	GPT-4o	Nano-Banana	Seedream4.0	UniCom	WorldEdit	Flux-Kontext	Bagel-Think	Bagel	Omnigen	Omnigen2	Emu2	Anyedit	Ip2p	Magicbrush
Time	VC	4.02	4.22	3.92	3.08	4.18	4.6	2.38	2.34	3.02	2.22	1.48	1.63	3.04	1.40
	VQ	5.00	5.00	4.86	4.78	4.55	4.88	4.52	4.02	4.20	4.35	4.56	3.25	4.51	4.28
	IF	3.86	3.90	4.25	4.08	3.71	1.46	2.46	1.66	1.38	1.33	1.54	1.79	1.51	1.19
	KP	4.20	4.14	4.41	4.38	3.94	1.42	2.86	1.80	1.48	1.25	1.75	1.92	1.65	1.34
	Avg	4.27	4.32	4.36	4.08	4.10	3.09	3.06	2.46	2.52	2.29	2.33	2.15	2.68	2.05
Temperature	VC	4.02	4.22	4.06	3.40	3.52	3.86	2.08	1.94	2.54	2.26	1.38	1.45	2.62	1.81

Appendix

Table 7: The performance of both commercial and open-source models on image editing tasks in WorldEdit-Test, evaluated across different causes and metrics.

Figure 13: More results on single image editing. From top to bottom: (1) Add/Remove/Extract : adding objects, removing elements, and extracting subjects; (2) Replace : substituting objects or materials while preserving context; (3) Style Transfer : transforming image appearance into different artistic styles; (4) Background : modifying or replacing image backgrounds; (5) Subject-Driven Generation : generating new images conditioned on reference subjects; (6) Controllable Generation : generating images with spatial or structural control signals.

Figure 14: More results on multi-element composition. Our model can compose multiple reference subjects into a coherent scene while preserving the identity of each element.

Figure 15: More results on intelligent image editing. These examples require complex world knowledge such as biological transformations, geographical understanding, and cultural context to perform accurate editing.

Figure 16: Chain-of-thought enhanced generation and editing. For each example, we show the generated image along with the corresponding thinking process enclosed in <think> tokens. The model first reasons about the required visual transformation, then generates the output image accordingly.

Explore similar work

UniCompress: Token Compression for Unified Vision-Language Understanding and Generation
Mar 11, 2026Ziyao Wang, Chen Chen, Jingtao Li +4Multimodal Token Compression Visual Token Compression
Sony AI · University of Maryland, College Park
Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation
Mar 13, 2026Yichen Zhang, Da Peng, Zonghao Guo +19Unified Multimodal Models Multimodal Generation
Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences
Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification
Jun 16, 2026Wujian Peng, Lingchen Meng, Yuxuan Cai +7Unified Multimodal Models Autoregressive Image Generation
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Qwen Team, Alibaba Inc.

Mar 11, 2026J/K move · Enter open · S save

UniCompress: Token Compression for Unified Vision-Language Understanding and Generation

Ziyao Wang, Chen Chen, Jingtao Li, Weiming Zhuang+3

Sony AI · University of Maryland, College Park

Multimodal Token Compression Visual Token Compression Efficient VLM Inference Unified Multimodal Models

Open paper AskPDF arXiv

Unified models aim to support both understanding and generation by encoding images into discrete tokens and processing them alongside text within a single autoregressive framework. This unified design offers architectural simplicity and cross-modal synergy, which facilitates shared parameterization, consistent training objectives, and seamless transfer between modalities. However, the large number of visual tokens required by such models introduces substantial computation and memory overhead, and this inefficiency directly hinders deployment in resource constrained scenarios such as embodied AI systems. In this work, we propose a unified token compression algorithm UniCompress that significantly reduces visual token count while preserving performance on both image understanding and generation tasks. Our method introduces a plug-in compression and decompression mechanism guided with learnable global meta tokens. The framework is lightweight and modular, enabling efficient integration into existing models without full retraining. Experimental results show that our approach reduces image tokens by up to 4 times, achieves substantial gains in inference latency and training cost, and incurs only minimal performance degradation, which demonstrates the promise of token-efficient unified modeling for real world multimodal applications.

Abstract

Explore similar work

UniCompress: Token Compression for Unified Vision-Language Understanding and Generation

Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation

Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification