CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension

Lihao Liu; Yan Wang; Biao Yang; Da Li; Jiangxia Cao; Yuxiao Luo; Xiang Chen; Xiangyu Wu

cs.CVFeb 22, 2026

CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension

Authors: Lihao Liu, Yan Wang, Biao Yang, Da Li, Jiangxia Cao, Yuxiao Luo, Xiang Chen, Xiangyu Wu,

Organizations: Tsinghua University · Kuaishou Technology

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable success in comprehension tasks such as visual description and visual question answering. However, their direct application to embedding-based tasks like retrieval remains challenging due to the discrepancy between output formats and optimization objectives. Previous approaches often employ contrastive fine-tuning to adapt MLLMs for retrieval, but at the cost of losing their generative capabilities. We argue that both generative and embedding tasks fundamentally rely on shared cognitive mechanisms, specifically cross-modal representation alignment and contextual comprehension. To this end, we propose CREM (Compression-driven Representation Enhanced Model), with a unified framework that enhances multimodal representations for retrieval while preserving generative ability. Specifically, we introduce a compression-based prompt design with learnable chorus tokens to aggregate multimodal semantics and a compression-driven training strategy that integrates contrastive and generative objectives through compression-aware attention. Extensive experiments demonstrate that CREM achieves state-of-the-art retrieval performance on MMEB while maintaining strong generative performance on multiple comprehension benchmarks. Our findings highlight that generative supervision can further improve the representational quality of MLLMs under the proposed compression-driven paradigm.

Figures & tables

Figure 1: Comparison of Different Paradigms. (a) Embedding models fall short on generation tasks. (b) Generative models lack retrieval capability. (c) Our proposed model CREM enables both in a single model.

Figure 2: Compression-Driven Training Framework of CREM. (a) The training pipeline integrates chorus tokens with contrastive and generative objectives under a unified prompt design equipped with compression-aware attention. Generation instructions and answers originate from diverse data sources. (b) Compression-aware attention mask enforcing token-level visibility constraints, where “+” indicates visible tokens and “–” indicates masked ones. (c) Two mixing strategies for generation training using different data sources. Homogeneous data are pseudo-labeled by an MLLM from retrieval pairs, whereas heterogeneous data are collected from open-source datasets.

Figure 3: CREM Inference Modes. (a) Retrieval embeddings are derived from pooled chorus tokens. (b) Native next-token prediction is performed with full access to all vision tokens ( Nat. ). (c) Efficient inference is achieved by pruning vision tokens and reducing KV caches ( Comp. ).

< 3B Models
Model	Backbone	#Params	Per Meta-Task Score				Average Score
Model	Backbone	#Params	Classification	VQA	Retrieval	Grounding	IND	OOD	Overall
# of Datasets $\to$			10	10	12	4	20	16	36
CLIP [ 41 ]	-	0.4B	55.2	19.7	53.2	62.2	47.6	42.8	45.4
OpenCLIP [ 10 ]	-	0.4B	56.0	21.9	55.4	64.1	50.5	43.1	47.2
VLM2Vec [ 21 ]	Qwen2-VL	2B	59.0	49.4	65.4	73.4	66.0	52.6	59.3

Table 1: Results on MMEB. “IND” denotes in-distribution datasets, while “OOD” refers to out-of-distribution datasets. Reported scores are the average Precision@1 across the respective dataset groups. The highest score in each column is shown in bold, and the second-best result is underlined.

Model	MMB	MMVet	AI2D	Hallusion	MMMU	MMStar	AVG
2B Models
Qwen2-VL	72.3	45.7	73.8	41.9	41.1	46.1	53.5
CREM G	73.1	44.5	72.9	41.0	41.2	46.2	53.2
CREM R	64.3	17.3	66.7	33.6	34.9	43.9	43.4
CREM	72.5	45.1	72.8	41.2	41.4	45.5	53.1
7B Models

Table 2: Results on Multimodal Comprehension Benchmarks. The comprehension benchmarks are used to evaluate the multimodal generative capability of different models. CREM G refers to Qwen2-VL fine-tuned solely on ShareGPT-4V datasets using the standard generative pipeline, while CREM R denotes Qwen2-VL trained exclusively on MMEB following the VLM2Vec [ 21 ] . “AVG” denotes the average score across all benchmarks. All evaluations are conducted using the same image resolution as in the retrieval experiments.

Figure 4: Visualization of Chorus Token Attention. We visualize the attention weights from chorus tokens to vision tokens. Each chorus token is assigned a unique color, and each vision token is colored based on its most attended chorus token, with color intensity reflecting attention strength.

Ret.	Gen.	CPD	CTS	Generation		Retrieval
Ret.	Gen.	CPD	CTS	Nat.	Comp.	MMEB
✓				43.4	-	62.3
	✓			53.2	-	2.9
✓		✓		47.2	-	66.1
	✓	✓	✓	53.0	43.9	21.1
✓	✓	✓		52.8	-	65.5

Table 3: Analysis on Retrieval and Generation. “Ret.” denotes training on retrieval datasets; “Gen.” denotes training on generation datasets. “CPD” indicates the compression-based prompt design, and “CTS” refers to the compression-driven training strategy.

CTok.	MMEB	Nat.	Comp.	Cache(%)
EOS	62.3	43.4	-	100%
1	65.6	51.9	38.2	0.07%
4	66.1	52.3	41.7	0.28%
8	66.5	52.5	43.1	0.56%
16	66.7	53.1	44.2	1.12%
32	66.7	52.9	44.6	2.24%

Table 4: Analysis on Different Chorus Token Design. “CTok.” refers to the type or the number of chorus token. “Cache” denotes the KV cache ratio during decoding, with 100% corresponds to full caching (

\sim

1280 tokens).

HMD	HTD	Generation		Retrieval
HMD	HTD	Nat.	Comp.	MMEB
		47.2	-	66.1
✓		49.8	42.5	66.5
	✓	53.0	44.0	66.2
✓	✓	53.1	44.2	66.7

Table 5: Analysis on Generation Data Mixing. “HMD” denotes training with homogeneous data, which are pseudo-labeled from retrieval pairs, while “HTD” denotes training with heterogeneous data collected from open-source datasets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Retrieval	Generation
Training Samples	1.1M	665K
Batch Size	1024	128
Resize Tokens	(256, 1280)
Learning rate	$5\times 10^{-5}$
Optimizer	AdamW
Learning Rate Decay	cosine

Appendix

Table 6: Training Hyperparameters and Computational Requirements for Retrieval and Generation Tasks.

Models	Short Caption Retrieval				Long Caption Retrieval				Compositional Retrieval
	Flickr30K		COCO		ShareGPT4V		Urban1K		SugarCrepe
	$q^{t}\rightarrow c^{i}$	$q^{i}\rightarrow c^{t}$	$q^{t}\rightarrow c^{i}$	$q^{i}\rightarrow c^{t}$	$q^{t}\rightarrow c^{i}$	$q^{i}\rightarrow c^{t}$	$q^{t}\rightarrow c^{i}$	$q^{i}\rightarrow c^{t}$	Replace	Swap	Add
VLM2Vec [ 21 ]	76.0	90.6	46.8	66.6	85.8	90.7	84.7	90.8	85.8	66.3	86.5
UniME [ 16 ]	83.3	94.4	54.8	74.0	93.9	89.3	94.3	95.5	80.5	65.5	82.2
CREM 2B	81.4	91.9	55.8	71.6	90.3	89.4	94.0	94.3	87.9	66.1	91.5
CREM 7B	82.9	95.5	56.4	76.1	90.7	90.8	97.6	98.0	88.7	74.0	91.7

Appendix

Table 7: Zero-shot cross-domain retrieval performance. Results are reported as average Recall@1 across short-caption, long-caption, and compositional retrieval benchmarks. The best results are in bold.

$\alpha_{r}$	$\alpha_{g}$	$p$	Generation		Retrieval
$\alpha_{r}$	$\alpha_{g}$	$p$	Nat.	Comp.	MMEB
1	1	1	52.4	45.3	66.4
1	1	0.5	53.1	44.5	66.3
1	1	0	52.9	-	65.3
1	0.5	0.5	53.1	44.2	66.7
0.5	1	0.5	53.2	45.0	65.5

Appendix

Table 8: Ablation on Loss Weights and Compression Probability.

\alpha_{r}

and

\alpha_{g}

denote the loss weights for retrieval and generation objectives, respectively, while

p

represents the probability of applying compression-driven generation.

Pooling	CLS	QA	RET	GD	Overall
MLP	64.6	60.7	67.6	76.0	65.8
Attention	63.6	59.4	67.7	76.7	65.3
Mean	65.8	60.7	68.3	78.9	66.7

Appendix

Table 9: Ablation on Pooling Methods. Evaluation of various pooling strategies used to aggregate chorus tokens for retrieval. “CLS”: classification, “QA”: question answering, “RET”: retrieval, “GD”: grounding.

Meta-Task	Dataset	Query $\to$ Target	Distribution Type	#Training	#Eval	#Candidates
Classification (10 Tasks)	ImageNet-1K	I $\to$ T	IND	100K	1000	1000
	N24News	I + T $\to$ I	IND	49K	1000	24
	HatefulMemes	I $\to$ T	IND	8K	1000	2
	VOC2007	I $\to$ T	IND	8K	1000	20
	SUN397	I $\to$ T	IND	20K	1000	397
	Place365	I $\to$ T	OOD	-	1000	365

Appendix

Table 10: Details of Retrieval Data. MMEB consists of 36 datasets across four meta-task categories. Of these, 20 in-distribution datasets are used for training and 16 out-of-distribution datasets are reserved for evaluation.

	CLIP	VLM2Vec	MMRet	UniME	mmE5	UNITE	CREM 2B	CREM 7B
Classification (10 tasks)
ImageNet-1K	55.8	74.5	58.8	71.3	77.8	80.2	82.6	83.6
N24News	34.7	80.3	71.3	79.5	81.7	80.3	77.8	81.2
HatefulMemes	51.1	67.9	53.7	64.6	64.2	67.1	62.1	65.0
VOC2007	50.7	91.5	85.0	90.4	91.0	84.9	80.8	85.6
SUN397	43.4	75.8	70.0	75.9	77.7	78.7	77.5	77.3

Appendix

Table 11: Detailed MMEB Results. Performance of baseline models and our CREM across 20 in-distribution (IND) and 16 out-of-distribution (OOD) datasets. OOD datasets are highlighted with a yellow background. For each baseline, we report the strongest variant with complete evaluation metrics: VLM2Vec 7B (LLaVA-1.6), MMRet 7B (LLaVA-1.6), UniME 7B (LLaVA-1.6), mmE5 11B (Llama-3.2-Vision), and UNITE 7B (Qwen2-VL).

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,073 papers

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings
Apr 24, 2026Peixi Wu, Ke Mei, Feipeng Ma +15Cross-Modal Alignment Multimodal Embedding
WeChat Vision, Tencent Inc. · Zhejiang University · Tsinghua University +1
TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval
Mar 3, 2026Xiangzhao Hao, Shijie Wang, Tianyu Yang +3Multimodal IR Multimodal Embedding
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences
PLUME: Latent Reasoning Based Universal Multimodal Embedding
Apr 2, 2026Chenwei He, Xiangzhao Hao, Tianyu Yang +6Multimodal Embedding Multimodal IR
Southeast University · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Apr 24, 2026J/K move · Enter open · S save

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

Peixi Wu, Ke Mei, Feipeng Ma, Bosong Chai+14

WeChat Vision, Tencent Inc. · Zhejiang University · Tsinghua University+1

Cross-Modal Alignment Multimodal Embedding Cross-Modal Retrieval

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks. However, Chain-of-Thought (CoT) reasoning tends to generate redundant thinking steps and introduce semantic ambiguity in the summarized answers in broader retrieval scenarios. To address this limitation, we propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimizes generation and embedding through a retrieval-friendly rewrite. Meanwhile, we present the Cross-Mode Alignment (CMA) to bridge the generative and discriminative embedding spaces, enabling flexible mutual retrieval to trade off efficiency and accuracy. Based on this, we also introduce Refine Reinforcement Learning (Refine-RL) that treats discriminative embeddings as stable semantic anchors to guide the rewrite optimization. Extensive experiments on MMEB-V2, MRMR and UVRB demonstrate that RIME substantially outperforms prior generative embedding models while significantly reducing the length of thinking. Code is available at https://github.com/PeppaWu/RIME.

Abstract

Multimodal Large Language Models

Explore similar work

Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval

PLUME: Latent Reasoning Based Universal Multimodal Embedding