Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation

Boyu Han; Qianqian Xu; Shilong Bao; Zhiyong Yang; Ruochen Cui; Xilin Zhao; Qingming Huang

cs.CVMar 5, 2026

Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation

Authors: Boyu Han, Qianqian Xu, Shilong Bao, Zhiyong Yang, Ruochen Cui, Xilin Zhao, Qingming Huang

Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · School of Computer Science and Tech., University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence · Institute of Information Engineering, CAS · School of Cyber Security, University of Chinese Academy of Sciences · School of Computer Science and Technology, Beijing Institute of Technology

Abstract

The limited understanding capacity of the visual encoder in Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance. This capacity includes both Discriminative Ability (D-Ability), which reflects class separability, and Detail Perceptual Ability (P-Ability), which focuses on fine-grained visual cues. Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens. We argue that such paradigms may compromise D-Ability and therefore fail to effectively address CLIP's representation limitations. To address this, we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations. We begin with a straightforward design that augments the diffusion process with contrastive learning on input images. However, empirical results show that the naive combination suffers from gradient conflict and yields suboptimal performance. To balance the optimization, we introduce the Diffusion Contrastive Reconstruction (DCR), which unifies the learning objective. The key idea is to inject contrastive signals derived from each reconstructed image, rather than from the original input, into the diffusion process. Our theoretical analysis shows that the DCR loss can jointly optimize D-Ability and P-Ability. Extensive experiments across various benchmarks and multi-modal large language models validate the effectiveness of our method. The code is available at https://github.com/boyuh/DCR.

Figures & tables

Figure 1: (a) Contrastive learning for D-Ability . (b) Reconstructive learning for P-Ability . (c) Our Diffusion Contrastive Reconstruction (DCR) for harmonizing D-Ability and P-Ability . (d) Performance overview of DCR and other methods on the OpenAI CLIP ViT-L@224 backbone.

Panel of Figure 2 — Figure 2: Training results of linearly combining contrastive learning and reconstructive learning (Naive Method) on the OpenAI CLIP ViT-L@224 backbone. The contrastive loss $\mathcal{L}_{\text{con}}$ dominates the gradients, and most training steps exhibit gradient conflicts.

Figure 3: An overview of Diffusion Contrastive Reconstruction (DCR). An image is encoded by CLIP and projected into the diffusion condition space. Predicted noises from original, augmented, and negative samples form a contrastive triplet in the reconstruction image space. Training proceeds in two stages: projector alignment and encoder enhancement.

CLIP Backbone	Resol.	#Params	Method										Avg
OpenAI ViT-L-14	224	427.6M	Original	13.3	13.3	20.0	20.0	13.3	53.3	20.0	6.7	13.3	19.2
			DIVA	13.3	20.0	40.0	6.7	20.0	53.3	46.7	20.0	13.3	25.9
			GenHancer	13.3	33.3	33.3	20.0	6.7	73.3	46.7	20.0	40.0	31.8
			un 2 CLIP	0.0	33.3	46.7	26.7	13.3	80.0	40.0	20.0	33.3	32.6
			Ours	13.3	33.3	33.3	26.7	6.7	73.3	46.7	20.0	46.7	33.3
OpenAI ViT-L-14	336	427.9M	Original	0.0	20.0	40.0	20.0	6.7	20.0	33.3	6.7	33.3	20.0

Table 1: Performance of Detail Perceptual Ability (P-Ability) on the MMVP-VLM benchmark. Results of baseline methods are taken from [ 77 , 57 , 42 ] . Our method outperforms across multiple CLIP backbones and exhibits robustness across various visual patterns. The visual patterns are symbolized as: : Orientation and Direction, : Presence of Specific Features, : State and Condition, : Quantity and Count, : Positional and Relational Context, : Color and Appearance, : Structural and Physical Characteristics, : Texts, : Viewpoint and Perspective.

Backbone	Method	MNIST [ 33 ]			CIFAR-10 [ 32 ]			Eurosat [ 22 ]			Caltech-101 [ 13 ]			DTD [ 7 ]			ImageNet-1K [ 10 ]			Avg
Backbone	Method	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
O-1	Original	0.63	0.63	0.50	0.82	0.82	0.76	0.71	0.72	0.61	0.78	0.53	0.35	0.55	0.45	0.30	0.79	0.53	0.41	0.71	0.61	0.49
	DIVA	0.66	0.60	0.49	0.81	0.82	0.75	0.70	0.72	0.61	0.78	0.52	0.34	0.51	0.41	0.26	0.79	0.52	0.40	0.71	0.60	0.48
	GenHancer	0.69	0.73	0.60	0.58	0.65	0.49	0.61	0.67	0.56	0.81	0.54	0.38	0.56	0.45	0.31	0.64	0.26	0.14	0.65	0.55	0.41
	un 2 CLIP	0.67	0.70	0.53	0.81	0.82	0.74	0.64	0.75	0.60	0.80	0.55	0.41	0.55	0.46	0.31	0.73	0.41	0.28	0.70	0.62	0.48
	Ours	0.73	0.74	0.62	0.89	0.86	0.80	0.68	0.73	0.61	0.83	0.57	0.43	0.58	0.48	0.33	0.84	0.66	0.45	0.76	0.67	0.54

Table 2: Performance of Discriminative Ability (D-Ability) on 6 standard zero-shot clustering benchmarks. Our method achieves better class separability. O-1, M-1, and S-1 separately represent OpenAI CLIP ViT-L@224, MetaCLIP ViT-L@224, and SigLIP ViT-SO@224.

Figure 4: Qualitative results of P-Ability on the MMVP-VLM benchmark. The predictions from the original CLIP and our improved version are indicated by red and green arrows, respectively. The improved CLIP effectively addresses the original model’s limitations in capturing fine-grained visual details.

Panel of Figure 5 — Figure 5: Qualitative results of D-Ability on the MNIST benchmark by using the t-SNE method. The improved CLIP achieves better class separability.

LLM	CLIP	Vision-Centric Benchmarks								Conventional MLLM Benchmarks
		MMVP- MLLM [ 75 ]	NaturalBench [ 36 ]				CV-Bench 2D [ 74 ]		CV-Bench 3D [ 74 ]	POPE [ 41 ]			SciQA- IMG [ 53 ]	Hallusion Avg. [ 17 ]
		MMVP- MLLM [ 75 ]	Acc	Q-Acc	I-Acc	G-Acc	ADE20K	COCO	CV-Bench 3D [ 74 ]	rand	pop	adv	SciQA- IMG [ 53 ]	Hallusion Avg. [ 17 ]
Vicuna-7B	Original	24.7	76.4	53.6	56.4	17.6	49.6	60.9	58.7	87.3	86.1	84.2	66.8	27.6
	DIVA	31.3	75.3	51.7	56.1	22.3	51.3	63.4	60.2	87.9	87.0	84.6	66.3	28.6
	GenHancer	30.7	77.3	55.6	59.1	24.4	52.9	63.6	63.2	88.1	86.7	84.6	66.5	28.4
	un 2 CLIP	31.3	-	-	-	-	53.9	65.1	61.2	88.0	87.4	85.4	68.4	28.4

Table 3: Performance of MLLMs (LLaVA-1.5 [ 48 ] ) on various benchmarks. The champion and the runner-up are highlighted in bold and underline . Results on NaturalBench follow the setting in [ 57 ] , which differs from that used in un 2 CLIP [ 42 ] , leading to the missing entries.

Table 4: Ablation studies in terms of P-Ability and D-Ability .

Figure 6: Ablation study on different diffusion model structures.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Notations	Descriptions
$\mathbf{x}\in\mathbb{R}^{H\times W\times 3}$	Input image.
$\mathcal{D}=\{\mathbf{x}_{i}\}_{i=1}^{n}$	Training dataset.
$\tilde{\mathbf{x}}$	VAE latent of image $\mathbf{x}$ .
$t\in\{1,\dots,T\}$	Diffusion timestep.
$\mathbf{x}_{t}$	Noisy latent at diffusion step $t$ .
$\epsilon_{\theta}$	Diffusion noise prediction network.

Appendix

Table 5: A summary of key notations and descriptions in this work.

Method	Classification						Retrieval-Image@5		Retrieval-Text@5
Method	MNIST	C10	Eur	C101	DTD	IN-1K	Flickr30k	COCO	Flickr30k	COCO
Original	76.4	95.6	60.1	86.6	55.4	75.5	87.2	61.1	97.4	79.2
Genhancer	69.7	73.7	58.5	71.5	48.4	73.8	81.6	51.1	87.3	61.4
Ours	76.5	95.6	60.3	86.7	55.4	75.6	87.3	61.1	97.3	79.3

Appendix

Table 6: Performance on zero-shot classification and retrieval.

Ratio	MMVP-VLM	Clustering
Ratio	ACC	NMI	ACC	ARI
0%	33.30	0.76	0.67	0.54
10%	31.85	0.76	0.63	0.53
50%	23.70	0.73	0.60	0.46
80%	21.48	0.69	0.57	0.44
100%	20.74	0.65	0.55	0.42

Appendix

Table 7: Results under different ratios of local tokens ( [CLS] +

n\%

local tokens).

Figure 7: Expanded version of qualitative results on the MMVP-VLM benchmark. The predictions from the original CLIP and our improved version are indicated by red and green arrows, respectively. The improved CLIP effectively addresses the original model’s limitations in capturing fine-grained visual details.

Panel of Figure 8 — Figure 8: Expanded version of qualitative results on the MLLM benchmarks.

CLIP Backbone	Training #Params	Training Time	Training GPU Memory
OpenAI CLIP ViT-L@224	350.5M	7.7h	53544M
OpenAI CLIP ViT-L@336	350.5M	7.6h	60472M
MetaCLIP ViT-L@224	350.5M	7.7h	60472M
MetaCLIP ViT-H@224	360.0M	9.2h	76430M
SigLIP ViT-SO@224	358.5M	8.0h	66798M
SigLIP ViT-SO@384	358.5M	9.7h	78792M

Appendix

Table 8: Training costs of our DCR.

Explore similar work

Probing Diffusion Denoising Dynamics for Contrastive Representation Learning
Jul 10, 2026Yasong Dai, Zeeshan Hayder, David Ahmedt-Aristizabal +1Contrastive Learning Diffusion Model Fine-Tuning
Australian National University · CSIRO Data61 · Amazon
AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization
Jul 15, 2026Yiyang Yao, Shanglin Liu, Jianming Lv +4Vision-Language Alignment Multimodal Contrastive Learning
School of Computer Science and Technology, South China University of Technology, Guangzhou, China
Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder
Oct 8, 2026Zidan Wang, Yaqian Li, Xiaokai Zhang +3VLM Adaptation Vision-Language Models
Huazhong University of Science and Technology · Li Auto Inc.

Jul 10, 2026J/K move · Enter open · S save

Probing Diffusion Denoising Dynamics for Contrastive Representation Learning

Yasong Dai, Zeeshan Hayder, David Ahmedt-Aristizabal, Hongdong Li

Australian National University · CSIRO Data61 · Amazon

Contrastive Learning Diffusion Model Fine-Tuning Diffusion Models Image Classification

Open paper AskPDF arXiv

Text-to-image diffusion models exhibit unprecedented generative capability and contain rich intermediate representations that can be useful for discriminative vision tasks. Motivated by this observation, we study a focused question: how can the denoising dynamics of a pretrained diffusion model be adapted to support discriminative representation learning while preserving its generative behavior under parameter-efficient updates? We present D

^3

CL as an investigation of this question. Our key observation is that noisy latents at different diffusion timesteps can be interpreted as stochastic views of the same underlying image, enabling a contrastive objective to be coupled with the standard denoising reconstruction loss. This formulation provides a simple way to probe the interaction between generative denoising and discriminative representation learning without training from scratch. To keep the adaptation lightweight, we apply LoRA updates to a pretrained Stable Diffusion backbone while freezing the original model parameters. D

^3

CL provides strong empirical evidence that reconstruction and noise-level contrastive objectives can be complementary: on ImageNet-1K, it obtains 80.1% linear-probing accuracy and an FID of 5.56 for

256 \times 256

unconditional generation. Additional ablations on the design space suggest that the usefulness of diffusion features depends on where and how denoising states are sampled. These results establish D

^3

CL as a parameter-efficient adaptation framework for pretrained diffusion models, showing that noise-level contrastive learning can structure denoising representations for discriminative tasks while maintaining generative performance.

Abstract

Explore similar work

Probing Diffusion Denoising Dynamics for Contrastive Representation Learning

AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder