Efficient Covariance-Aware Vicinal Latent Diffusion for Continuous Conditional Image Generation

Xin Ding; Shuo Huang; Zhenglong Ding; Sen Zhang; Nenglun Chen; Peibei Cao; Yongwei Wang; Fei Wu

cs.CVOct 29, 2026

Efficient Covariance-Aware Vicinal Latent Diffusion for Continuous Conditional Image Generation

Authors: Xin Ding, Shuo Huang, Zhenglong Ding, Sen Zhang, Nenglun Chen, Peibei Cao, Yongwei Wang, Fei Wu

Organizations: School of Artificial Intelligence/School of Future Technology, Nanjing University of Information Science & Technology (NUIST), Nanjing, China · College of Media, Zhejiang University, Hangzhou, China · College of Artificial Intelligence, Zhejiang University, Hangzhou, China

Abstract

Continuous Conditional Generative Modeling (CCGM) aims to synthesize high-dimensional data conditioned on continuously valued scalar labels. However, existing continuous diffusion models face severe data sparsity under imbalanced and long-tailed label distributions, which can lead to visual artifacts and condition inconsistency. Moreover, inference-time classifier-free guidance and dense pixel-space integration introduce substantial sampling overhead, limiting practical deployment. To address these challenges, we propose CAViLD, an efficient latent diffusion framework for continuous conditional generation. CAViLD first constructs a purified condition-aware latent space through density-adaptive vicinal learning, auxiliary semantic regularization, and NIQE-guided negative data augmentation. It then internalizes the conditional guidance direction through an EMA teacher, enabling single-branch conditional inference, and introduces covariance-aware progressive distillation for label-dependent probability-flow ODEs, supporting sampling down to four NFEs. Experiments on RC-49, UTKFace, and Steering Angle demonstrate a favorable trade-off among generation fidelity, continuous condition consistency, and sampling efficiency.

Figures & tables

Fig. 1: Quality–efficiency comparison on Steering Angle at $64\times 64$ . The axes denote sampling throughput and SFID, while marker size indicates aggregate peak GPU memory. CAViLD achieves a favorable trade-off among the evaluated diffusion-based CCGM methods.

Fig. 2: Overview of CAViLD. Yellow blocks denote adapted or integrated components, while orange blocks denote the proposed components. Unhighlighted blocks represent standard pipeline elements.

Dataset	Condition	Framework	Method	SFID $\downarrow$	NIQE $\downarrow$	Diversity $\uparrow$	LS $\downarrow$	img/sec $\uparrow$	Mem(GiB) $\downarrow$
RC-49 (64 $\times$ 64)	Text	Diffusion	SD 3 Medium (ICML’24) [ 32 ]	1.215	1.996	2.272	28.726	1.90	19.74
			FLUX.1 (arXiv’25) [ 37 ]	0.515	2.330	2.512	12.751	3.87	42.08
			Qwen-Image (arXiv’25) [ 38 ]	1.132	2.436	2.470	27.757	9.58	56.91
			MIRO (ICML’26) [ 39 ]	2.956	3.146	0.791	10.66	1.19	6.53
	Continuous	GAN	CcGAN (T-PAMI’23) [ 2 ]	0.126	1.809	3.451	2.655	6243.06	2.62
			Dual-NDA (AAAI’24) [ 11 ]	0.148	1.808	3.344	2.211	6243.06	2.62

TABLE I: Quantitative comparison of image generation performance under low-to-medium resolutions (

\leq 128\times 128

). Arrows (

\uparrow/\downarrow

) present the preferred optimization directions. For the primary quality metric (SFID), the best results are highlighted in bold and the second-best in underlined . Evaluation profiles corrupted by severe training pathologies (e.g., severe label inconsistency or mode collapse) are denoted in red .

Dataset	Condition	Framework	Method	SFID $\downarrow$	NIQE $\downarrow$	Diversity $\uparrow$	LS $\downarrow$	img/sec $\uparrow$	Mem(GiB) $\downarrow$
UTKFace (192 $\times$ 192)	Text	Diffusion	SD 3 Medium (ICML’24) [ 32 ]	1.682	2.436	0.383	5.188	1.48	20.98
			FLUX.1 (arXiv’25) [ 37 ]	0.771	2.499	0.602	9.191	3.96	42.08
			Qwen-Image (arXiv’25) [ 38 ]	1.617	1.909	0.943	9.868	9.64	62.43
			MIRO (ICML’26) [ 39 ]	3.774	2.197	1.036	11.026	1.19	6.28
	Continuous	GAN	CcGAN (T-PAMI’23) [ 2 ]	0.499	1.661	1.207	7.885	617.28	18.18
			Dual-NDA (AAAI’24) [ 11 ]	0.487	1.483	1.201	6.730	617.28	18.18

TABLE II: Average quality and efficiency of high-resolution generated images (

\geq 192\times 192

). Arrows (

\uparrow/\downarrow

) indicate the preferred optimization directions. For the primary quality metric (SFID), the best results are highlighted in bold and the second-best in underlined . Evaluation profiles corrupted by severe training pathologies (e.g., severe label inconsistency or mode collapse) are denoted in red .

Fig. 3: Effect of NDA on CVAE reconstruction quality. CVAE+NDA achieves lower NIQE across most steering-angle conditions.

Configurations	NFEs $\downarrow$	img/sec $\uparrow$	SFID $\downarrow$	LS $\downarrow$
Baseline (iCCDM) [ 16 ]	64	12.82	0.501	4.891
+ CVAE (no $\mathcal{L}_{aux}$ )	64	26.83	0.270	5.805
+ Aux. Regularization	64	23.46	0.257	5.504
+ Model-Guidance (MG)	32	44.62	0.272	3.950
+ Distillation (16)	16	81.84	0.412	5.246
+ Distillation (8)	8	139.66	0.369	4.038

TABLE III: Ablation of core components and acceleration strategies on the Steering Angle (

64\times 64

) dataset. ‘NFEs’ denotes the number of function evaluations, and ‘img/sec’ denotes sampling throughput.

Dataset	Guidance	NFEs $\downarrow$	img/s $\uparrow$	SFID $\downarrow$	Div. $\uparrow$	LS $\downarrow$
Steering	CFG	64	23.46	0.257	1.072	5.504
	MG (Ours)	32	44.62	0.272	1.053	3.950
RC-49	CFG	64	38.66	0.045	3.708	1.040
	MG (Ours)	32	59.34	0.049	3.697	0.944
UTKFace	CFG	64	36.60	0.360	1.154	5.219
	MG (Ours)	32	54.13	0.350	1.217	5.195

TABLE IV: MG vs. CFG under 32-step sampling.

Dataset	Configuration	SFID $\downarrow$	NIQE $\downarrow$
UTKFace	Baseline (w/o NDA)	0.286	1.422
	NDA	0.275	1.180
Steering Angle	Baseline (w/o NDA)	0.371	2.817
	NDA	0.316	2.584

TABLE V: NDA ablation for

128\times 128

latent diffusion.

Dataset	Method	NFEs $\downarrow$	img/s $\uparrow$	SFID $\downarrow$	Div. $\uparrow$	LS $\downarrow$
Steering Angle	LCM	4	472.64	0.886	0.923	8.257
	PCM	4	368.61	0.695	0.909	4.343
	Ours-2	2	840.36	0.576	1.133	12.039
	Ours-4	4	609.76	0.358	1.048	5.596
RC-49	LCM	4	766.35	0.151	3.498	2.074
	PCM	4	1107.22	0.469	2.714	1.626

TABLE VI: Cross-dataset comparison on

64\times 64

benchmarks. NFEs indicate the number of function evaluations.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 4: Label consistency comparison between the standard CVAE and the proposed Vicinal CVAE on Steering Angle. Vicinal sampling achieves lower Label Scores across most conditions, particularly in sparse extreme-angle regions.

Dataset	Configuration	SFID $\downarrow$	LS $\downarrow$	Diversity $\uparrow$
Steering Angle	CVAE w/o vicinal	0.260	6.890	1.065
	CVAE w/ vicinal	0.255	5.749	1.023
	CVAE w/ vicinal & $\mathcal{L}_{\mathrm{aux}}$	0.257	5.504	1.072

Appendix

TABLE VII: Ablation of vicinal learning and auxiliary regularization in the

64\times 64

CVAE on Steering Angle.

Conditional Image Generation

Conditional image generation creates images guided by inputs such as text, sketches, layouts, masks, reference images, or physical measurements.

See all 162 papers

Imbalance-Robust and Sampling-Efficient Continuous Conditional GANs via Adaptive Vicinal Learning and Auxiliary Regularization
Aug 3, 2025Xin Ding, Yun Chen, Yongwei Wang +4Conditional Generative Modeling Conditional Generation
School of Artificial Intelligence/School of Future Technology, Nanjing University of Information Science & Technology, Nanjing, China · Perceptual and Generative AI Lab, Nanjing University of Information Science & Technology, Nanjing, China · College of Media, Zhejiang University, Hangzhou, China
Semi-Supervised Conditional Diffusion via Label Augmentation
Jul 18, 2026Jin Su, Yuan Gao, Yong Zhou +1Data Augmentation Semi-Supervised Learning
aDepartment of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China · bSchool of Statistics and Data Science, Nankai University, Tianjin, China · cSchool of Statistics, East China Normal University, Shanghai, China +1
CoFi-UCGen: Coarse-to-Fine Unsupervised Conditional Generation without Label Priors
Jun 4, 2026Shengxi Li, Zhaokun Hu, Ce Zheng +3Disentangled Representation Learning Conditional Image Generation
Department of Electronic Information Engineering, Beihang University, Beijing 100191, China · School of Cyber Science and Technology, Beihang University, Beijing 100191, China · College of Electronic Science, National University of Defense Technology, Hunan 410073, China +1

Aug 3, 2025J/K move · Enter open · S save

Imbalance-Robust and Sampling-Efficient Continuous Conditional GANs via Adaptive Vicinal Learning and Auxiliary Regularization

Xin Ding, Yun Chen, Yongwei Wang, Kao Zhang+3

School of Artificial Intelligence/School of Future Technology, Nanjing University of Information Science & Technology, Nanjing, China · Perceptual and Generative AI Lab, Nanjing University of Information Science & Technology, Nanjing, China · College of Media, Zhejiang University, Hangzhou, China

Conditional Generative Modeling Conditional Generation Generative Adversarial Networks

Open paper AskPDF arXiv

Recent advances in continuous conditional generative modeling, including Continuous conditional Generative Adversarial Network (CcGAN) and Continuous Conditional Diffusion Model (CCDM), estimate high-dimensional data distributions conditioned on scalar regression labels such as angles, ages, or temperatures. However, fixed-size vicinal training in CcGAN can be sensitive to non-uniform label densities, whereas CCDM relies on computationally expensive iterative sampling. To address these issues, we propose CcGAN-AVAR, an imbalance-aware extension of CcGAN that combines soft/hybrid adaptive vicinity with auxiliary discriminator-guided regularization. The adaptive vicinity constructs a label-dependent local radius according to the available samples around each target condition, and the multi-task discriminator supplies both a regression signal for label consistency and a density-ratio-estimation signal for distribution matching. We further provide a theoretical interpretation characterizing how adaptive vicinal weighting affects the local bias-variance behavior of the discriminator target, how hybrid truncation reduces objective-level cross-condition mixing, and how the density-ratio-based generator penalty approximates a Pearson Chi-square discrepancy up to the estimation error of the density-ratio branch. Extensive experiments on four datasets, including the newly constructed imbalanced RC-49-I, covering resolutions from 64x64 to 256x256 across eleven settings, demonstrate that CcGAN-AVAR obtains strong generation quality and label consistency while preserving the one-step sampling efficiency of GANs, achieving 300x--2000x faster inference than CCDM.

Abstract

Conditional Image Generation

Explore similar work

Imbalance-Robust and Sampling-Efficient Continuous Conditional GANs via Adaptive Vicinal Learning and Auxiliary Regularization

Semi-Supervised Conditional Diffusion via Label Augmentation

CoFi-UCGen: Coarse-to-Fine Unsupervised Conditional Generation without Label Priors