Overcoming Spurious Correlations in Conditional Diffusion Models via Doubly Robust Target Correction
Authors: Eiichi Sakurai, Kenji Yamanishi
Organizations: Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology, Tokyo, Japan
While conditional diffusion models have achieved remarkable success in generation tasks, they memorize spurious correlations within training data. This causes them to deviate from ideal target distributions. Minimizing a covariate shift-adjusted target risk can address this issue. However, retraining models from scratch for different distributions is computationally prohibitive. To overcome this limitation, we propose DR-Diff, a framework that efficiently fine-tunes pre-trained conditional diffusion models. Specifically, DR-Diff leverages a functional gradient approach to derive a pseudo-target. This pseudo-target captures the residual between the base model's noise prediction and the ground-truth noise under the target distribution. Training the model to fit this pseudo-target enables efficient target adaptation. Theoretically, we prove that DR-Diff guarantees target risk reduction at the population level. It also achieves a pseudo-doubly robust property, providing resilience against density ratio estimation errors and base model misspecifications. Empirically, DR-Diff matches the noise prediction accuracy of models trained from scratch on synthetic datasets. Furthermore, it mitigates inter-group disparities in downstream classification tasks.
Figures & tables
Fig. 1: Spurious correlation and OOD targets in the evaluation datasets. (Top Left) The marginal distributions of Xcond given each class label in the Causal Cluster dataset. The target condition for Class 0 (red star) falls into a zero-density region. (Top Right) The 2D density plot of Xcond for Label 0 in the Swiss Roll dataset. The target condition (red star) is situated outside the observed high-density areas. (Bottom Row) The 2D density plots of Xcond for Label 1 (left) and Label 2 (right) in the Swiss Roll dataset, illustrating the conditional distributions for the remaining classes. In both scenarios, the out-of-distribution (OOD) targets serve as an evaluation of generation capabilities.
Fig. 2: Quantitative comparison of SWD across methods for OOD targets. The vertical axis represents the Sliced Wasserstein Distance (SWD) between the true distribution and the empirical distribution obtained from each method under OOD target conditions. The color of each plotted point corresponds to results from models trained on datasets generated with the same seed value. Notably, the proposed method achieves high generation performance comparable to WDDPM through only additional training from the Base model.
Method
Causal Cluster
Swiss Roll
Base model
0.632 / 0.734 / 0.897
4.442 / 4.785 / 5.041
CFG
1.219 / 1.555 / 1.700
3.756 / 4.080 / 4.344
WDDPM
0.619 / 0.698 / 0.810
3.349 / 3.770 / 3.898
DR-Diff(Ours)
0.651 / 0.719 / 0.854
3.386 / 3.661 / 3.871
TABLE I: Quantitative comparison of the SWD across methods for OOD targets. (Values reported as Q1 / Median / Q3).
Fig. 3: Visualization of generated samples for the Swiss Roll dataset. To visualize the underlying data manifold structure, the original R10 space is projected down to 3 dimensions via PCA, upon which the generated points are plotted. (a) The light gray points denote the training data, while the stars represent the ground-truth points generated under the current OOD condition. (b) Generation results from a model yielding a slightly poor SWD of 3.7. (c) Generation results from a model achieving an SWD of 2.12. Both models successfully recover the global roll structure; however, it can be observed that the large SWD values stem from the high variance in estimating the angular velocity of the roll.
Fig. 4: Ablation study evaluating the impact of accurate weight estimation. The vertical axis shows the SWD between the true and empirical distributions for OOD targets. As an ablation, we present the results of replacing the properly estimated w with random weights w∼U(0,1.9) . It is observed that destroying the structure of w limits the improvement in generation quality, preventing it from surpassing the performance of CFG. This confirms that while bounded random weights are theoretically safe, accurately estimating w is critical for the optimal estimation of the noise prediction function.
Method
Causal Cluster
Swiss Roll
Base model
0.616 / 0.645 / 0.840
4.116 / 4.570 / 4.796
CFG
1.209 / 1.466 / 1.828
3.355 / 3.947 / 4.357
WDDPM
0.605 / 0.658 / 0.734
3.853 / 4.113 / 4.616
DR-Diff(Ours)
0.603 / 0.644 / 0.757
3.749 / 4.225 / 4.539
TABLE II: Quantitative comparison of the SWD across methods for OOD targets under the ablation setup.
Metric
Group
Original
OS
RW
Ours
Accuracy
All
0.911
0.902
0.902
0.907
Group0
0.921
0.910
0.922
0.916
Group1
0.755
0.777
0.592
0.777
Gap
0.166
0.133
0.330
0.139
FPR
All
0.763
0.559
0.883
0.635
Group0
0.897
0.638
0.845
0.749
TABLE III: Performance metrics of the classifiers on the LSAC dataset across racial subgroups. ’Gap’ indicates the absolute difference between Group 0 (White) and Group 1 (Non-White) defined by the race attribute.
Metric
Group
Original
OS
RW
Ours
Accuracy
All
0.910
0.905
0.908
0.908
Group0
0.915
0.911
0.914
0.913
Group1
0.758
0.753
0.742
0.762
Gap
0.158
0.158
0.173
0.152
FPR
All
0.029
0.043
0.029
0.030
Group0
0.027
0.039
0.029
0.028
TABLE IV: Performance metrics of the classifiers on the Jigsaw dataset (Target attribute: mentions of the ’Black’ identity). Group 1 represents texts containing mentions of the identity, while Group 0 represents texts without. ’Gap’ indicates the absolute difference between the subgroups.
Metric
Group
Original
OS
RW
Ours
Accuracy
All
0.910
0.905
0.908
0.908
Group0
0.913
0.910
0.912
0.912
Group1
0.878
0.869
0.876
0.872
Gap
0.035
0.041
0.036
0.040
FPR
All
0.029
0.043
0.036
0.030
Group0
0.028
0.040
0.035
0.028
TABLE V: Performance metrics of the classifiers on the Jigsaw dataset (Target attribute: mentions of the ’Male’ identity). Group 1 represents texts containing mentions of the identity, while Group 0 represents texts without. ’Gap’ indicates the absolute difference between the subgroups.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# Samp.
# Feat.
Target ( Xlabel )
Auxiliary Attributes ( Xcond )
LSAC
21,791
12
Bar exam passage
Race, Gender, LSAT score, Undergraduate GPA, Family income level
Conditional independence (CI) is a fundamental concept in statistics and machine learning. Recent advances in conditional generative modeling provide flexible tools for generative-model-based CI tests, which rely on an estimated conditional distribution to generate randomized samples. However, errors in estimating this distribution accumulate in existing Type I error bounds, and consistency of the generative estimator alone does not guarantee asymptotic Type I error control. To address this limitation, we formulate conditional generative modeling as a domain adaptation problem and leverage auxiliary data from multiple source domains to improve estimation in the target CI testing domain. We propose Domain-Adapted Diffusion (DA-Diff), a multi-source domain adaptation framework for conditional diffusion models based on weighted empirical risk minimization over both target and source domains. We establish the convergence rate of DA-Diff and show how transferable source data can improve target-domain estimation through an increased effective sample size while controlling transfer bias. Building on DA-Diff, we further propose Domain-Adapted Conditional Independence Testing (DA-CIT) and show that its Type I error satisfies P(p≤α)≤α+o(1). Experiments demonstrate that DA-Diff improved conditional generation quality compared with transfer-learning diffusion baselines, while DA-CIT provides strong Type I error control and competitive power.
Yanfeng Yang, Junda Zhao, Yijie Gao +8
The Graduate University for Advanced Studies, SOKENDAI, Tokyo, Japan · The Institute of Statistical Mathematics, Tokyo, Japan · East China Normal University, Shanghai, China +4
We establish finite-sample guarantees for conditional diffusion models learned from data and implemented with finitely many denoising steps. Our approach aligns the score-training objective with the sampling grid, allowing statistical estimation and numerical discretization errors to be analyzed together. Under a smooth conditional density model, we derive high-probability bounds on the generated distribution's error and quantify the required number of score evaluations. For every smoothness index β>0, a suitable grid yields averaged conditional Kullback--Leibler (KL) divergence Op(n−2β/(dx+dy+2β)) relative to the original target with or without early stopping. Both the full and early-stopped samplers attain average target total-variation (TV) error Op(n−β/(dx+dy+2β)). Here n is the sample size, dx,dy are the data and conditioning dimensions, and errors are averaged over the conditioning variable. The total-variation bound matches the rate established in existing conditional distribution-estimation theory, up to logarithmic factors.
Qiuyun Zhu
Department of Statistics and Actuarial Science University of Iowa
Iterative sampling in diffusion models can lead to error accumulation due to exposure bias. The denoiser is trained on real images interpolated with noise, but at inference its inputs are constructed from its own imperfect predictions from the earlier steps, something it has never seen at training time. This might be one reason diffusion models often struggle without additional help, such as CFG. We introduce a correction step for diffusion models which attempts to reduce this training-inference gap. The corrector takes the base model's prediction as input and is trained against the same ground-truth target. This makes correction an explicit learning task, with training inputs provided by the base model. During sampling, at each denoising step, we iteratively refine the denoiser's clean-sample prediction. This allows us to correct predictions at intermediate denoising steps, preventing early artifacts from persisting into the final sample. Although the corrector is trained only on predictions from the base denoiser, it can be applied repeatedly to its own outputs at inference for better results. This simple framework leads to substantial improvement in sample quality while largely preserving diversity. We evaluate our method across DiT, SiT, and JiT. Our method substantially improves upon previously reported FID scores for unguided generation and, notably, unconditional generation, where producing high-quality samples without guidance has been particularly challenging. Our method achieves performance competitive with classifier-free guidance (CFG), and combining correction with CFG further improves diversity and reduces oversaturation typical of CFG. Finally, we show that the additional inference cost of correction can be amortized by distilling the refined predictions back into the base diffusion model, retaining the quality gains without additional inference-time overhead.