Overcoming Spurious Correlations in Conditional Diffusion Models via Doubly Robust Target Correction
Authors: Eiichi Sakurai, Kenji Yamanishi
Organizations: Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology, Tokyo, Japan
While conditional diffusion models have achieved remarkable success in generation tasks, they memorize spurious correlations within training data. This causes them to deviate from ideal target distributions. Minimizing a covariate shift-adjusted target risk can address this issue. However, retraining models from scratch for different distributions is computationally prohibitive. To overcome this limitation, we propose DR-Diff, a framework that efficiently fine-tunes pre-trained conditional diffusion models. Specifically, DR-Diff leverages a functional gradient approach to derive a pseudo-target. This pseudo-target captures the residual between the base model's noise prediction and the ground-truth noise under the target distribution. Training the model to fit this pseudo-target enables efficient target adaptation. Theoretically, we prove that DR-Diff guarantees target risk reduction at the population level. It also achieves a pseudo-doubly robust property, providing resilience against density ratio estimation errors and base model misspecifications. Empirically, DR-Diff matches the noise prediction accuracy of models trained from scratch on synthetic datasets. Furthermore, it mitigates inter-group disparities in downstream classification tasks.
Figures & tables
Fig. 1: Spurious correlation and OOD targets in the evaluation datasets. (Top Left) The marginal distributions of Xcond given each class label in the Causal Cluster dataset. The target condition for Class 0 (red star) falls into a zero-density region. (Top Right) The 2D density plot of Xcond for Label 0 in the Swiss Roll dataset. The target condition (red star) is situated outside the observed high-density areas. (Bottom Row) The 2D density plots of Xcond for Label 1 (left) and Label 2 (right) in the Swiss Roll dataset, illustrating the conditional distributions for the remaining classes. In both scenarios, the out-of-distribution (OOD) targets serve as an evaluation of generation capabilities.
Fig. 2: Quantitative comparison of SWD across methods for OOD targets. The vertical axis represents the Sliced Wasserstein Distance (SWD) between the true distribution and the empirical distribution obtained from each method under OOD target conditions. The color of each plotted point corresponds to results from models trained on datasets generated with the same seed value. Notably, the proposed method achieves high generation performance comparable to WDDPM through only additional training from the Base model.
Method
Causal Cluster
Swiss Roll
Base model
0.632 / 0.734 / 0.897
4.442 / 4.785 / 5.041
CFG
1.219 / 1.555 / 1.700
3.756 / 4.080 / 4.344
WDDPM
0.619 / 0.698 / 0.810
3.349 / 3.770 / 3.898
DR-Diff(Ours)
0.651 / 0.719 / 0.854
3.386 / 3.661 / 3.871
TABLE I: Quantitative comparison of the SWD across methods for OOD targets. (Values reported as Q1 / Median / Q3).
Fig. 3: Visualization of generated samples for the Swiss Roll dataset. To visualize the underlying data manifold structure, the original R10 space is projected down to 3 dimensions via PCA, upon which the generated points are plotted. (a) The light gray points denote the training data, while the stars represent the ground-truth points generated under the current OOD condition. (b) Generation results from a model yielding a slightly poor SWD of 3.7. (c) Generation results from a model achieving an SWD of 2.12. Both models successfully recover the global roll structure; however, it can be observed that the large SWD values stem from the high variance in estimating the angular velocity of the roll.
Fig. 4: Ablation study evaluating the impact of accurate weight estimation. The vertical axis shows the SWD between the true and empirical distributions for OOD targets. As an ablation, we present the results of replacing the properly estimated w with random weights w∼U(0,1.9) . It is observed that destroying the structure of w limits the improvement in generation quality, preventing it from surpassing the performance of CFG. This confirms that while bounded random weights are theoretically safe, accurately estimating w is critical for the optimal estimation of the noise prediction function.
Method
Causal Cluster
Swiss Roll
Base model
0.616 / 0.645 / 0.840
4.116 / 4.570 / 4.796
CFG
1.209 / 1.466 / 1.828
3.355 / 3.947 / 4.357
WDDPM
0.605 / 0.658 / 0.734
3.853 / 4.113 / 4.616
DR-Diff(Ours)
0.603 / 0.644 / 0.757
3.749 / 4.225 / 4.539
TABLE II: Quantitative comparison of the SWD across methods for OOD targets under the ablation setup.
Metric
Group
Original
OS
RW
Ours
Accuracy
All
0.911
0.902
0.902
0.907
Group0
0.921
0.910
0.922
0.916
Group1
0.755
0.777
0.592
0.777
Gap
0.166
0.133
0.330
0.139
FPR
All
0.763
0.559
0.883
0.635
Group0
0.897
0.638
0.845
0.749
TABLE III: Performance metrics of the classifiers on the LSAC dataset across racial subgroups. ’Gap’ indicates the absolute difference between Group 0 (White) and Group 1 (Non-White) defined by the race attribute.
Metric
Group
Original
OS
RW
Ours
Accuracy
All
0.910
0.905
0.908
0.908
Group0
0.915
0.911
0.914
0.913
Group1
0.758
0.753
0.742
0.762
Gap
0.158
0.158
0.173
0.152
FPR
All
0.029
0.043
0.029
0.030
Group0
0.027
0.039
0.029
0.028
TABLE IV: Performance metrics of the classifiers on the Jigsaw dataset (Target attribute: mentions of the ’Black’ identity). Group 1 represents texts containing mentions of the identity, while Group 0 represents texts without. ’Gap’ indicates the absolute difference between the subgroups.
Metric
Group
Original
OS
RW
Ours
Accuracy
All
0.910
0.905
0.908
0.908
Group0
0.913
0.910
0.912
0.912
Group1
0.878
0.869
0.876
0.872
Gap
0.035
0.041
0.036
0.040
FPR
All
0.029
0.043
0.036
0.030
Group0
0.028
0.040
0.035
0.028
TABLE V: Performance metrics of the classifiers on the Jigsaw dataset (Target attribute: mentions of the ’Male’ identity). Group 1 represents texts containing mentions of the identity, while Group 0 represents texts without. ’Gap’ indicates the absolute difference between the subgroups.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# Samp.
# Feat.
Target ( Xlabel )
Auxiliary Attributes ( Xcond )
LSAC
21,791
12
Bar exam passage
Race, Gender, LSAT score, Undergraduate GPA, Family income level
The Graduate University for Advanced Studies, SOKENDAI, Tokyo, Japan · The Institute of Statistical Mathematics, Tokyo, Japan · East China Normal University, Shanghai, China +4