cs.LGNov 12, 2026

Overcoming Spurious Correlations in Conditional Diffusion Models via Doubly Robust Target Correction

Authors: Eiichi Sakurai, Kenji Yamanishi

Organizations: Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology, Tokyo, Japan

Abstract

While conditional diffusion models have achieved remarkable success in generation tasks, they memorize spurious correlations within training data. This causes them to deviate from ideal target distributions. Minimizing a covariate shift-adjusted target risk can address this issue. However, retraining models from scratch for different distributions is computationally prohibitive. To overcome this limitation, we propose DR-Diff, a framework that efficiently fine-tunes pre-trained conditional diffusion models. Specifically, DR-Diff leverages a functional gradient approach to derive a pseudo-target. This pseudo-target captures the residual between the base model's noise prediction and the ground-truth noise under the target distribution. Training the model to fit this pseudo-target enables efficient target adaptation. Theoretically, we prove that DR-Diff guarantees target risk reduction at the population level. It also achieves a pseudo-doubly robust property, providing resilience against density ratio estimation errors and base model misspecifications. Empirically, DR-Diff matches the noise prediction accuracy of models trained from scratch on synthetic datasets. Furthermore, it mitigates inter-group disparities in downstream classification tasks.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Domain-Adapted Diffusion Models for Conditional Independence Testing

    Sep 27, 2026Yanfeng Yang, Junda Zhao, Yijie Gao +8Multi-Source Domain AdaptationDomain Adaptation

  2. Finite-Sample Guarantees for Fully Discrete Conditional Diffusion Models

    Oct 9, 2026Qiuyun ZhuConditional Diffusion ModelsDiffusion Sampling

  3. Correction Training for Diffusion Models

    Oct 9, 2026Donghoon Ahn, Runqian Wang, Alexei A. Efros +2Diffusion-Based Image GenerationDiffusion Models