Provably Reliable Classifier Guidance via Cross-Entropy Control
Authors: Sharan Sahu, Arisina Banerjee, Yuchen Wu
Organizations: Department of Statistics and Data Science, Cornell University · School of Operations Research and Information Engineering, Cornell University
Classifier-guided diffusion models generate conditional samples by augmenting the reverse-time score with the gradient of the log-probability predicted by a probabilistic classifier. In practice, this classifier is usually obtained by minimizing an empirical loss function. While existing statistical theory guarantees good generalization performance when the sample size is sufficiently large, it remains unclear whether such training yields an effective guidance mechanism. We study this question in the context of cross-entropy loss, which is widely used for classifier training. Under mild smoothness assumptions on the classifier, we show that controlling the cross-entropy at each diffusion model step is sufficient to control the corresponding guidance error. In particular, probabilistic classifiers achieving conditional KL divergence ε2 induce guidance vectors with mean squared error O(dε), up to constant and logarithmic factors. Our result yields an upper bound on the sampling error of classifier-guided diffusion models and bears resemblance to a reverse log-Sobolev--type inequality. To the best of our knowledge, this is the first result that quantitatively links classifier training to guidance alignment in diffusion models, providing both a theoretical explanation for the empirical success of classifier guidance, and principled guidelines for selecting classifiers that induce effective guidance.
Figures & tables
Figure 1: Numerical validation of Theorem 3.1 . The plots in the top and bottom panels correspond to Regimes 1 and 2, respectively. The first and third plots display the conditional KL divergence, and the second and fourth plots show the guidance MSE.
Figure 2: Effect of perturbation regularity on guidance error at comparable levels of conditional KL divergence. We consider δn∈{1/n2,1/n,1/n,1/n1/4} and evaluate the corresponding guidance MSEs on a shared conditional-KL grid.
Figure 3: Numerical validation of Theorem 3 under a GMM with d=50 . The figures correspond to a diffusion time t=1.0 . The left panel displays the conditional KL divergence and the right panel shows the guidance MSE for y=0 and y=1 .
Figure 4: Top panel: Guidance effectiveness by target digit. Bottom panel: Averaged gradient norm and Hessian trace as functions of diffusion time t .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Choice of δn
Cross entropy loss
Guidance vector error
δn=1/n
O(1/n)
Θ(1)
δn=1/n
O(1/n)
Θ(n)
Appendix
Table 1: Two illustrative scaling regimes.
Figure 5: Training loss and accuracy curves for smooth and non-smooth classifiers. Both achieve similar final performance in terms of cross-entropy and accuracy.
Diffusion models achieve strong generative performance but often rely on classifier-free guidance (CFG), an inference-time intervention that modifies the sampling trajectory. Since standard denoising score matching (DSM) should recover the target distribution when the score is learned accurately, the practical necessity of CFG suggests deficiencies in learned conditional scores. We identify weak inter-class separation as one such limitation: samples generated from the same noise under different labels can retain similar class-agnostic structure. To address this deficiency, we introduce MCLR, a principled fine-tuning objective that explicitly maximizes inter-class likelihood-ratios, thereby strengthening class-specific structures under standard sampling. Across diverse models and datasets, MCLR substantially improves guidance-free conditional generation and narrows the gap to CFG. Theoretically, we prove that the CFG-guided score is the unique minimizer of a sample-adaptively weighted, ELBO-approximated MCLR objective, interpreting CFG as an implicit inference-time contrastive alignment procedure. We further compare MCLR with related contrastive objectives, including DPO, CCA, and CFM, clarifying their connections and key differences in an otherwise fragmented literature.
Xiang Li, Yixuan Jia, Xiao Li +3
University of Michigan · Michigan State University
Diffusion models have emerged as state-of-the-art generative models for high-fidelity image synthesis, particularly in their classifier-free guided and classifier-guided forms. However, standard classifier guidance concentrates probability mass around high-density class mean, leading to poor coverage of rare samples in the tails of the class-conditional distributions. Recent work on diffusion-based tail sampling mitigates this by training an additional low-density-seeking classifier with a synthetic-vs-real discriminator, at the cost of additional networks and training. In parallel, a number of samplers and distillation techniques accelerate or refine diffusion sampling, but do not explicitly address long-tail coverage. We propose a purely sampling-time, density-aware extension of classifier-guided conditional diffusion model that targets low-density regions without any additional training. We have applied guidance at noisy images not on predicted noise like most diffusion models. Starting from a pretrained conditional diffusion model and classifier on ImageNet, we modify the guided reverse dynamics by steering trajectories toward low-confidence regions via the modified classifier gradient, and at each time step, we also guide the sampling process toward the predicted real image. 1st guidance helps explore low-probability samples, and 2nd guidance helps to generate samples to be close to the real data manifold. The proposed sampler consistently improves ADM model recall at 64x64 resolution while maintaining a comparable FID, and with a 256x256 ADM model, we showed the results visually with different combinations of both guidance. We also showed that standard ADM classifier guidance, combined with predicted real image guidance, helps generate high perceptual quality samples with a 256x256 ADM model on ImageNet.
Classifier-Free Guidance (CFG) is a cornerstone of modern conditional diffusion models, yet its reliance on the fixed or heuristic dynamic guidance weight is predominantly empirical and overlooks the inherent dynamics of the diffusion process. In this paper, we provide a rigorous theoretical analysis of the Classifier-Free Guidance. Specifically, we establish strict upper bounds on the score discrepancy between conditional and unconditional distributions at different timesteps based on the diffusion process. This finding explains the limitations of fixed-weight strategies and establishes a principled foundation for time-dependent guidance. Motivated by this insight, we introduce \textbf{Control Classifier-Free Guidance (C2FG)}, a novel, training-free, and plug-in method that aligns the guidance strength with the diffusion dynamics via an exponential decay control function. Extensive experiments demonstrate that C2FG is effective and broadly applicable across diverse generative tasks, while also exhibiting orthogonality to existing strategies.
Jiayang Gao, Tianyi Zheng, Jiayang Zou +9
Shanghai Jiao Tong University, Shanghai, China · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China