Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models
Organizations: CSE & GSAI, POSTECH, South Korea
Abstract
Denoising generative models predict clean data (), noise (), or velocity (), while training losses can be measured in any of these spaces. Although the resulting nine combinations are algebraically related, they can yield substantially different denoising accuracy and sample quality. We disentangle the effects of prediction and loss spaces. For a fixed prediction target, we show that changing the loss space is exactly equivalent to timestep reweighting and provide a heuristic analysis of how this reweighting affects generation quality. We then show that a rank- linear bottleneck compressing a -dimensional noisy input imposes an irreducible squared-error floor of on noise and velocity prediction, whereas clean-data prediction can exploit low-dimensional data structure. Combining this bound with prediction-dependent error conversion yields distinct necessary conditions on bottleneck capacity. Experiments with U-Net and JiT across CIFAR-10, Imagenette, and ImageNet-256 support these theoretical predictions. Under compression, prediction choice strongly affects both denoising accuracy and sample quality, with noise-prediction errors closely matching the theoretical lower bound. Our results provide a quantitative understanding of how architectural bottlenecks interact with prediction and loss choices in denoising generative models.
Figures & tables
| Weight | Loss | Loss | Loss |
|---|---|---|---|
| Pred | |||
| Pred | |||
| Pred |
| MSE ( ) | FID | |||||||
|---|---|---|---|---|---|---|---|---|
| Dataset | Pred Loss | |||||||
| CIFAR-10 | 5.395 | 5.360 | 5.361 | 11.79 | 2.20 | 2.73 | ||
| 5.401 | 5.372 | 5.367 | 16.45 | 25.49 | 5.88 | |||
| 5.392 | 5.356 | 5.359 | 12.51 | 1.12 | 1.68 | |||
| Imagenette | 4.810 | 4.664 | 4.632 | 53.40 | 12.76 | 21.05 | ||
| n/a † | 4.709 | 4.670 | 85.77 | 95.41 | 100.98 | |||
| MSE ( -space) | FID | |||||||
|---|---|---|---|---|---|---|---|---|
| Config | Pred Loss | |||||||
| (ratio ) | 0.050 | 0.048 | 0.048 | 41.0 | 37.4 | 36.0 | ||
| 5.855 | 5.848 | 5.852 | 146.6 | 147.6 | 151.0 | |||
| 0.152 | 0.152 | 0.152 | 113.2 | 108.3 | 113.0 | |||
| (ratio ) | 0.049 | 0.048 | 0.048 | 42.1 | 38.9 | 37.1 | ||
| 0.051 | 0.051 | 0.051 | 114.1 | 100.1 | 123.6 | |||
| -MSE | FID | |||||||
|---|---|---|---|---|---|---|---|---|
| Config | Pred Loss | |||||||
| ( ) | 0.023 | 0.023 | 0.023 | 23.9 | 36.8 | 20.1 | ||
| 14.61 | 14.61 | 14.61 | 421.2 | 419.0 | 420.1 | |||
| 0.294 | 0.294 | 0.294 | 201.5 | 202.3 | 199.0 | |||
| ( ) | — | — | 0.022 | — | — | 5.7 | ||
| — | — | 0.043 | — | — | 330.6 | |||
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| U-Net (no bottleneck) | JiT-B ( , bottleneck) | ||||||
| Pred Loss | |||||||
| 0.0481 | 0.0466 | 0.0463 | 0.050 | 0.048 | 0.048 | ||
| 0.0486 | 0.0471 | 0.0467 | 5.9 | 5.8 | 5.9 | ||
| 0.0476 | 0.0465 | 0.0463 | 0.152 | 0.152 | 0.152 | ||
| null / | null / | error | |||||
| learned | random | learned | learned | random | |||
| 16 | 2 | 1 | 1.001 | 1.001 | 1.96 | 17.0 | 23.5 |
| 16 | 2 | 2 | 1.002 | 1.003 | 2.04 | 11.5 | 22.7 |
| 16 | 2 | 4 | 1.003 | 1.003 | 2.19 | 11.4 | 16.0 |
| 16 | 2 | 8 | 1.006 | 1.005 | 2.80 | 11.6 | 13.0 |
| 16 | 2 | 12 | 0.997 | 1.012 | 4.64 | 11.6 | 12.2 |
| Ratio | -pred MSE | -pred MSE | Gap | |
|---|---|---|---|---|
| 32 | 0.67 | 0.054 | 0.158 | |
| 64 | 1.33 | 0.054 | 0.054 | |
| 128 | 2.67 | 0.054 | 0.054 | |
| 256 | 5.33 | 0.054 | 0.054 |
| Ratio | -pred MSE | -pred MSE | Gap | Floor | |
|---|---|---|---|---|---|
| 32 | 0.17 | 0.049 | 14.61 | 14.61 | |
| 256 | 1.33 | 0.048 | 0.095 | — | |
| 384 | 2.00 | 0.048 | 0.056 | — | |
| 512 | 2.67 | 0.048 | 0.053 | — | |
| 768 | 4.00 | 0.048 | 0.050 | — |
| -space error | -space error | ||||||
|---|---|---|---|---|---|---|---|
| -pred | -pred | -pred | -pred | ||||
| 2 | 12 | 2.67 | — | 0.054 | 0.055 | 0.054 | 0.060 |
| 4 | 48 | 0.67 | 0.333 | 0.054 | 5.85 | 0.158 | 0.353 |
| 8 | 192 | 0.17 | 0.833 | 0.056 | 14.62 | 0.321 | 0.842 |
| (compression) | (no compr.) | |||
|---|---|---|---|---|
| -pred | -pred | -pred | -pred | |
| Linear | ( ) | ( ) | ||
| GeLU | ( ) | ( ) | ||
| Time distribution | Sensitivity | -loss | -loss | -loss |
|---|---|---|---|---|
| logit-normal , U-Net | ||||
| logit-normal , JiT | ||||
| Velocity-space error | Native error | |||
|---|---|---|---|---|
| -model | -model | -model | -model | |
| 0.06 | 0.106 | 0.443 | 0.116 | 0.0016 |
| 0.10 | 0.081 | 0.198 | 0.093 | 0.0020 |
| 0.50 | 0.051 | 0.067 | 0.066 | 0.017 |
| 0.90 | 0.189 | 0.197 | 0.197 | 0.160 |
| 0.95 | 0.462 | 0.337 | 0.333 | 0.304 |
| FID ( K samples) | ODE (Heun) | SDE |
|---|---|---|
| -prediction | 14.5 | 17.1 |
| -prediction | 333.4 | 330.2 |
Diffusion Models
Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.
See all 1,254 papersZEUS: Accelerating Diffusion Models with Only Second-Order Predictor
Depth as Time in One-Step Generative Models
SiT-L/2 model can be compressed by in parameters into a single time-conditioned block. We offer an explanation for this denoise-then-renoise behavior and show that, when we treat the layerwise computation explicitly as a flow, a single time-conditioned block can be trained to denoise across layers, compressing a MeanFlow SiT-L/2 model by in parameters. Together, these results suggest that the temporal computation of diffusion is not eliminated by one-step generation, but reorganized across network depth.