Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data

Jérémie Klinger; Raphaël Urfin; Giulio Biroli; Marylou Gabrié

cs.LGSep 28, 2026

Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data

Authors: Jérémie Klinger, Raphaël Urfin, Giulio Biroli, Marylou Gabrié

Organizations: Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France

Abstract

Score-based generative models generate new samples by integrating a time-dependent drift that carries Gaussian noise onto the target distribution. In practice this drift is modeled by a neural network, trained on a loss integrated over time $t$ with a weighting schedule $w(t)$ . Along the backward dynamics, and for multi-modal distributions, trajectories commit to modes of the target within a narrow time window, the \textit{speciation time}. In this work, focusing on high-dimensional data, we decompose the integrated loss into its single-time contributions and analyze each at fixed signal-to-noise ratio $Λ(t)$ : we show that $Λ(t)$ sets the rate at which each feature of a multimodal target - the mode directions and their relative weights - is acquired during training. Crucially, at high $Λ(t)$ all mode directions are acquired together, on a single timescale insensitive to their amplitudes, while the relative weights are not learned at all. Only near the speciation time, where $Λ(t)$ becomes of order one, do all features become learnable, each on its own timescale: the weights are acquired jointly with the directions, and the directions at rates set by their relative amplitudes. For models trained on time-integrated objectives, the learning dynamics is then governed by how much of the weighting effectively sits near the speciation time, which provides insights on $w(t)$ design choices. These results follow from an exact high-dimensional analysis of the training dynamics of unbalanced and hierarchical Gaussian mixtures. Numerical experiments on image and human genome haplotype generation recover the predicted hierarchy of learning timescales in more complex settings.

Figures & tables

Figure 1 : Exact score parametrization for

(\mathcal{D}_{1})

. Overlap

m

(dashed), skip connection

c

(solid) and mode weight (dotted), against training steps rescaled by

d

, for the unbalanced GMM at

\omega=0.8

, different values of

d

and two noising times,

t=0.5

(left) and

t=t_{s}

(middle) . Typical bimodal histograms of projected samples are shown on the

(right)

panel. In Regime II (left)

c

plateaus on

O(1)

times and

m

converges on

O(d)

times, while the weight stays frozen at its initial value

1/2

. At speciation (right) ,

c

first reaches its final value and then weight and overlap rise together, on a shared

O(d)

timescale.

Figure 2 : Exact score parametrization for

(\mathcal{D}_{2})

. Block overlaps

m_{1}

(dashed) and

m_{2}

(solid), against training steps, for the quadrimodal GMM at

d=1024

, structure parameter

\kappa\in\{0.1,0.2,0.3\}

and two noising times,

t=0.5

(left) and

t=t_{s}

(middle and right) . In Regime II (left) the two overlaps are learned on the same training timescale, independently of

\kappa

. At speciation (middle) they evolve on different timescales with

\bm{\mu}_{1}

being learned after

\bm{\mu}_{2}

. Rescaling each block’s training time by its own closed-form rate

\lambda(\gamma_{i})

of equation 13 , with

\gamma_{i}^{2}=\kappa_{i}\Lambda_{t}\Gamma_{t}

and

\kappa_{1}=\kappa,\ \kappa_{2}=1-\kappa

, collapses the curves (right) . The inset shows

(\mathcal{D}_{2})

samples, projected on

(\bm{\mu}_{1},\bm{\mu}_{2})

. The smaller

\kappa

, the closer the modes pair up along

\bm{\mu}_{1}

and the more hierarchical the target.

Panel of Figure 3 — Figure 3 : ResNet for GMMs at $d=64$ , under uniform DSM ( left ) and FM-weighted DSM ( right ) in each panel, against training steps. (a) $(\mathcal{D}_{1})$ at $\omega=0.8$ : cosine similarity with $\bm{\mu}$ (blue) and positive fraction of $p=\bm{\mathrm{x}}^{T}\bm{\mu}/\lVert\bm{\mu}\rVert^{2}$ (red); insets show the histogram of $p$ at the three marked steps. (b) $(\mathcal{D}_{2})$ : cosine similarity with $\bm{\mu}_{1}$ for $\kappa\in\{0.2,0.3,0.4\}$ ; the inset replots the FM curves against training steps rescaled by $\lambda(\sqrt{\kappa})$ .

Panel of Figure 4 — Figure 4 : Experiments on MNIST. (a) Cosine similarity with the digit direction (blue) and fraction of 6s (orange), against training steps, for MNIST restricted to 3s and 6s, of which a fraction $\omega=0.8$ are 6s (dotted line), under DSM (left) and FM (right) . Both objectives acquire direction and proportion within a single decade of training, MNIST offering too short an SNR range for the two weightings to separate. (b) Overlap with the digit direction (left) and with the color direction (right) , against training steps, for tinted MNIST trained with FM at tint intensities $\alpha$ , the dataset being built so that the squared amplitude of the color mode scales as $\alpha^{2}$ . The digit direction is learned at a rate nearly insensitive to $\alpha$ , while the color direction separates with it; rescaling training time by $\lambda(\alpha)$ collapses the color curves (inset).

Figure 5 : Sequential feature learning on HGD. (Left) HGD samples projected onto the first two PCA eigenvectors, showing a complex multimodal structure. (Right) Overlap of the generated samples with the leading PCA components

\bm{\mu}_{i}

i=1,\dots,5

, against training steps, under Masked Discrete Diffusion: the directions are recovered one after the other, and rescaling training time by the closed-form rate

\lambda(\sqrt{v_{i}})

of equation 13 , with

v_{i}

the variance explained along the

i

-th reference PCA direction, collapses the two leading ones (inset).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Unbalanced and Kappa-dependence
Optimizer	plain SGD (no momentum, no weight decay, constant LR)
Learning rate	$10^{-1}$
Batch size	$512$
Training steps	$10^{6}$ (SGD steps, no early stopping)
$t_{\rm noise}$	$0.5$ or $t_{s}=\tfrac{1}{2}\log d$
Seeds per config	$20$

Appendix

Table 1 : Hyperparameters shared by all

20

runs averaged to produce each figure. Metrics are logged at

200

log-spaced steps.

	Unbalanced GMM	Quadrimodal GMM
$d$	$64$	$64$
Optimizer	plain SGD (no momentum, no weight decay, constant LR)
Learning rate	$10^{-2}$	$10^{-2}$
Batch size	$2048$	$2048$
Training steps (SGD)	$10^{6}$	$10^{6}$
$t_{\min},t_{\max}$	$5\times 10^{-3},\ 6.2$	$10^{-3},\ 6.2$

Appendix

Table 2 : Hyperparameters associated to each figure. Both objectives (uniform / FM-weighted DSM) use identical hyperparameters otherwise.

Figure 6 : Image sample at tint $\alpha=0.7$ .

	Structure sweep (tinted)	Weight sweep (imbalanced)
Optimizer	AdamW, weight decay $10^{-2}$ (default), constant LR
Learning rate	$10^{-3}$
Batch size	$128$
Training steps	$2{,}000$	$10{,}000$
Sweep values	$\alpha\in\{0.4,\dots,1.0\}$ (step $0.1$ )	$\omega=0.8$
Seeds per value	$10$	$5$

Appendix

Table 3 : Hyperparameters for the two experiments.

Optimizer	AdamW, weight decay $10^{-2}$
Peak learning rate	$3\times 10^{-2}$
LR schedule	linear warm-up ( $50$ steps) $\to$ constant $\to$ linear decay to $10^{-3}\times$ peak over the
	final $20\%$ of training (decay starts at step $80{,}000$ )
Batch size	$256$
Training steps	$100{,}000$
Gradient clipping	global norm $1.0$

Appendix

Table 4 : Optimization hyperparameters for the run shown in Fig. 5

Explore similar work

May 11, 2026cs.LG

The two clocks and the innovation window: When and how generative models learn rules

Generative models trained on finite data face a fundamental tension: their score-matching or next-token objective converges to the empirical training distribution rather than the population distribution we seek to learn. Using rule-valid synthetic tasks, we trace this tension across two training timescales:

τ_{\mathrm{rule}}

, the step at which generations first become rule-valid, and

τ_{\mathrm{mem}}

, the step at which models begin reproducing training samples. Focusing on parity and extending to other binary rules and combinatorial puzzles, we characterize how these two clocks,

τ_{\mathrm{rule}}

and

τ_{\mathrm{mem}}

, depend on key aspects of the learning setup. Specifically, we show that

τ_{\mathrm{rule}}

increases with rule complexity and decreases with model capacity, while

τ_{\mathrm{mem}}

is approximately invariant to the rule and scales nearly linearly with dataset size

N

. We define the \emph{innovation window} as the interval

[τ_{\mathrm{rule}}, τ_{\mathrm{mem}}]

. This window widens with increasing

N

and narrows with rule complexity, and may vanish entirely when

τ_{\mathrm{rule}} \geq τ_{\mathrm{mem}}

. The same two-clock structure arises in both diffusion (DiT) and autoregressive (GPT) models, with architecture-dependent offsets. Dissecting the learned score of DiT models reveals a corresponding evolution of the optimization landscapes, where rule-valid samples' basins expand substantially around

τ_{\mathrm{rule}}

, while training samples' basins begin to dominate around

τ_{\mathrm{mem}}

. Together, these results yield a unified and predictive account of when and how generative models exhibit genuine innovation.

Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu

Kempner Institute at Harvard University

Generative Models Clocks Innovation Synthetic Task

Ask PDF

Jul 6, 2026stat.ML

Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models

Score-based Generative Models (SGMs) have achieved impressive performance in data generation across a wide range of applications. While the statistical properties of their sampling procedures are increasingly well understood, the optimization dynamics underlying their training remain less explored. SGMs are typically trained by minimizing a weighted denoising scorematching objective, yet optimization guarantees with stochastic gradients remain limited. In this work, we study Stochastic Gradient Descent (SGD) for SGMs, contributing results in two complementary regimes. First, for general score parameterizations, we establish a non-convex convergence rate for SGD on the weighted denoising score-matching objective, with explicit dependence on the schedule-dependent weighting factors. Second, for overparameterized two-layer ReLU networks, we develop a Neural Tangent Kernel analysis tailored to diffusion training with stochastic gradients, yielding score-approximation error bounds along the SGD trajectory. Finally, our analysis quantifies the role of the reweighting factor in the score approximation error, providing theoretical guidance for weighting choices used in practice.

Stanislas Strasman, Sobihan Surendran, Sylvain Le Corff

SU, LPSM · Sorbonne Université and Université Paris Cité, CNRS, LPSM, F-75005 Paris, France · LOPF, Califrais’ Machine Learning Lab, Paris, France

Stochastic Gradient Descent Generative Models Finite Time Analysis Rectified Linear Unit Networks

Ask PDF

Sep 15, 2026cs.LG

Walking the Score Manifold: Continuous-time Generative Dynamics on Learned Data Manifolds

Generative modeling of time-dependent data is typically formulated on a discrete temporal grid, restricting supervision to the observed timestamps in the training data. We instead frame generation as continuous-time evolution on a learned data manifold. To this end, we leverage pretrained score-based models as geometric priors and learn a vector field that evolves data along score-induced interpolation paths. Because these dynamics follow transitions that respect the geometry learned by the score model, they support generation at arbitrary timestamps and temporal super-resolution beyond the discretization of the training data. Moreover, this geometric formulation allows us to train the vector field simulation-free through a regression objective. To improve long-horizon rollout robustness, we introduce an objective that promotes path-relative transverse exponential stability. While motivated by stability theory, it admits a practical interpretation as denoising score matching transverse to the interpolation path. Further, we extend the framework to a probabilistic setting that models a distribution over plausible future trajectories. We demonstrate the method on natural video and scientific dynamical data, including temporal super-resolution, PDE-based spatiotemporal fields, and molecular dynamics. Our results show that score-based priors provide a strong foundation for learning stochastic continuous-time generative dynamics.

Jan Tauberschmidt, Brian B. Moser, Stanislav Frolov +3

DFKI Kaiserslautern · RPTU Kaiserslautern · Imperial College London

Generative Models Score-Based Diffusion Model Spatiotemporal Fields Interpolation Geometric Priors

Ask PDF