Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data

Jérémie Klinger; Raphaël Urfin; Giulio Biroli; Marylou Gabrié

cs.LGSep 28, 2026

Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data

Authors: Jérémie Klinger, Raphaël Urfin, Giulio Biroli, Marylou Gabrié

Organizations: Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France

Abstract

Score-based generative models generate new samples by integrating a time-dependent drift that carries Gaussian noise onto the target distribution. In practice this drift is modeled by a neural network, trained on a loss integrated over time $t$ with a weighting schedule $w(t)$ . Along the backward dynamics, and for multi-modal distributions, trajectories commit to modes of the target within a narrow time window, the \textit{speciation time}. In this work, focusing on high-dimensional data, we decompose the integrated loss into its single-time contributions and analyze each at fixed signal-to-noise ratio $Λ(t)$ : we show that $Λ(t)$ sets the rate at which each feature of a multimodal target - the mode directions and their relative weights - is acquired during training. Crucially, at high $Λ(t)$ all mode directions are acquired together, on a single timescale insensitive to their amplitudes, while the relative weights are not learned at all. Only near the speciation time, where $Λ(t)$ becomes of order one, do all features become learnable, each on its own timescale: the weights are acquired jointly with the directions, and the directions at rates set by their relative amplitudes. For models trained on time-integrated objectives, the learning dynamics is then governed by how much of the weighting effectively sits near the speciation time, which provides insights on $w(t)$ design choices. These results follow from an exact high-dimensional analysis of the training dynamics of unbalanced and hierarchical Gaussian mixtures. Numerical experiments on image and human genome haplotype generation recover the predicted hierarchy of learning timescales in more complex settings.

Figures & tables

Figure 1 : Exact score parametrization for

(\mathcal{D}_{1})

. Overlap

m

(dashed), skip connection

c

(solid) and mode weight (dotted), against training steps rescaled by

d

, for the unbalanced GMM at

\omega=0.8

, different values of

d

and two noising times,

t=0.5

(left) and

t=t_{s}

(middle) . Typical bimodal histograms of projected samples are shown on the

(right)

panel. In Regime II (left)

c

plateaus on

O(1)

times and

m

converges on

O(d)

times, while the weight stays frozen at its initial value

1/2

. At speciation (right) ,

c

first reaches its final value and then weight and overlap rise together, on a shared

O(d)

timescale.

Figure 2 : Exact score parametrization for

(\mathcal{D}_{2})

. Block overlaps

m_{1}

(dashed) and

m_{2}

(solid), against training steps, for the quadrimodal GMM at

d=1024

, structure parameter

\kappa\in\{0.1,0.2,0.3\}

and two noising times,

t=0.5

(left) and

t=t_{s}

(middle and right) . In Regime II (left) the two overlaps are learned on the same training timescale, independently of

\kappa

. At speciation (middle) they evolve on different timescales with

\bm{\mu}_{1}

being learned after

\bm{\mu}_{2}

. Rescaling each block’s training time by its own closed-form rate

\lambda(\gamma_{i})

of equation 13 , with

\gamma_{i}^{2}=\kappa_{i}\Lambda_{t}\Gamma_{t}

and

\kappa_{1}=\kappa,\ \kappa_{2}=1-\kappa

, collapses the curves (right) . The inset shows

(\mathcal{D}_{2})

samples, projected on

(\bm{\mu}_{1},\bm{\mu}_{2})

. The smaller

\kappa

, the closer the modes pair up along

\bm{\mu}_{1}

and the more hierarchical the target.

Panel of Figure 3 — Figure 3 : ResNet for GMMs at $d=64$ , under uniform DSM ( left ) and FM-weighted DSM ( right ) in each panel, against training steps. (a) $(\mathcal{D}_{1})$ at $\omega=0.8$ : cosine similarity with $\bm{\mu}$ (blue) and positive fraction of $p=\bm{\mathrm{x}}^{T}\bm{\mu}/\lVert\bm{\mu}\rVert^{2}$ (red); insets show the histogram of $p$ at the three marked steps. (b) $(\mathcal{D}_{2})$ : cosine similarity with $\bm{\mu}_{1}$ for $\kappa\in\{0.2,0.3,0.4\}$ ; the inset replots the FM curves against training steps rescaled by $\lambda(\sqrt{\kappa})$ .

Panel of Figure 4 — Figure 4 : Experiments on MNIST. (a) Cosine similarity with the digit direction (blue) and fraction of 6s (orange), against training steps, for MNIST restricted to 3s and 6s, of which a fraction $\omega=0.8$ are 6s (dotted line), under DSM (left) and FM (right) . Both objectives acquire direction and proportion within a single decade of training, MNIST offering too short an SNR range for the two weightings to separate. (b) Overlap with the digit direction (left) and with the color direction (right) , against training steps, for tinted MNIST trained with FM at tint intensities $\alpha$ , the dataset being built so that the squared amplitude of the color mode scales as $\alpha^{2}$ . The digit direction is learned at a rate nearly insensitive to $\alpha$ , while the color direction separates with it; rescaling training time by $\lambda(\alpha)$ collapses the color curves (inset).

Figure 5 : Sequential feature learning on HGD. (Left) HGD samples projected onto the first two PCA eigenvectors, showing a complex multimodal structure. (Right) Overlap of the generated samples with the leading PCA components

\bm{\mu}_{i}

i=1,\dots,5

, against training steps, under Masked Discrete Diffusion: the directions are recovered one after the other, and rescaling training time by the closed-form rate

\lambda(\sqrt{v_{i}})

of equation 13 , with

v_{i}

the variance explained along the

i

-th reference PCA direction, collapses the two leading ones (inset).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Unbalanced and Kappa-dependence
Optimizer	plain SGD (no momentum, no weight decay, constant LR)
Learning rate	$10^{-1}$
Batch size	$512$
Training steps	$10^{6}$ (SGD steps, no early stopping)
$t_{\rm noise}$	$0.5$ or $t_{s}=\tfrac{1}{2}\log d$
Seeds per config	$20$

Appendix

Table 1 : Hyperparameters shared by all

20

runs averaged to produce each figure. Metrics are logged at

200

log-spaced steps.

	Unbalanced GMM	Quadrimodal GMM
$d$	$64$	$64$
Optimizer	plain SGD (no momentum, no weight decay, constant LR)
Learning rate	$10^{-2}$	$10^{-2}$
Batch size	$2048$	$2048$
Training steps (SGD)	$10^{6}$	$10^{6}$
$t_{\min},t_{\max}$	$5\times 10^{-3},\ 6.2$	$10^{-3},\ 6.2$

Appendix

Table 2 : Hyperparameters associated to each figure. Both objectives (uniform / FM-weighted DSM) use identical hyperparameters otherwise.

Figure 6 : Image sample at tint $\alpha=0.7$ .

	Structure sweep (tinted)	Weight sweep (imbalanced)
Optimizer	AdamW, weight decay $10^{-2}$ (default), constant LR
Learning rate	$10^{-3}$
Batch size	$128$
Training steps	$2{,}000$	$10{,}000$
Sweep values	$\alpha\in\{0.4,\dots,1.0\}$ (step $0.1$ )	$\omega=0.8$
Seeds per value	$10$	$5$

Appendix

Table 3 : Hyperparameters for the two experiments.

Optimizer	AdamW, weight decay $10^{-2}$
Peak learning rate	$3\times 10^{-2}$
LR schedule	linear warm-up ( $50$ steps) $\to$ constant $\to$ linear decay to $10^{-3}\times$ peak over the
	final $20\%$ of training (decay starts at step $80{,}000$ )
Batch size	$256$
Training steps	$100{,}000$
Gradient clipping	global norm $1.0$

Appendix

Table 4 : Optimization hyperparameters for the run shown in Fig. 5

Explore similar work

The two clocks and the innovation window: When and how generative models learn rules
May 11, 2026Binxu Wang, Emma Lucia Byrnes Finn, Bingbin LiuGenerative Models Clocks
Kempner Institute at Harvard University
Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
Jul 6, 2026Stanislas Strasman, Sobihan Surendran, Sylvain Le CorffStochastic Gradient Descent Generative Models
SU, LPSM · Sorbonne Université and Université Paris Cité, CNRS, LPSM, F-75005 Paris, France · LOPF, Califrais’ Machine Learning Lab, Paris, France
Walking the Score Manifold: Continuous-time Generative Dynamics on Learned Data Manifolds
Sep 15, 2026Jan Tauberschmidt, Brian B. Moser, Stanislav Frolov +3Generative Models Score-Based Diffusion Model
DFKI Kaiserslautern · RPTU Kaiserslautern · Imperial College London

May 11, 2026, cs.LGJ/K move · Enter open · S save

The two clocks and the innovation window: When and how generative models learn rules

Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu

Kempner Institute at Harvard University

Generative Models Clocks Innovation Synthetic Task

Open paper AskPDF arXiv

Generative models trained on finite data face a fundamental tension: their score-matching or next-token objective converges to the empirical training distribution rather than the population distribution we seek to learn. Using rule-valid synthetic tasks, we trace this tension across two training timescales:

τ_{\mathrm{rule}}

, the step at which generations first become rule-valid, and

τ_{\mathrm{mem}}

, the step at which models begin reproducing training samples. Focusing on parity and extending to other binary rules and combinatorial puzzles, we characterize how these two clocks,

τ_{\mathrm{rule}}

and

τ_{\mathrm{mem}}

, depend on key aspects of the learning setup. Specifically, we show that

τ_{\mathrm{rule}}

increases with rule complexity and decreases with model capacity, while

τ_{\mathrm{mem}}

is approximately invariant to the rule and scales nearly linearly with dataset size

N

. We define the \emph{innovation window} as the interval

[τ_{\mathrm{rule}}, τ_{\mathrm{mem}}]

. This window widens with increasing

N

and narrows with rule complexity, and may vanish entirely when

τ_{\mathrm{rule}} \geq τ_{\mathrm{mem}}

. The same two-clock structure arises in both diffusion (DiT) and autoregressive (GPT) models, with architecture-dependent offsets. Dissecting the learned score of DiT models reveals a corresponding evolution of the optimization landscapes, where rule-valid samples' basins expand substantially around

τ_{\mathrm{rule}}

, while training samples' basins begin to dominate around

τ_{\mathrm{mem}}

. Together, these results yield a unified and predictive account of when and how generative models exhibit genuine innovation.

Abstract

Explore similar work

The two clocks and the innovation window: When and how generative models learn rules

Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models

Walking the Score Manifold: Continuous-time Generative Dynamics on Learned Data Manifolds