Two-Time-Scale Learning Dynamics: A Population View of Neural Network Training

Giacomo Borghi; Hyesung Im; Lorenzo Pareschi

cs.LGMar 20, 2026

Two-Time-Scale Learning Dynamics: A Population View of Neural Network Training

Authors: Giacomo Borghi, Hyesung Im, Lorenzo Pareschi

Organizations: Maxwell Institute for Mathematical Sciences and School of Mathematical and Computer Sciences Heriot–Watt University, Edinburgh, UK · Department of Mathematics and Computer Science University of Ferrara, Italy

Abstract

Population-based learning paradigms, including evolutionary strategies, Population-Based Training (PBT), and recent model-merging methods, combine fast within-model optimisation with slower population-level adaptation. Despite their empirical success, a general mathematical description of the resulting collective training dynamics remains incomplete. We introduce a theoretical framework for neural network training based on two-time-scale population dynamics. We model a population of neural networks as an interacting agent system in which network parameters evolve through fast noisy gradient updates of SGD/Langevin type, while hyperparameters evolve through slower selection--mutation dynamics. We prove the large-population limit for the joint distribution of parameters and hyperparameters and, under strong time-scale separation, derive a selection--mutation equation for the hyperparameter density. For each fixed hyperparameter, the fast parameter dynamics relaxes to a Boltzmann--Gibbs measure, inducing an effective fitness for the slow evolution. The averaged dynamics connects population-based learning with bilevel optimisation and classical replicator--mutator models, yields conditions under which the population mean moves toward the fittest hyperparameter, and clarifies the role of noise and diversity in balancing optimisation and exploration. Numerical experiments illustrate both the large-population regime and the reduced two-time-scale dynamics, and indicate that access to the effective fitness, either in closed form or through population-level estimation, can improve population-level updates.

Figures & tables

Panel of Figure 1 — Figure 1: Two-scale modelling pipeline. Large-population limits ( $N\to\infty$ ) connect the agent-based neural-network dynamics to a two-scale kinetic PDE for the joint density $f(t,\theta,h)$ over parameters and hyperparameters, in which a training operator and a selection–mutation evolutionary operator act on different time scales. A two-scale separation ( $\varepsilon\to 0$ ) averages out the fast parameter dynamics, which relaxes to a Boltzmann–Gibbs equilibrium $\mu^{\infty}_{\mathcal{L}}(\theta|h)$ , and yields a closed averaged PDE for the hyperparameter density $\rho(t,h)$ driven by the effective fitness $\overline{\mathcal{F}}(h)$ .

Panel of Figure 2 — Figure 2: Quadratic objective fitness and loss. Median and $10\%-90\%$ quantiles of the population fitness $\mathcal{F}(\theta)$ and loss $\mathcal{L}(\theta,h)$ over $100$ iterations of Algorithm 1 . PBT updates, where agents are resampled according to their fitness values, occur every 50 training steps and lead to a discontinuity in the fitness evolution. Population size is $N=100$ .

Panel of Figure 3 — Figure 3: Many-agent limit $N\to\infty$ . Evolution of the hyperparameter distributions for different population sizes $N=10^{3},10^{4},10^{5}$ and quadratic problem. Agents are initially sampled uniformly from $[-1,1]$ and evolved according with Algorithm 1 . The total number of iterations is $500$ , with selection jumps at every 50 training steps. The curves are constructed by connecting bin centres (markers) with straight lines and the number of bins is automatically determined.

Panel of Figure 4 — Figure 4: Fast training limit $\varepsilon\to 0$ . Evolution of the hyperparameter distributions for different fast-training regimes. The limit $\varepsilon\to 0$ in the PDE model ( 6 ) corresponds in Algorithm 1 to increasing the number of internal SGD steps performed between successive PBT hyperparameter updates. We consider 20, 50, and 100 inner training steps. The case $\varepsilon=0$ corresponds to directly sampling from the equilibrium distribution ( 25 ) and evolving the agents according to Algorithm 2 , which simulates the averaged PDE model ( 13 ). The population size is $N=10^{5}$ , and the curves are obtained by connecting bin centers with straight lines using 50 bins.

Panel of Figure 5 — Figure 5: Himmelblau test, parameters’ evolution. The population size is $N=10^{5}$ , and the parameters $\theta$ are initially sampled uniformly from the box $[-0.5,0.5]^{2}$ . The density is reconstructed using a 40-bin histogram in each dimension. The second row in (b) shows the trajectories of 5 randomly selected agents. The solid orange curves correspond to the SGD training dynamics, while the red dashed lines indicate the jumps induced by the PBT update after an inner training loop.

Panel of Figure 6 — Figure 6: Cart–Pole control: two-time-scale dynamics. Mean episodic reward of top five agents (solid line with markers) and entire population (dashed line) across generations. We solve the RL problem with $N=500$ NN agents via Population-Based Training evolution. The parameter averaging window size indicates the number of recent episodes ( $m$ ) averaged to compute the agent’s fitness score. Larger window sizes yield more stable evaluations, reducing fast-scale noise and leading to faster convergence.

Panel of Figure 7 — Figure 7: Cart–Pole control: hyperparameter distribution. For two different values of the mutation noise $\sigma$ , we show the distribution of hyperparameters across $N=500$ agents, using $1000$ training steps and an averaging window of size $m=2$ . The maximum episodic reward is $500$ . The marginal distributions in each panel are estimated via Gaussian kernel density estimation.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 507 papers

Singular perturbations and hierarchical learning in two-layer neural networks
Jul 12, 2026Cédric Gerbelot, Jean-Christophe MourratMultilayer Perceptrons Neural Network Training Dynamics
Unité de Mathématiques Pures et Appliquées (UMPA), ENS Lyon · CNRS
HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball
Sep 7, 2026Jinghui Yuan, Hongtao Zhang, Jade Zou +4Neural Network Optimization Neural Network Training Dynamics
StepFun · University of Chinese Academy of Sciences, StepFun · University of Chinese Academy of Sciences +2
A statistical physics framework for optimal learning
Jul 10, 2025Francesca Mignacco, Francesco MoriStatistical Physics of Learning Neural Network Training Dynamics
1Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ 08544, USA · 2Graduate Center, City University of New York, New York, NY 10016, USA · Center of Mathematical Sciences and Applications, Harvard University, Cambridge, MA 02138, USA

Jul 12, 2026J/K move · Enter open · S save

Singular perturbations and hierarchical learning in two-layer neural networks

Cédric Gerbelot, Jean-Christophe Mourrat

Unité de Mathématiques Pures et Appliquées (UMPA), ENS Lyon · CNRS

Multilayer Perceptrons Neural Network Training Dynamics Two-Layer Neural Networks

Open paper AskPDF arXiv

We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are optimized jointly, with a perturbative parameter tuning the relative training speed between the first and second layer. This setting was considered by Berthier, Montanari and Zhou in \cite{berthier2024learning}, who conjectured a hierarchical learning scenario with explicit timescales as the second layer is trained faster than the first. In this paper, we prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales, at sharp explicit thresholds. We then analyze the onset of learning of the quadratic component and show that the components learned at earlier stages continue to influence the dynamics in an essential way. Our proof is based on quantitative approximation results for singularly perturbed flows evolving near a manifold defined by integral constraints. At a phenomenological level, we also show that the empirical measure of the weights displays singular behaviour when reaching the quadratic component of the hidden link, with a small fraction of neurons growing significantly while the remaining ones rearrange to preserve the components already learned.

Abstract

Neural Network Training Dynamics

Explore similar work

Singular perturbations and hierarchical learning in two-layer neural networks

HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball

A statistical physics framework for optimal learning