Phase Transitions in Learning Driven by Extremal Data

Ethan Levien; Trevor GrandPre

cs.LGOct 21, 2026

Phase Transitions in Learning Driven by Extremal Data

Authors: Ethan Levien, Trevor GrandPre

Organizations: Department of Mathematics, Dartmouth College, Hanover, NH 03755, USA · Department of Physics, Washington University in St. Louis, St. Louis, MO 63130, USA · National Institute for Theory and Mathematics in Biology, Northwestern University and The University of Chicago, Chicago, IL, USA

Abstract

We study an extremal regression problem, an example of multiple instance learning (MIL), where the predictors are given as multisets, called bags. The bag elements (called instances) possess hidden labels which are pooled to produce an observed bag-level label. Max pooling is common in MIL applications, but gives rise to vanishing gradients. To handle this, modern MIL pipelines approximate the max with a softmax, treating the sharpness as a learnable parameter, denoted $λ$ . When the max pooling model is the true data generating process, learning will be successful when $λ$ is increased during gradient descent, thereby driving the dynamics into regions of parameter space with vanishing gradients. To understand the conditions under which gradient descent breaks down, we study a minimal model and derive a low dimensional projection of the gradient flow in the joint limit of large bags $n$ and high dimension $d$ with $α=\ln n/d$ fixed. The flow has two distinct phase transitions: an intrinsic freezing transition related to the random energy model of disordered systems, and a retrieval transition occurring in modern Hopfield networks. Numerical simulation confirms both the phase diagram and the convergence of the finite-size training gradient to its predicted asymptotic form.

Figures & tables

Figure 1: Standard supervised learning maps each feature vector

x_{i}

to a label

y_{i}=x_{i}^{\top}\beta

. In MIL, bags

X^{k}

of instances are mapped to a single label

w^{k}

produced by an unobserved selection rule (e.g. the max). The goal is to learn

\beta

from

(w^{k},X^{k})

pairs alone without ever observing which instance determined the label.

Figure 2: Training loss against epoch, for gradient descent from a grid of initial overlaps $u_{0}$ (colorbar) at two bag sizes. (a) $d=22$ , $n=3$ and (b) $d=150$ , $n=1808$ . Parameters are $(\lambda_{0},\rho_{0})=(0.2,1.7)$ , $\alpha=0.05$ , $\rho_{\rm true}=1.2$ , $q=0.2$ , $\ell=100$ , with learning rate $0.1$ .

Phase	$\bar{w}^{+}$	$\bar{w}^{-}$	$\Delta_{+}$	$\Delta_{-}$
HR	$\rho\rho_{\rm true}u$	$\lambda\rho$	$\rho_{\rm true}^{2}-\rho\rho_{\rm true}u$	$\sqrt{2\alpha}\rho_{\rm true}-\lambda\rho$
LR	$\rho\rho_{\rm true}u$	$\sqrt{2\alpha}\,\rho$	$\rho_{\rm true}^{2}-\rho\rho_{\rm true}u$	$\sqrt{2\alpha}(\rho_{\rm true}-\rho)$
HN	$\lambda\rho$	$\lambda\rho$	$\rho_{\rm true}^{2}-\lambda\rho$	$\sqrt{2\alpha}\rho_{\rm true}-\lambda\rho$
LN	$\sqrt{2\alpha}\,\rho$	$\sqrt{2\alpha}\,\rho$	$\rho_{\rm true}^{2}-\sqrt{2\alpha}\rho$	$\sqrt{2\alpha}(\rho_{\rm true}-\rho)$
Exact	$\rho_{\rm true}^{2}$	$\sqrt{2\alpha}\,\rho_{\rm true}$

Table 1: Asymptotic normalized bag output

\hat{w}_{\lambda}(X^{\pm}\boldsymbol{\beta})/d

in each of the four phases (H/L

\times

R/N), the corresponding large-

d

limit of the target

w^{k}/d

, which is set by the true generative process alone and so is identical across all four phases, and the resulting per-bag-type residuals

\Delta_{\pm}

. The color of the cells is used to highlight which quantities are the same.

Figure 3: Gradient-descent trajectories with different $u_{0}$ (colorbar). The top row shows the dynamics in $(u,\lambda)$ space with the H/L and R/N phase boundaries shown. The bottom panels show trajectories in $(\rho,\lambda)$ space, with the H/L boundary shows. the dashed lines denote the invariant manifold defined by Eqs. 15 and 19 . Parameters are $(\lambda_{0},\rho_{0})=(0.2,1.7)$ , $\alpha=0.05$ , $\rho_{\rm true}=1.2$ , $q=0.2$ , $\ell=100$ , run to flow time $t=15000$ (learning rate $0.1$ ).

Figure 4: Convergence of the finite-size training gradient to the asymptotic gradient-flow field. Empirical

(\dot{\lambda},\dot{u},\dot{\rho})

compared to the

d\to\infty

theory (dashed line). Parameters are

(\lambda_{0},\rho_{0})=(0.2,1.7)

\alpha=0.05

\rho_{\rm true}=1.2

q=0.2

\ell=100

. The vertical line marks

u^{\star}

	REM	MIL Model
System size	$N$	$\alpha d/\ln 2$
Number of terms	$2^{N}$	$n$
Exponents	$-\beta\sqrt{N}\,x_{i}$	$\lambda\,{\boldsymbol{x}}_{i}^{k}\cdot\boldsymbol{b}$
Inverse temperature	$\beta$	$\lambda\sqrt{\ln 2/\alpha}$
Freezing transition	$\beta_{c}=\sqrt{2\ln 2}$	$\lambda^{\star}=\sqrt{2\alpha}$
CLT transition	$\beta_{c}/2$	$\lambda^{\star}/2$

Table S1: Correspondence between the REM of Sec. S1.2 and the off-target partition function

Z_{n}

. The sign of the exponents is immaterial because their distribution is symmetric.

Figure S1: Empirical

(\dot{\lambda},\dot{u},\dot{\rho})

(solid) and the asymptotic prediction (dashed, same color) against the overlap

u

, at

\rho_{0}=1.7

\rho_{\rm true}=1.2

q=0.2

\ell=100

and

n=1808

throughout. (a) Three values of

\alpha

\lambda_{0}=0.2

, with

d=300

150

and

75

. (b) Three initial temperatures at

\alpha=0.05

and

d=150

. Dotted vertical lines mark

u^{\star}

Figure S2:

\dot{\lambda}

against

\lambda_{0}

at fixed overlap, for increasing

d

(with

n=e^{\alpha d}

), and the asymptotic prediction (dashed). Parameters are

\alpha=0.05

\rho_{0}=1.7

\rho_{\rm true}=1.2

q=0.2

\ell=100

. Left:

u=0.1

, in N throughout. Right:

u=0.7

, in R throughout. The vertical line is

\lambda^{\star}=0.316

Figure S3: Error in the initial

\dot{\lambda}

vs.

\ln d/d

, for

u=0.1

and

u=0.7

, at

\lambda_{0}=0.1

\rho_{0}=1.7

\alpha=0.05

\rho_{\rm true}=1.2

q=0.2

\ell=100

d

runs from 22 to 270 (

n=3

7.3\times 10^{5}

), marked on the top axis. Grey points are individual datasets, black their mean with the standard error: eight datasets per

d

up to

d=225

, four above. Dashed line:

\ln d/d

through the origin and the smallest-

d

point.

Multiple Instance Learning

Multiple Instance Learning learns from labeled sets of instances when labels apply to bags, enabling prediction without dense annotations.

See all 96 papers

Grokking as Dimensional Phase Transition in Neural Networks
Apr 6, 2026Ping WangPhase Transitions in Neural Networks Grokking
Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China
Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions
Feb 4, 2026Dmitry Yarotsky, Eugene Golikov, Yaroslav GusevTensor Decomposition Mean-Field Theory
Applied AI Institute, Moscow, Russia · Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia.
Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
Aug 6, 2026Björn Ladewig, Ibrahim Talha Ersoy, Karoline WiesnerStatistical Physics of Learning Representation Learning
Institute of Physics and Astronomy, University of Potsdam

Apr 6, 2026J/K move · Enter open · S save

Grokking as Dimensional Phase Transition in Neural Networks

Ping Wang

Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

Phase Transitions in Neural Networks Grokking Neural Network Training Dynamics Statistical Physics of Learning Gradient Descent Dynamics

Open paper AskPDF arXiv

Neural network grokking -- the abrupt memorization-to-generalization transition -- challenges our understanding of learning dynamics. Through finite-size scaling of gradient avalanche dynamics across eight model scales, we find that grokking is a dimensional phase transition: effective dimensionality~

D

crosses from sub-diffusive (subcritical,

D < 1

) to super-diffusive (supercritical,

D > 1

) at generalization onset, exhibiting self-organized criticality (SOC). Crucially,

D

reflects gradient field geometry, not network architecture: synthetic i.i.d.\ Gaussian gradients maintain

D \approx 1

regardless of graph topology, while real training exhibits dimensional excess from backpropagation correlations. The grokking-localized

D(t)

crossing -- robust across topologies -- offers new insight into the trainability of overparameterized networks.

Abstract

Multiple Instance Learning

Explore similar work

Grokking as Dimensional Phase Transition in Neural Networks

Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions

Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks