We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* η range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.
Figures & tables
Figure 1: Staged disambiguation. Loss curves for K∈{5,10,20,36} at nb=1,000 . Dashed lines mark logK . The model converges to the marginal solution within a few hundred steps, then plateaus until a sharp transition to near-zero loss.
B
τsteps
τtok (K)
Step ratio
Token ratio
32
23,200
742
23.2×
1.81×
64
7,200
461
7.2×
1.12×
128
3,200
410
3.2×
1.00×
256
1,600
410
1.6×
1.00×
512
1,000
512
1.0×
1.25×
Table 1: Batch-size sweep at K=20 , η=10−3 : steps vs. tokens. The 23× step ratio is predominantly a throughput effect; the token-normalized residual is 1.8× .
Figure 2: Noise delays escape. (a) Batch-size effect: a modest 1.8× delay in tokens processed (the 23× step-count ratio is predominantly throughput), consistent with entropic stabilization [ Ziyin et al., 2024 ] . (b) Label noise shows a 16× delay in tokens, but also degrades the conditional signal (§ 3.4 ); the LR sweep ( 3.6× at constant throughput) provides the cleanest evidence.
Figure 3: Internal precursors. (a) z -dependence leads loss across all K . (b) Transition is onset of coherent descent.
K
τfwd ( A→B )
τbwd ( (B,z)→A )
Ratio
Transfer
5
2,650
600
4.4×
0.75×
10
5,400
1,400
3.9×
0.88×
20
9,500
3,200
3.0×
1.07×
36
13,050
7,600
1.7×
1.31×
Table 2: Directional asymmetry. Unambiguous forward is consistently slower.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
K
3
5
7
10
13
17
20
25
30
36
D
3K
5K
7K
10K
13K
17K
20K
25K
30K
36K
τ
450
800
1,050
1,850
2,100
3,300
3,950
5,250
6,950
8,750
Appendix
Table 3: K-sweep at nb=1,000 , equivalently a D-sweep. Since D=nb×K=1,000K and K has no independent effect on τ (§ 3.2 ), this table is interpretable as τ vs. D . Power-law fit: τ∝D1.19 ( R2=0.98 , 95% CI [1.14,1.31] ).
Figure 4: τ versus D on log-log axes. Blue circles: 10-point K-sweep at nb=1,000 , reinterpreted as a D-sweep ( D=1,000K ). Red squares: fixed- D control ( D=10 K and 20 K, four K values each), confirming K -independence. Dashed line: power-law fit τ∝D1.19 .
Figure 5: Duration depends on D , not K . At fixed D , τ is flat across K . Doubling D roughly doubles τ .
D
K
η
Mean τ
Std τ
CV
5,000
5
5×10−3
2,850
1,727
61%
5,000
5
7×10−3
3,250
2,295
71%
5,000
5
1×10−2
2,183
504
23%
10,000
10
3×10−3
5,950
2,299
39%
10,000
10
5×10−3
9,867
3,838
39%
10,000
10
7×10−3
8,533
1,893
22%
Appendix
Table 4: Multi-seed τ across D values. 3 seeds per condition from phase-boundary experiments. SE =std/3 .
α
δ
95% CI
n
0.3
1.22
[1.16,1.34]
10
0.4
1.20
[1.13,1.32]
10
0.5
1.20
[1.14,1.30]
10
0.6
1.17
[1.11,1.31]
10
0.7
1.12
[1.07,1.23]
10
Appendix
Table 5: Threshold sensitivity. Power-law exponent δ for τ∝Dδ at different threshold fractions α of logK .
K
Step
Frac. τ
≥80%
100%
Mean acc.
10
500
27%
0.0%
0.0%
0.118
10
900
49%
0.0%
0.0%
0.154
10
1,800
97%
31.5%
1.5%
0.645
10
2,800
151%
99.0%
75.0%
0.969
20
1,900
48%
0.0%
0.0%
0.083
20
2,900
73%
0.0%
0.0%
0.192
Appendix
Table 6: Collective transition. 200 sampled base groups evaluated at four checkpoints per K value.
∣z∣
Plateau (nats)
Plateau / logK
τ
1
2.34
1.017
1,650
2
2.39
1.037
1,850
3
2.33
1.014
1,500
4
2.32
1.007
1,450
Appendix
Table 7: Selector length variants at K=10 ( logK=2.30nats ).
K
D
Max all-succeed η
Min any-fail η
η∗
5
5,000
1.0×10−2
1.5×10−2
1.2×10−2
10
10,000
7.0×10−3
1.0×10−2
8.4×10−3
20
20,000
3.0×10−3
5.0×10−3
3.9×10−3
36
36,000
1.0×10−3
2.0×10−3
1.4×10−3
Appendix
Table 8: Phase boundary. Critical learning rate η∗ per K .
K=10
K=20
Phase
Head
ΔL
Head
ΔL
Pre
L0H3
0.019
L0H3
0.022
L0H1
0.012
L0H0
0.015
Mid
L0H3
1.41
L0H3
1.72
L0H0
1.30
L0H0
1.07
L0H2
1.28
L0H2
1.05
Appendix
Table 9: Head ablation. Loss increase ( ΔL , nats) when zeroing each head. Top 4 heads shown per phase.
K
Anisotropy (mean)
λmax
3
17,700
9.83
5
11,700
5.90
10
10,800
4.59
20
12,100
2.83
36
17,600
0.95
Appendix
Table 10: Hessian anisotropy ( ∣λmax/λmin∣ ) during the plateau for each K .
K
Q ( η=3×10−4 )
Q ( η=10−3 )
Q ( η=3×10−3 )
5
2.72
3.96
7.05
10
5.92
7.42
3.48
20
16.4
9.90
159.8
Appendix
Table 11: Gradient dissipation Q during transition, by K and η .
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
Neural processes are meta-learning models that map context sets to predictive distributions. While inspired by stochastic processes, NPs do not generally satisfy the Kolmogorov consistency conditions required to define a valid stochastic process. This inconsistency is widely acknowledged but poorly understood. Practitioners note that NPs work well despite the violation, without quantifying what this means. We address this gap by defining the conditioning consistency gap, a KL divergence measuring how much a conditional neural process's (CNP) predictions change when a point is added to the context versus conditioned upon. Our main results show that for CNPs with bounded encoders and Lipschitz decoders, the consistency gap is O(1/n2) in context size n, and that this rate is tight. These bounds establish the precise sense in which CNPs approximate valid stochastic processes. The inconsistency is negligible for moderate context sizes but can be significant in the few-shot regime.
Robin Young
Department of Computer Science and Technology University of Cambridge
Understanding the learning-rate (LR) dependence of trained model performance is essential for better optimization of neural networks, but the appropriate size of the LR depends on learning dynamics, architecture, and their hidden biases, and thus our understanding of this complex dependence remains limited. The behavior of the loss curve as a function of the LR has recently been observed in extensive experiments on LR transfer across different settings, such as model sizes, and it is becoming more important to explore methodologies for quantitatively characterizing the shape of this LR curve. In this work, we provide two relational equations that give a good empirical explanation of the gradient of the LR transfer curve, i.e., the LR gradient. One is a theoretical formula that holds at the edge of stability, and the other is an approximate expression obtained by naively treating this gradient as a hyper-parameter gradient. Using these two relational equations in a complementary manner, we show a lower bound on the gradient of the curve and a zero gradient under specific conditions for normalization layers in theory. The approximate expression also applies to a wide range of experimental settings and allows us to decompose the LR gradient into layer-wise components and quantify the strong contribution of specific layers to the curve shape. Thus, this serves as a foundation for better understanding the internal mechanisms of LR dependence.
Engression (Shen and Meinshausen, 2024) learns a conditional distribution by fitting a generative model Y=f(X,ε) under the energy score, a strictly proper scoring rule. We provide a theoretical error analysis of engression implemented with deep neural networks. We decompose the excess risk into three components: the approximation error, the stochastic error, and the Monte Carlo error. Based on this decomposition, we establish convergence rates under the assumption that the target conditional generator admits a compositional smoothness structure.
Juntong Chen, Zijian Guo, Xinwei Shen
School of Mathematical Sciences, Xiamen University · Center for Data Science, Zhejiang University · Department of Statistics, University of Washington