Marginals Before Conditionals

Mihir Sahasrabudhe

cs.LGMar 10, 2026

Marginals Before Conditionals

Authors: Mihir Sahasrabudhe

Organizations: School of Information Sciences, University of Illinois Urbana-Champaign

Abstract

We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* η range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.

Figures & tables

Figure 1: Staged disambiguation. Loss curves for

K\in\{5,10,20,36\}

n_{b}{=}1{,}000

. Dashed lines mark

\log K

. The model converges to the marginal solution within a few hundred steps, then plateaus until a sharp transition to near-zero loss.

$B$	$\tau_{\text{steps}}$	$\tau_{\text{tok}}$ (K)	Step ratio	Token ratio
32	23,200	742	$23.2\times$	$1.81\times$
64	7,200	461	$7.2\times$	$1.12\times$
128	3,200	410	$3.2\times$	$1.00\times$
256	1,600	410	$1.6\times$	$1.00\times$
512	1,000	512	$1.0\times$	$1.25\times$

Table 1: Batch-size sweep at

K{=}20

\eta{=}10^{-3}

: steps vs. tokens. The

23\times

step ratio is predominantly a throughput effect; the token-normalized residual is

1.8\times

Panel of Figure 2 — Figure 2: Noise delays escape. (a) Batch-size effect: a modest $1.8\times$ delay in tokens processed (the $23\times$ step-count ratio is predominantly throughput), consistent with entropic stabilization [ Ziyin et al., 2024 ] . (b) Label noise shows a $16\times$ delay in tokens, but also degrades the conditional signal (§ 3.4 ); the LR sweep ( $3.6\times$ at constant throughput) provides the cleanest evidence.

Panel of Figure 3 — Figure 3: Internal precursors. (a) $z$ -dependence leads loss across all $K$ . (b) Transition is onset of coherent descent.

$K$	$\tau_{\text{fwd}}$ ( $A{\to}B$ )	$\tau_{\text{bwd}}$ ( $(B,z){\to}A$ )	Ratio	Transfer
5	2,650	600	$4.4\times$	$0.75\times$
10	5,400	1,400	$3.9\times$	$0.88\times$
20	9,500	3,200	$3.0\times$	$1.07\times$
36	13,050	7,600	$1.7\times$	$1.31\times$

Table 2: Directional asymmetry. Unambiguous forward is consistently slower.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

$K$	3	5	7	10	13	17	20	25	30	36
$D$	3K	5K	7K	10K	13K	17K	20K	25K	30K	36K
$\tau$	450	800	1,050	1,850	2,100	3,300	3,950	5,250	6,950	8,750

Appendix

Table 3: K-sweep at

n_{b}=1{,}000

, equivalently a D-sweep. Since

D=n_{b}\times K=1{,}000K

and

K

has no independent effect on

\tau

(§ 3.2 ), this table is interpretable as

\tau

vs.

D

. Power-law fit:

\tau\propto D^{1.19}

(

R^{2}=0.98

, 95% CI

[1.14,1.31]

Figure 4:

\tau

versus

D

on log-log axes. Blue circles: 10-point K-sweep at

n_{b}=1{,}000

, reinterpreted as a D-sweep (

D=1{,}000K

). Red squares: fixed-

D

control (

D=10

K and

20

K, four

K

values each), confirming

K

-independence. Dashed line: power-law fit

\tau\propto D^{1.19}

Figure 5: Duration depends on

D

, not

K

. At fixed

D

\tau

is flat across

K

. Doubling

D

roughly doubles

\tau

$D$	$K$	$\eta$	Mean $\tau$	Std $\tau$	CV
5,000	5	$5\times 10^{-3}$	2,850	1,727	61%
5,000	5	$7\times 10^{-3}$	3,250	2,295	71%
5,000	5	$1\times 10^{-2}$	2,183	504	23%
10,000	10	$3\times 10^{-3}$	5,950	2,299	39%
10,000	10	$5\times 10^{-3}$	9,867	3,838	39%
10,000	10	$7\times 10^{-3}$	8,533	1,893	22%

Appendix

Table 4: Multi-seed

\tau

across

D

values. 3 seeds per condition from phase-boundary experiments. SE

=\text{std}/\sqrt{3}

$\alpha$	$\delta$	95% CI	$n$
0.3	1.22	$[1.16,1.34]$	10
0.4	1.20	$[1.13,1.32]$	10
0.5	1.20	$[1.14,1.30]$	10
0.6	1.17	$[1.11,1.31]$	10
0.7	1.12	$[1.07,1.23]$	10

Appendix

Table 5: Threshold sensitivity. Power-law exponent

\delta

for

\tau\propto D^{\delta}

at different threshold fractions

\alpha

\log K

$K$	Step	Frac. $\tau$	${\geq}80\%$	$100\%$	Mean acc.
10	500	27%	0.0%	0.0%	0.118
10	900	49%	0.0%	0.0%	0.154
10	1,800	97%	31.5%	1.5%	0.645
10	2,800	151%	99.0%	75.0%	0.969
20	1,900	48%	0.0%	0.0%	0.083
20	2,900	73%	0.0%	0.0%	0.192

Appendix

Table 6: Collective transition. 200 sampled base groups evaluated at four checkpoints per

K

value.

$\|z\|$	Plateau (nats)	Plateau / $\log K$	$\tau$
1	2.34	1.017	1,650
2	2.39	1.037	1,850
3	2.33	1.014	1,500
4	2.32	1.007	1,450

Appendix

Table 7: Selector length variants at

K=10

(

\log K=2.30\,\text{nats}

$K$	$D$	Max all-succeed $\eta$	Min any-fail $\eta$	$\eta^{*}$
5	5,000	$1.0\times 10^{-2}$	$1.5\times 10^{-2}$	$1.2\times 10^{-2}$
10	10,000	$7.0\times 10^{-3}$	$1.0\times 10^{-2}$	$8.4\times 10^{-3}$
20	20,000	$3.0\times 10^{-3}$	$5.0\times 10^{-3}$	$3.9\times 10^{-3}$
36	36,000	$1.0\times 10^{-3}$	$2.0\times 10^{-3}$	$1.4\times 10^{-3}$

Appendix

Table 8: Phase boundary. Critical learning rate

\eta^{*}

per

K

	$K=10$		$K=20$
Phase	Head	$\Delta\mathcal{L}$	Head	$\Delta\mathcal{L}$
Pre	L0H3	0.019	L0H3	0.022
Pre	L0H1	0.012	L0H0	0.015
Mid	L0H3	1.41	L0H3	1.72
	L0H0	1.30	L0H0	1.07
	L0H2	1.28	L0H2	1.05

Appendix

Table 9: Head ablation. Loss increase (

\Delta\mathcal{L}

, nats) when zeroing each head. Top 4 heads shown per phase.

$K$	Anisotropy (mean)	$\lambda_{\text{max}}$
3	$17{,}700$	9.83
5	$11{,}700$	5.90
10	$10{,}800$	4.59
20	$12{,}100$	2.83
36	$17{,}600$	0.95

Appendix

Table 10: Hessian anisotropy (

|\lambda_{\text{max}}/\lambda_{\text{min}}|

) during the plateau for each

K

$K$	$Q$ ( $\eta=3\times 10^{-4}$ )	$Q$ ( $\eta=10^{-3}$ )	$Q$ ( $\eta=3\times 10^{-3}$ )
5	2.72	3.96	7.05
10	5.92	7.42	3.48
20	16.4	9.90	159.8

Appendix

Table 11: Gradient dissipation

Q

during transition, by

K

and

\eta

Hypothesis	Prediction	Result	Verdict
Gradient cancellation drives plateau	$\tau\propto$ number of groups $G$	$\beta\approx 0.02$ ; no scaling with $G$	Falsified
Barrier crossing (noise helps)	More noise $\to$ faster escape	LR: $3.6\times$ slower; BS: $1.8\times$ (tokens)	Falsified
Incremental group coverage	Groups solved independently over time	0% of groups ${\geq}80\%$ at $\tau/2$	Falsified
Label noise breaks symmetry	Noise perturbs away from flat solution	$16\times$ delay ( $p{=}0.2$ ); confounded ∗	Falsified ∗
$K$ drives duration	$\tau\propto K^{\delta}$ at fixed $D$	$\delta=-0.004$ ; no $K$ effect	Falsified
Linear networks suffice	Two-layer linear escapes	Stays at $\log K$ permanently	Falsified

Appendix

Table 12: Tested mechanisms and their outcomes.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 499 papers

On the Conditioning Consistency Gap in Conditional Neural Processes
Apr 21, 2026Robin YoungNeural Processes Conditional Generative Modeling
Department of Computer Science and Technology University of Cambridge
Measuring the local shape of learning-rate transfer curves
Oct 15, 2026Satoki Ishikawa, Rio Yokota, Ryo KarakidaNeural Network Optimization Neural Network Training Dynamics
Institute of Science Tokyo · AIST
Error Analysis of Neural-Network-Based Engression
Jul 30, 2026Juntong Chen, Zijian Guo, Xinwei ShenConditional Distribution Estimation Neural Network Approximation Theory
School of Mathematical Sciences, Xiamen University · Center for Data Science, Zhejiang University · Department of Statistics, University of Washington

Apr 21, 2026J/K move · Enter open · S save

On the Conditioning Consistency Gap in Conditional Neural Processes

Robin Young

Department of Computer Science and Technology University of Cambridge

Neural Processes Conditional Generative Modeling

Open paper AskPDF arXiv

Neural processes are meta-learning models that map context sets to predictive distributions. While inspired by stochastic processes, NPs do not generally satisfy the Kolmogorov consistency conditions required to define a valid stochastic process. This inconsistency is widely acknowledged but poorly understood. Practitioners note that NPs work well despite the violation, without quantifying what this means. We address this gap by defining the conditioning consistency gap, a KL divergence measuring how much a conditional neural process's (CNP) predictions change when a point is added to the context versus conditioned upon. Our main results show that for CNPs with bounded encoders and Lipschitz decoders, the consistency gap is

O(1/n^2)

in context size

n

, and that this rate is tight. These bounds establish the precise sense in which CNPs approximate valid stochastic processes. The inconsistency is negligible for moderate context sizes but can be significant in the few-shot regime.

Abstract

Neural Network Training Dynamics

Explore similar work

On the Conditioning Consistency Gap in Conditional Neural Processes

Measuring the local shape of learning-rate transfer curves

Error Analysis of Neural-Network-Based Engression