Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity

Datong Zhou; Blake Bordelon

cs.LGOct 24, 2026

Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity

Authors: Datong Zhou, Blake Bordelon

Organizations: Laboratoire Jacques-Louis Lions (LJLL) & Laboratoire de Probabilités, Statistique et Modélisation (LPSM) · Oden Institute and Department of Neuroscience University of Texas at Austin

Abstract

Deep learning models tend to perform best at large widths and depths. The growing interest in scaling laws, especially in the language modeling domain, has motivated procedures for scaling up network widths and depths in order to preserve stability and computational efficiency of training dynamics. Existing depth-scaling approaches force a choice between retaining locally maximal feature learning and preserving a form of feature diversity in the residual stream arising from random differences between channels. We examine a joint scaling recipe that preserves both. With residual width $N_\flat$ , hidden width $N_\sharp$ , and depth $L$ , the key quantity is, by default, the limiting ratio $N_\flat/(N_\sharp L)\toα$ , while correlated initialization (Half Copy) can partially relax the resulting shape constraints. Theory and residual-MLP experiments support convergence at fixed positive $α$ to an SDE limit with locally maximal feature learning. Hidden computations can then be redistributed between width and depth while preserving this limit, like turning melody into harmony and back. Language-model experiments nevertheless show performance differences under regrouping at fixed parameter count. These results distinguish approximating a limit (ODE or SDE) from scaling for better performance.

Figures & tables

Figure 1: Upper: a standard residual MLP structure. Lower: the continuous-depth picture. Residual coordinates evolve through depth, while hidden units contribute at specific depths. The symbols

\flat

and

\sharp

distinguish the two parts.

h

denotes preactivations and

g

rescaled gradients (Jacobian).

Figure 2: Left: growth paths and their limits. Right: the coupling–learning tradeoff within the limiting family. Colors indicate the variance of the random depth contribution on the left and the effective learning-rate ceiling on the right. The color trends and contours are schematic.

Figure 3: Selected GPT-2 activations (Appendix I ). Left: one residual coordinate through depth. Right: paired standardized activations for two inputs; ellipses show covariance-matched Gaussian 95% contours.

Quantity	Input	Up	Down	Output
Initialization std.	$1$	$1/\sqrt{N_{\flat}}$	$\sqrt{N_{\flat}}/K$	$1/N_{\flat}$
SGD learning rate	$N_{\flat}$	$K/N_{\flat}$	$N_{\flat}/K$	$1/N_{\flat}$
Adam learning rate	$1$	$1/N_{\flat}$	$1/K$	$1/N_{\flat}$
Muon learning rate	$\sqrt{N_{\flat}}$	$\sqrt{N_{\sharp}/N_{\flat}}$	$\sqrt{N_{\flat}N_{\sharp}}/K$	$1/\sqrt{N_{\flat}}$
Muon clipping threshold	$1/\sqrt{N_{\flat}}$	$\sqrt{N_{\flat}N_{\sharp}}/K$	$\sqrt{N_{\sharp}/N_{\flat}}$	$\sqrt{N_{\flat}}$

Table 1: Size dependence of initialization and updates for code weights W in Eq. 1 , with

K=N_{\sharp}L

. Fixed task dimensions and multipliers, including

\beta_{0},\gamma_{0}

, are omitted. Here, Muon is understood as a normalized spectral-clipping map defined in Appendix A .

Figure 4: One-step variance: finite-network means (solid) versus Result 2 (dashed), with

\pm 2

Monte Carlo SE shaded;

\beta_{0}=1

L=128

(Appendix C.2.9 ).

Figure 5: CIFAR-10 convergence: output and final-residual cosine-kernel errors against

q=96

ensemble means (left/middle; bands:

\pm 2

conditional SE; dashed:

q^{-1}

), and coupled regrouping against the finest finite partition at

N_{\flat}=K=24^{2}

(right).

Figure 6: Larger-scale comparisons, counting block weights only. Tiny: initial

Q

and loss after 5,000 Adam steps; CIFAR-10: loss after five SGD epochs. Means

\pm

sample SD over three seeds; full ranges: Appendices G.2 , H .

Figure 7: Residual energy before and after training. For complete attention–MLP stage increments

\Delta\bm{h}_{\ell}

E_{0}=\|\bm{h}_{0}\|^{2}/N_{\flat}

Q=\sum_{\ell}\|\Delta\bm{h}_{\ell}\|^{2}/N_{\flat}

, and

C=\|\bm{h}_{L}-\bm{h}_{0}\|^{2}/N_{\flat}-Q

. Each quantity is computed per token and averaged over the same four 1024-token sequences. The signed cross-depth term

C

is retained; the panels use different vertical scales (Appendix I.2 ).

Figure 8: Width–depth allocation and finite performance. Left: Tiny Shakespeare training loss on 512 fixed sequences at a fixed parameter budget, after 5,000 Adam updates (Appendix H.5 ). Right: GPT-2 training loss averaged over the final 100 updates at matched parameter and token budgets (Appendix I.1 ).

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer	Quantity	W_in	W_up	W_down	W_out
Initialization std.		$1$	$1/\sqrt{N_{\flat}}$	$\sqrt{N_{\flat}}/K$	$1/N_{\flat}$
SGD	LR (code)	$N_{\flat}$	$K/N_{\flat}$	$N_{\flat}/K$	$1/N_{\flat}$
	LR (standard Gaussian)	$N_{\flat}$	$K$	$K$	$N_{\flat}$
Muon	LR (code)	$\sqrt{N_{\flat}}$	$\sqrt{N_{\sharp}/N_{\flat}}$	$\sqrt{N_{\flat}N_{\sharp}}/K$	$1/\sqrt{N_{\flat}}$
	Clip $\lambda$ (code)	$1/\sqrt{N_{\flat}}$	$\sqrt{N_{\flat}N_{\sharp}}/K$	$\sqrt{N_{\sharp}/N_{\flat}}$	$\sqrt{N_{\flat}}$
	LR (standard Gaussian)	$\sqrt{N_{\flat}}$	$\sqrt{N_{\sharp}}$	$\sqrt{N_{\sharp}}$	$\sqrt{N_{\flat}}$

Appendix

Table 2: Complete size scalings in code and standard-Gaussian coordinates. Fixed task dimensions and hyperparameter multipliers, including

\epsilon_{0}

, are suppressed.

Panel of Figure 9 — Figure 9: Finite random histories and two scaling paths. The dashed path keeps the residual-to-hidden ratio fixed before increasing depth, and therefore has $\alpha=0$ . The diagonal permits a positive limiting $N_{\flat}/K$ . These are different scaling families, not an assertion that two identical limiting operations can be interchanged.

Figure 10: Residual activation–Jacobian pairs after learning the nonlinear logic task. Each axis is centered, with its original scale retained. Ellipses enclose 95% of the fitted Gaussian reference; no whitening is applied.

Figure 11: All-NS5 training and distributional diagnostics. Top: complete validation trajectory and excess-kurtosis ranges over the predefined slices at each measurement time; the bands are ranges, not confidence intervals. Bottom: the final forward/backward pair for the first predefined text, with a fitted Gaussian 95% ellipse, and quantiles for the retained backward slice with the largest intermediate excess kurtosis. Each plotted vector is standardized separately, without whitening.

Figure 12: Solid curves: full finite-step means; dashed curves: ( 131 ). Shading is

\pm 2

Monte Carlo standard errors, not a simultaneous confidence band. All runs freeze both endpoints and update both residual projections.

Figure 13: A stress test at

\alpha=2

. Left: the actual finite-step energy, the exact initialization-plus-unit-error tangent energy, and the continuum formula. Right: the directly measured higher-order remainder ( 150 ). Error bars are

\pm 2

Monte Carlo standard errors. The tangent curve is a diagnostic, not a replacement training rule.

Residual pair	SGD	Taylor	Adam
$h^{0\flat}(11),g_{\mathrm{AND}}^{0\flat}(11)$	$.0601\pm.0293$	$.0687\pm.0327$	$.6109\pm.0172$
$h^{0\flat}(11),h^{0\flat}(00)$	$.0667\pm.0216$	$.0712\pm.0217$	$1.1086\pm.1195$
$h^{1\flat}(11),h^{1\flat}(00)$	$.0790\pm.0327$	$.0692\pm.0303$	$.3654\pm.0591$

Appendix

Table 3: Joint fourth-cumulant RMS after 1000 updates at width 1024, reported as mean

\pm

sample standard deviation across eight seeds.

$N_{\flat}$	$N_{\sharp}$	$N_{\flat}/K$	$\gamma_{0}$	Seeds	Full-step $C_{h}(1;1)$	Limit ( 131 )
1024	16	$0.5$	1	128	$18.63\pm 0.28$	19.32
1024	16	$0.5$	2	128	$71.21\pm 2.10$	72.32
2048	16	$1$	1	256	$83.06\pm 1.40$	84.00
1024	6	$4/3$	1	256	$233.83\pm 9.35$	228.38
1024	4	$2$	1	128	$2876.95\pm 423.02$	1645.33
2048	8	$2$	1	256	$2206.47\pm 250.25$	1645.33

Appendix

Table 4: Residual energy after one finite SGD step for the squared loss. The continuum formula is not asserted to be exact at these sizes. In particular, the large-

\alpha

discrepancy remains visible.

		$18\to 36$		$36\to 72$		$72\to 144$
$q$	Epoch	$R$	$S$	$R$	$S$	$R$	$S$
24	0	0.0173	0.0023	0.0136	0.0002	0.0106	0.0016
24	1	0.2480	0.0628	0.2073	-0.0209	0.1866	0.0369
24	10	1.1092	-0.1466	1.0145	0.1718	0.9123	-0.1930
36	0	0.0124	-0.0005	0.0100	0.0002	0.0082	-0.0010
36	1	0.2118	0.0927	0.1550	-0.0097	0.1498	-0.0211

Appendix

Table 5: Adjacent regroupings of each coupled pool. All selected cases are shown; epoch zero records the initial disagreement.

Figure 14: CIFAR-10 growth recipes over the full measured range, including the shared anchor and early points. Final training cross-entropy follows five SGD epochs; bars are sample standard deviations over three seeds. All recipes use their actual block parameter counts, including the extra largest fixed-ratio point omitted from Fig. 6 .

Figure 15: CIFAR-10 learning-rate scans, shown for

m\geq 1/2

. Top: scan overview. Bottom: the same data with tighter vertical limits to resolve the basin bottoms; all six panels use identical horizontal axes. Colors identify recipes. Small points show three seeds; bars show sample standard deviations after five epochs. Stars mark grid minima; the semi-freeze minimum at

j=14

is not bracketed. The dotted line marks the anchor-selected

\sqrt{2}

. The shared anchor is drawn once. The anchor arrow at

m=8

marks three divergent runs, not a finite loss.

Parameter group	Stored-weight Adam rate	Gradient unit $s_{j}$
Token embedding	$\eta_{\rm in}\gamma$	$1/(\gamma n)$
Readout	$\eta_{\rm out}\gamma$	$1/(\gamma n)$
MLP up	$\eta_{\rm bulk}\gamma\beta_{\rm b}/\sqrt{n}$	$\beta_{\rm b}/(\gamma K_{\rm b}\sqrt{n})$
MLP down	$\eta_{\rm bulk}\gamma\beta_{\rm b}/\sqrt{n}$	$\beta_{\rm b}/(\gamma K_{\rm b}\sqrt{n})$
Attention Q	$\eta_{\rm bulk}\gamma\beta_{\rm h}/\sqrt{n}$	$\beta_{\rm h}/(\gamma K_{\rm h}\sqrt{n})$
Attention K	$\eta_{\rm bulk}\gamma\beta_{\rm h}/\sqrt{n}$	$\beta_{\rm h}/(\gamma K_{\rm h}\sqrt{n})$

Appendix

Table 6: The eight Adam groups. The compensated-coupling recipe changes the bulk constants and branch coefficients as specified below.

$R$	$L$	$n$	$K_{\rm h}$	$P_{\rm core}$ (M)	$n$	$K_{\rm h}$	$P_{\rm core}$ (M)
		ODE / compensated / Half Copy			Fixed ratios
4	16	512	256	25.165824	1024	128	25.165824
5	20	640	400	49.152000	1432	179	49.214976
6.25	25	800	625	96.000000	2000	250	96.000000
8	32	1024	1024	201.326592	2896	362	201.283584
10	40	1280	1600	393.216000	4048	506	393.271296

Appendix

Table 7: Actual growth geometry and core parameter counts. Every row has

K_{\rm b}=64K_{\rm h}

. Add

130n

for total parameters; the two geometry routes therefore need not have identical total parameter counts even when their core counts match.

Figure 16: Tiny Shakespeare growth recipes over the full measured range, including the shared anchor and available early points. Left: initial summed squared residual increments

Q

. Right: training loss after 5,000 Adam updates. Bars are sample standard deviations over three seeds. Half Copy has no

R=2

run; losses include

R=12.75

, where initial

Q

was not measured. Figure 6 focuses on

R\geq 4

$R$	$L=1$	$L=2$	$L=3$	$L=4$
$4$	$0.426116$	$0.595115$	$0.677284$	$0.689741$
$8$	$0.257038$	$0.339469$	—	$0.382781$

Appendix

Table 8: Final fixed-training-window cross-entropy in the separate low-depth scan. The dash denotes no completed endpoint in this evidence set.

Figure 17: Paired depth comparisons on Tiny Shakespeare at five budgets (

R=4,5,6.25,8,10

). Each value is four-block minus original-depth final training loss. Faint lines show the three paired seeds; thick lines and bars show their mean and sample standard deviation. All sixty differences are negative. The separate

R=12.75

extension is not included.

Recipe	$L=40$	$L=4$	$R=12.75,\ L=51$
ODE route	$0.4389\pm 0.0003$	$0.4085\pm 0.0067$	$0.3916\pm 0.0010$
Fixed ratios	$0.4092\pm 0.0058$	$0.3704\pm 0.0085$	$0.3680\pm 0.0025$
Compensated coupling	$0.3993\pm 0.0054$	$0.3688\pm 0.0059$	$0.3857\pm 0.0025$
Half Copy	$0.3539\pm 0.0069$	$0.3362\pm 0.0082$	$0.3059\pm 0.0038$

Appendix

Table 9: Final training loss, mean and sample standard deviation over three seeds. The first two columns are paired at

R=10

; the last column is a larger growth experiment, not part of that pair.

Figure 18: Raw residual kernels after 5,000 Tiny Shakespeare updates. Left: mean diagonal of the depth-integrated kernel, with three-seed sample standard deviations. Right: relative Frobenius dispersion across those seeds. For the integer grouping used in this experimental series, Half Copy measurements begin at

R=4

$R$	Seed	Recipe	Uniform	Width allocation	Depth mesh
8	100001	ODE	0.506104	0.477521	—
8	100001	Half Copy	0.397821	0.380109	—
8	100002	ODE	—	0.483269	0.488810
8	100002	Half Copy	—	0.402959	0.388330
10	100001	ODE	—	0.432979	0.435738
10	100001	Half Copy	—	0.339870	0.337943

Appendix

Table 10: Exploratory nonuniform allocations, each trained for 5,000 updates. Dashes indicate comparisons not included here, not failed training. The second

R=8

seed contributed to allocation selection.

Figure 19: Complete validation trajectories for the five AdamW depth allocations, all with 124,475,904 parameters and seed 1337. Left: all 78 recorded evaluations per model, including initialization. Right: an enlarged view of the second half of training. Lines connect the saved evaluations without smoothing; values retain the four-decimal precision of the logs.

Construction	Stages	Validation CE	HellaSwag
Parallel regrouping	1	3.6148	0.2808
Parallel regrouping	2	3.2359	0.2927
Parallel regrouping	4	3.1347	0.2942
Original	12	3.0764	0.3091
Split blocks	24	3.0627	0.3061

Appendix

Table 11: Completed GPT-2 comparisons at the common evaluation endpoint.

Stages	Updates	$E_{0}$	$Q$	$C$	$B_{0}$
12	0	0.000801	0.08099	$-0.001308$	$-0.00000351$
12	15,000	0.001385	7.84837	1.85463	0.006261
24	0	0.000801	0.08406	0.000121	0.00000726
24	15,000	0.001394	6.40892	3.91831	0.006519
24	19,072	0.001338	5.63515	2.70808	0.005173

Appendix

Table 12: Mean energy components in the same residual coordinates. The full-precision components satisfy the accounting identity to less than

10^{-13}

in the saved aggregates.

Figure 20: Cross-depth contributions at update 15,000 in the AdamW models. Left: the signed lag contribution

2\sum_{\ell}G_{\ell,\ell+k}

, averaged over the four probe texts and their tokens; these are magnitude-weighted inner products, not average cosines. Middle: its cumulative sum, ending at

C

. Right: regrouping the same increments into equal observation windows changes

C_{\rm window}/D

while keeping squared residual displacement

D

fixed. The dotted line marks six windows, each spanning

1/6

of the depth.

	Initialization		Update 4500
First-block variable	Skewness	Excess kurt.	Skewness	Excess kurt.
Pre-GELU	$-0.0214$	$0.0256$	$5.5374$	$54.8062$
Post-GELU	$1.6722$	$3.0864$	$10.9190$	$138.0886$
MLP increment	$0.0006$	$0.0035$	$0.0572$	$-0.0393$
Post-block residual	$0.0081$	$0.0132$	$0.0374$	$-0.1160$

Appendix

Table 13: Internal-variable contrast in the all-NS5 model. Entries are medians over sixteen separately measured first-block slices, not uncertainty estimates; coordinates and slices are not assumed independent.

Figure 21: Internal distributions at update 4500 for the first fixed text, token 1023, and blocks 1, 6, and 12. Each panel standardizes one vector separately and retains its full tails; vertical ranges differ across panels. Preactivations have 3072 coordinates, increments and residuals 768. Dashed lines give the standard Gaussian reference.

Stages	Intervention	Validation CE	HellaSwag
12	Fitted tail, HC	3.0878	0.2996
24	Two ends, HC	3.0611	0.3089
12	Mild tail, HC	3.0741	0.3075
24	Soft head, HC	3.1034	0.3037
24	Strong head, HC	3.0981	0.3011

Appendix

Table 14: All five completed adaptive candidates, seed 1337, evaluated at the same endpoint as the uniform baselines in Table 11 . Favorable and unfavorable outcomes are shown.

Neural Scaling Laws

Neural scaling laws describe how model performance changes with training data, compute, model capacity, and other resources.

See all 83 papers

Oct 22, 2026

Width and Depth Limits Commute During Training

Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by

1/\sqrt{depth}

. In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same

1/\sqrt{depth}

residual-branch scaling (Depth-

μ

P) and the

μ

P width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in

L^2

along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding

L^2

convergence does not hold in general, but we establish the weaker convergence in probability.

Ruijia Zhang, Soufiane Hayou

Neural Network Training Dynamics Deep Linear Networks Residual Learning

Ask PDF

Oct 22, 2026

Early learning in large residual networks

We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We analyze networks in a sequential limit where first width and depth diverge, then the number of training data points diverges, and finally the input dimension diverges. This is a more refined setup compared to the typical mean-field theory analysis in which depth and width diverge but the number of training datapoints and input dimension are fixed. Our main technical result is a diagrammatic computation of both the network output and the hidden representations after the few steps of gradient descent in terms of the statistics of inner products of inputs from the training set. One of our principle conceptual findings is a negative result: in the limit of large input dimension the network output can learn only a linear function of its inputs early in training since higher learned monomials are more and more suppressed in the input dimension and vanish in the large dimension limit. This suggests that mean-field limits of ResNets are surprisingly rigid in the regime of high input dimension and may not be sufficient toy models for understanding feature learning. Finally, we investigate how to ensure hyperparameter transfer for the learning rate as a function of dataset size and input dimension. We test our predictions numerically and find excellent agreement with our theory.

Vladimir Narovlansky, Boris Hanin

ORFE Department, Princeton University, Princeton, NJ 08544, USA · Racah Institute of Physics, The Hebrew University of Jerusalem, Jerusalem 91904, Israel

Neural Network Training Dynamics Residual Learning Representation Learning

Ask PDF

Jun 15, 2026

Scaling Adaptive Depth with Norm-Agnostic Residual Networks

Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth. As a result, updates from later layers become small relative to the accumulated residual state. This reduces their impact on the representation and limits the benefits of scaling models in depth. To address this, we introduce NAG, a norm-agnostic residual architecture that separates magnitude from directional information in the residual stream, preserving meaningful layer contributions throughout depth and preventing later updates from being systematically suppressed by residual-norm growth. Importantly, NAG introduces only a negligible number of additional parameters and relies on simple operations that are easily kernel-fusible, preserving training efficiency in practice. We show that this architecture outperforms baseline Transformers, with gains that increase substantially as depth grows, enabling effective training of much deeper models. The norm-agnostic formulation also leads to an interpretable Mixture-of-Depths (MoD) mechanism that adaptively skips both attention and MLP layers. Beyond serving as a post-training accuracy-compute tradeoff, this mechanism can be used as a pretraining-time scaling strategy: under iso-FLOP training, compute saved by reducing per-token forward-pass cost can be reinvested into training on more tokens while keeping the total parameter count and KV-cache budget fixed. In our experiments, moderate Mixture-of-Depths rates of approximately 20%-25% match full-depth baseline performance under equal training compute while substantially reducing the number of executed layer parameters and forward-pass FLOPs. These results identify sparsity in depth as a new scaling axis for fixed-compute training, enabling very deep yet FLOP-efficient models.

Tomás Figliolia, Beren Millidge

Zyphra San Francisco, CA

Efficient Neural Network Inference Dynamic Neural Networks Transformer Residual Learning

Ask PDF

Abstract

Neural Scaling Laws

Explore similar work

Width and Depth Limits Commute During Training

Early learning in large residual networks

Scaling Adaptive Depth with Norm-Agnostic Residual Networks