Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

Taishi Watanabe; Ryo Karakida; Jun-nosuke Teramae

stat.MLOct 26, 2026

Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

Authors: Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae

Organizations: Kyoto University · AIST · Institute of Science Tokyo

Abstract

Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easy tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.

Figures & tables

Panel of Figure 1 — Figure 1: Increasing depth improves the training-time scaling exponent for difficult tasks with $p<1$ , affects only the logarithmic correction at the boundary $p=1$ , and leaves the power-law exponent unchanged for easy tasks with $p>1$ . Dashed lines indicate the asymptotic training-time scaling laws given by Theorem 4.5 .

Panel of Figure 2 — Figure 2: Growth of the squared predictor norm distinguishes three task regimes: polynomial growth for difficult tasks with $p<1$ , logarithmic growth at the boundary $p=1$ , and bounded behavior for easy tasks with $p>1$ . Dashed lines indicate the asymptotic scaling laws given by Proposition 4.3 . Panel (b) uses a linear vertical axis to make the logarithmic growth visible.

Panel of Figure 3 — Figure 3: In reparameterized time, the loss follows a common depth-independent spectral scaling law, with exponent determined solely by the task spectrum. Dashed lines indicate the asymptotic reparameterized-time scaling laws given by Proposition 4.4 .

Figure 4: Task difficulty determines the magnitude of the depth-induced improvement in the training-time power-law exponent. Solid curves show the asymptotic exponents given by Theorem 4.5 . Markers show numerical estimates averaged over five random seeds at dimension

d=2048

, and error bars show one standard deviation across seeds. At

p=1

, we account for the logarithmic correction by fitting

\mathcal{L}(\bar{t})(\log\bar{t})^{(L-1)/L}

to a power law.

Panel of Figure 5 — Figure 5: The depth-dependent asymptotic scaling under standard initialization ( $s=1$ ) emerges with increasing dimension, while finite-size effects make this scaling harder to observe in deeper networks. (a) Loss dynamics for the $p=0.4$ task at dimension $d=8192$ . Dashed lines show the asymptotic power-law decay predicted by Corollary 5.1 , with exponents corresponding to the asymptotic values shown by the dotted horizontal lines in panel (b). (b) As dimension increases, the fitted decay exponents approach these asymptotic values. Markers show the mean over five random seeds, and error bars show one standard deviation across seeds.

Panel of Figure 6 — Figure 6: The depth-dependent acceleration persists on a task derived from benchmark image data. (a) In training time, the loss decays faster with increasing depth. (b) In reparameterized time, the loss trajectories nearly collapse onto a common spectral scaling law. (c) The squared predictor norm exhibits the predicted polynomial growth. Dashed lines show the theoretical slopes obtained from $p_{\mathrm{eff}}\approx 0.478$ : $p_{\mathrm{eff}}L/[1+p_{\mathrm{eff}}(L-1)]$ in (a), $p_{\mathrm{eff}}$ in (b), and $1-p_{\mathrm{eff}}$ in (c).

Panel of Figure 7 — Figure 7: The task-dependent effect of depth persists qualitatively in nonlinear ReLU networks. (a) For the difficult task with $p=0.6$ , deeper networks achieve lower loss at late training times. (b) For the easy task with $p=1.4$ , the loss trajectories converge and cross, with no consistent late-time advantage from increasing depth.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: The same trajectories as in Figure 1 are shown in training time $t$ , without a time shift. Vertical dotted lines mark $t_{\mathrm{e}}$ , where $\tau(t_{\mathrm{e}})=0.1$ , for $L=1,2,3,4$ .

Figure 9: The tail sum

\sum_{i>k}\lambda_{i}\beta_{i}^{2}

versus

\lambda_{k}

for the CIFAR-5M task. The dashed line is a power-law fit with

p_{\mathrm{eff}}\approx 0.478

, using

10\leq k\leq\operatorname{round}(0.9d)=2765

Scaling Laws

Scaling laws describe how model performance and learning dynamics change with model size, data, compute, training time, and other resources.

See all 74 papers

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Jul 8, 2026Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1Learning Rate Scheduling Deep Linear Networks
Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London
Unified Neural Scaling Laws
May 25, 2026Ethan Caballero, Priyank Jaini, David Krueger +1Language Model Scaling Laws Scaling Laws
Mila, University of Montreal · Google DeepMind
Emergent Inverse-Depth Scaling From Nonlinearity In Attention
Oct 8, 2026Zirui Peng, Yizhou Liu, Ziming Liu +1Neural Scaling Laws Scaling Laws
Department of Physics, Peking University · Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · College of AI, Tsinghua University

Jul 8, 2026J/K move · Enter open · S save

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara, Andrew Saxe

Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London

Learning Rate Scheduling Deep Linear Networks Gradient Descent Dynamics

Open paper AskPDF arXiv

In this short note we consider the gradient descent dynamics of deep scalar linear networks,

f(x) = \prod_{l=1}^L w_l x

, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.

Abstract

Scaling Laws

Explore similar work

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

Unified Neural Scaling Laws

Emergent Inverse-Depth Scaling From Nonlinearity In Attention