Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

Taishi Watanabe; Ryo Karakida; Jun-nosuke Teramae

stat.MLOct 26, 2026

Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

Authors: Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae

Organizations: Kyoto University · AIST · Institute of Science Tokyo

Abstract

Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easy tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.

Figures & tables

Panel of Figure 1 — Figure 1: Increasing depth improves the training-time scaling exponent for difficult tasks with $p<1$ , affects only the logarithmic correction at the boundary $p=1$ , and leaves the power-law exponent unchanged for easy tasks with $p>1$ . Dashed lines indicate the asymptotic training-time scaling laws given by Theorem 4.5 .

Panel of Figure 2 — Figure 2: Growth of the squared predictor norm distinguishes three task regimes: polynomial growth for difficult tasks with $p<1$ , logarithmic growth at the boundary $p=1$ , and bounded behavior for easy tasks with $p>1$ . Dashed lines indicate the asymptotic scaling laws given by Proposition 4.3 . Panel (b) uses a linear vertical axis to make the logarithmic growth visible.

Panel of Figure 3 — Figure 3: In reparameterized time, the loss follows a common depth-independent spectral scaling law, with exponent determined solely by the task spectrum. Dashed lines indicate the asymptotic reparameterized-time scaling laws given by Proposition 4.4 .

Figure 4: Task difficulty determines the magnitude of the depth-induced improvement in the training-time power-law exponent. Solid curves show the asymptotic exponents given by Theorem 4.5 . Markers show numerical estimates averaged over five random seeds at dimension

d=2048

, and error bars show one standard deviation across seeds. At

p=1

, we account for the logarithmic correction by fitting

\mathcal{L}(\bar{t})(\log\bar{t})^{(L-1)/L}

to a power law.

Panel of Figure 5 — Figure 5: The depth-dependent asymptotic scaling under standard initialization ( $s=1$ ) emerges with increasing dimension, while finite-size effects make this scaling harder to observe in deeper networks. (a) Loss dynamics for the $p=0.4$ task at dimension $d=8192$ . Dashed lines show the asymptotic power-law decay predicted by Corollary 5.1 , with exponents corresponding to the asymptotic values shown by the dotted horizontal lines in panel (b). (b) As dimension increases, the fitted decay exponents approach these asymptotic values. Markers show the mean over five random seeds, and error bars show one standard deviation across seeds.

Panel of Figure 6 — Figure 6: The depth-dependent acceleration persists on a task derived from benchmark image data. (a) In training time, the loss decays faster with increasing depth. (b) In reparameterized time, the loss trajectories nearly collapse onto a common spectral scaling law. (c) The squared predictor norm exhibits the predicted polynomial growth. Dashed lines show the theoretical slopes obtained from $p_{\mathrm{eff}}\approx 0.478$ : $p_{\mathrm{eff}}L/[1+p_{\mathrm{eff}}(L-1)]$ in (a), $p_{\mathrm{eff}}$ in (b), and $1-p_{\mathrm{eff}}$ in (c).

Panel of Figure 7 — Figure 7: The task-dependent effect of depth persists qualitatively in nonlinear ReLU networks. (a) For the difficult task with $p=0.6$ , deeper networks achieve lower loss at late training times. (b) For the easy task with $p=1.4$ , the loss trajectories converge and cross, with no consistent late-time advantage from increasing depth.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: The same trajectories as in Figure 1 are shown in training time $t$ , without a time shift. Vertical dotted lines mark $t_{\mathrm{e}}$ , where $\tau(t_{\mathrm{e}})=0.1$ , for $L=1,2,3,4$ .

Figure 9: The tail sum

\sum_{i>k}\lambda_{i}\beta_{i}^{2}

versus

\lambda_{k}

for the CIFAR-5M task. The dashed line is a power-law fit with

p_{\mathrm{eff}}\approx 0.478

, using

10\leq k\leq\operatorname{round}(0.9d)=2765

Scaling Laws

Scaling laws describe how model performance and learning dynamics change with model size, data, compute, training time, and other resources.

See all 74 papers

Jul 8, 2026

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

In this short note we consider the gradient descent dynamics of deep scalar linear networks,

f(x) = \prod_{l=1}^L w_l x

, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.

Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1

Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London

Learning Rate Scheduling Deep Linear Networks Gradient Descent Dynamics

Ask PDF

May 25, 2026

Unified Neural Scaling Laws

We present a functional form (that we refer to as a Unified Neural Scaling Law (UNSL)) that accurately models and extrapolates the scaling behaviors of deep neural networks as multiple dimensions all vary simultaneously (i.e. how the evaluation metric of interest varies as one simultaneously varies the number of model parameters, training dataset size, number of training steps, number of inference steps, amount of compute, and various hyperparameters) for various architectures and for each of various tasks within a varied set of upstream and downstream tasks. This set includes large-scale vision, language, math, and reinforcement learning. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set.

Ethan Caballero, Priyank Jaini, David Krueger +1

Mila, University of Montreal · Google DeepMind

Language Model Scaling Laws Scaling Laws Neural Scaling Laws

Ask PDF

Oct 8, 2026

Emergent Inverse-Depth Scaling From Nonlinearity In Attention

Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.

Zirui Peng, Yizhou Liu, Ziming Liu +1

Department of Physics, Peking University · Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · College of AI, Tsinghua University

Neural Scaling Laws Scaling Laws Attention Mechanisms Transformer Attention

Ask PDF

Abstract

Scaling Laws

Explore similar work

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

Unified Neural Scaling Laws

Emergent Inverse-Depth Scaling From Nonlinearity In Attention