Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs

Yi Liu

cs.PFMar 29, 2026

Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs

Authors: Yi Liu

Organizations: Independent Researcher

Abstract

Scaling laws relate model quality to compute budget (FLOPs), but practitioners face wall-clock time constraints, not compute budgets. We study optimal model sizing under fixed time budgets from 5 minutes to 24 hours on consumer GPUs (RTX 4090). Across 70+ runs spanning 50M--1031M parameters, we find: (1)~at each time budget a U-shaped curve emerges where too-small models overfit and too-large models undertrain; (2)~optimal model size follows $N^* \propto t^{0.60}$ , growing faster than Chinchilla's $N^* \propto C^{0.50}$ , with $α= 0.60 \pm 0.07$ robustly exceeding compute-optimal across all sensitivity analyses; (3)~a dual U-shape mechanism: short-budget U-curves arise from compute bottlenecks, while long-budget U-curves emerge from data bottlenecks (overfitting), with an intermediate regime where the U-curve temporarily disappears. These findings have immediate implications for researchers training on consumer hardware, where wall-clock time -- not FLOPs -- is the binding constraint. We release all code, logs, and 70+ experimental configurations.

Figures & tables

Model	Params	5min	30min	1h	2h	4h	8h	12h	24h
D8	50M	1.133	0.977	0.979 $\uparrow$	0.906	0.925 $\uparrow$	0.886	0.919 $\uparrow$	—
D10	86M	1.178	0.973	0.976	0.906	0.892	0.886	0.885	—
D12	135M	1.363	1.001	0.991	0.904	0.878	0.873	0.871	0.870
D14	201M	1.578	1.016	0.945	0.901	0.866	0.854	0.852	0.857 $\uparrow$
D16	285M	1.566	1.026	0.951	0.901	0.862	0.844	0.841	0.851 $\uparrow$
D18	384M	—	—	—	—	0.866	0.837	0.833	0.845 $\uparrow$

Table 1: Validation BPB across model sizes and time budgets. Bold = optimal at each budget.

\uparrow

= overfitting (worse than shorter training). At 12h, no U-shape: BPB decreases monotonically with model size. At 24h, U-shape returns due to data bottleneck.

Model	Params	Seed 42	Seed 123	Seed 456	Mean $\pm$ Std	CV (%)
D8	50M	0.977	0.975	0.975	$0.976\pm 0.001$	0.13
D10	86M	0.973	0.974	0.973	$\mathbf{0.973\pm 0.000}$	0.04
D14	201M	1.016	1.022	1.017	$1.018\pm 0.003$	0.29
D16	285M	1.026	1.032	1.029	$1.029\pm 0.003$	0.31

Table 2: Multi-seed validation at 30 minutes. Three seeds per model. Max

\sigma=0.003

BPB vs. inter-model gaps of 0.002–0.056 BPB. D10 is optimal across all seeds.

Figure 1: Main result. (a) U-shaped optimality curves at 8 time budgets (5min–24h). Stars mark optimal model at each budget. (b) Optimal model size vs. time, fitted with $N^{*}=14.2\times t^{0.595}$ ( $R^{2}=0.963$ ). The blue dotted line shows Chinchilla’s $\alpha=0.50$ for reference. (c) Best achievable BPB vs. time, fitted with $L^{*}=1.22\times t^{-0.061}$ ( $R^{2}=0.971$ ), showing severe diminishing returns.

Panel of Figure 2 — Figure 2: Key phenomena. Left: the dual U-shape mechanism across three regimes. Right: training trajectories showing model-specific overfitting dynamics. Additional figures ( $\alpha$ convergence, heatmap, dual U-shape detail) in Appendix.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

DEPTH	Layers	Dim	Params (M)	tok/sec	MFU (%)
8	8	512	50.3	428K	10.4
10	10	640	85.9	252K	10.7
12	12	768	135.3	160K	10.9
14	14	896	200.9	110K	11.4
16	16	1024	285.2	78K	11.4
18	18	1152	$\sim$ 384	56K	$\sim$ 11

Appendix

Table 3: Full model configurations parameterized by DEPTH. Throughput measured on single RTX 4090.

Variant	$\alpha$	95% CI	$R^{2}$
8-point (baseline)	$0.595\pm 0.067$	$[0.53,0.67]$	0.963
7-point (exclude 24h)	$0.747\pm 0.107$	$[0.64,0.86]$	0.957
7-point (2h = 285M)	$0.706\pm 0.110$	$[0.60,0.82]$	0.948
6-point (exclude 2h)	$0.805\pm 0.145$	$[0.66,0.95]$	0.960

Appendix

Table 4: Sensitivity of the scaling exponent

\alpha

to fitting choices. All variants yield

\alpha>0.50

Interval	$\Delta t$	$\Delta$ BPB	BPB/hour	Efficiency
5min $\to$ 30min	25min	$-0.160$	$-0.384$	Very high
30min $\to$ 1h	30min	$-0.028$	$-0.056$	High
1h $\to$ 2h	1h	$-0.044$	$-0.044$	Moderate
2h $\to$ 4h	2h	$-0.039$	$-0.020$	Low
4h $\to$ 8h	4h	$-0.026$	$-0.007$	Very low
8h $\to$ 12h	4h	$-0.012$	$-0.003$	Diminishing

Appendix

Table 5: Marginal BPB improvement per time increment. Returns diminish rapidly after 1 hour.

Model	Params	BPB 12h	BPB 24h	$\Delta$
D12	135M	0.871	0.870	$-0.001$ (saturated)
D14	201M	0.852	0.857	$+0.005$ (overfitting)
D16	285M	0.841	0.851	$+0.010$ (overfitting)
D18	384M	0.833	0.845	$+0.012$ (overfitting)
D20	519M	0.828	0.838	$+0.010$ (overfitting)
D22	621M	0.826	0.829	$+0.003$ (marginal)

Appendix

Table 6: Change in BPB from 12h to 24h. Positive values indicate overfitting. Only models

\geq

856M continue improving.

Time multiplier	Ours ( $2^{0.60}$ )	Chinchilla ( $2^{0.50}$ )
$2\times$ time	$1.52\times$ model	$1.41\times$ model
$4\times$ time	$2.30\times$ model	$2.00\times$ model
$10\times$ time	$3.98\times$ model	$3.16\times$ model
$24\times$ time (1h $\to$ 24h)	$7.22\times$ model	$4.90\times$ model

Appendix

Table 7: Model size recommendations: time-constrained (ours,

\alpha=0.60

) vs. compute-constrained (Chinchilla,

\alpha=0.50

Figure 3: BPB heatmap across model sizes and time budgets. The diagonal band of optimal values traces the time-constrained scaling law. Gold boxes mark optimal configurations.

Figure 4: $\alpha$ evolution as time points are added: 0.44 (5pt) $\to$ 0.55 (6pt) $\to$ 0.75 (7pt) $\to$ 0.60 (8pt). The non-monotonic convergence reflects regime transitions (Section 5 ).

Figure 5: Dual U-shape mechanism. (a) Change in BPB from 12h to 24h per model. Models D14–D20 overfit (red bars, positive $\Delta$ ), while D24–D26 continue improving (green bars, negative $\Delta$ ). (b) Conceptual illustration of the dual U-shape.

Figure 6: Multi-seed validation. (a) Validation BPB for 3 seeds per model at 30 minutes. Error bars show $\pm 1\sigma$ . Individual seed values (markers) overlap almost perfectly. (b) Inter-model gaps (red) vs. inter-seed variance (gray). Model choice matters orders of magnitude more than random seed.

Time budget	Model size	DEPTH	Expected BPB
5 minutes	50M	D8	1.133
30 minutes	86M	D10	0.973
1 hour	200M	D14	0.945
2 hours	200–285M	D14–D16	0.901
4 hours	285M	D16	0.862
8 hours	519M	D20	0.836

Appendix

Table 8: Recommended model sizes for RTX 4090 training with FineWeb-Edu-scale data.

Scaling Laws

Scaling laws describe how model performance and learning dynamics change with model size, data, compute, training time, and other resources.

See all 72 papers

Amortizing Scaling Law Construction Costs
Sep 4, 2026Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6Language Model Scaling Laws Bayesian Optimization
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4
Prescriptive Scaling Laws for Data Constrained Training
May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight Decay Language Model Scaling Laws
Department of Computer Science Cornell University Ithaca, NY
Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
May 9, 2026Christopher M. Bryant, Hao LiuLanguage Model Scaling Laws Neural Scaling Laws
Arena Physica

Sep 4, 2026J/K move · Enter open · S save

Amortizing Scaling Law Construction Costs

Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik, Swagatam Haldar+5

ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology+4

Language Model Scaling Laws Bayesian Optimization Hyperparameter Optimization

Open paper AskPDF arXiv

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to

10\text{--}100\times

Abstract

Scaling Laws

Explore similar work

Amortizing Scaling Law Construction Costs

Prescriptive Scaling Laws for Data Constrained Training

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World