Distributed training increases the number of batches processed per iteration either by scaling-out (adding more nodes) or scaling-up (increasing the batch-size). However, the largest configuration does not necessarily yield the best performance. Horizontal scaling introduces additional communication overhead, while vertical scaling is constrained by computation cost and device memory limits. Thus, simply increasing the batch-size leads to diminishing returns: training time and cost decrease initially but eventually plateaus, creating a knee-point in the time/cost versus batch-size pareto curve. The optimal batch-size therefore depends on the underlying model, data and available compute resources. Large batches also suffer from worse model quality due to the well-known generalization gap. In this paper, we present Tula, an online service that automatically optimizes time, cost, and convergence quality for large-batch training of convolutional models. It combines parallel-systems modeling with statistical performance prediction to identify the optimal batch-size. Tula predicts training time and cost within 7.5-14% error across multiple models, and achieves up to 20x overall speedup and improves test accuracy by 9% on average over standard large-batch training on various vision tasks, thus successfully mitigating the generalization gap and accelerating training at the same time.
Figures & tables
Fig. 1: Epoch time of ResNet50 on ImageNet, VGG11 on CIFAR100, AlexNet on CalTech101, and MobileNetv3 on CalTech256 across two cluster-sizes.
Fig. 2: Test accuracy vs. batch-size. Smaller batches achieve better test accuracy than large-batch training, illustrating generalization gap in the latter.
Fig. 3: A schematic overview of Tula’s workflow.
Fig. 4: (a) Mact∝ batch-size. (b) Linear model to predict Mbatch .
Fig. 5: (a) Gradient ℓ2 -norm in early iterations of ResNet50. (b) Largest Hessian eigenvalue and norm of the gradients over the iterations of VGG11.
Fig. 6: Gradient variability and test accuracy are correlated such that model convergence path is mirrored in the trajectory of the rate of gradient change.
Fig. 7: Convergence quality and Kernel Density Estimates (KDE) of iterations for batch-size 1024 under static gradient scaling U(s) with δ=0.5 and various scaling factors s . Static scaling improves accuracy over vanilla large-batch training ( s=1 ), but excessive scaling degrades convergence quality.
Fig. 8: Impact of gradient variability threshold δ . Larger δ biases training toward scaled updates, while a smaller δ favors unscaled (sensitive) updates.
Fig. 9: Training time vs. batch-size on a 16-node cluster. Dashed lines denote predictions from performance models built using full and partial-search. Both strategies accurately capture runtime trends and identify near-optimal batch-size. Larger batches offer diminishing returns; training time does not significantly reduce from the knee-point to the minimum-time configuration at the largest batch-size. However, the latter suffers considerably more generalization loss.
Fig. 10: Training cost under a memory-based pricing model that charges $0.15/hour/GB of GPU memory usage. The optimal batch-size shifts under this cost model due to increased memory footprint on larger batches.
Fig. 11: Relative error of Tula in estimating GPU memory and training time.
Model
Batch-size
Test accuracy (%)
LBT
LARS
PoLo
LRS
Tula
ResNet50
1024
79.67
83.26
74.03
83.33
85.28
2048
75.55
85.02
65.58
74.44
84.58
4096
64.72
86.77
62.46
70.11
82.16
VGG11
1024
83.30
52.93
82.90
7.50
84.42
2048
75.92
50.91
81.13
5.00
82.58
Table I: Top-1 test accuracy with vanilla large-batch training (LBT), LARS, Post-local SGD (PoLo), LR scaling (LRS), and Tula ’s AGS . For every batch-size, green → high, red → low, yellow → intermediate accuracy.
Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics. In this paper, we study the deep learning dynamics through the lens of convex optimization and derive a joint characterization of loss in terms of both schedules, applicable to general optimizers and model architectures. This characterization yields a closed-form optimal batch size schedule for any prescribed learning rate schedule, and further leads to joint scaling laws that consistently outperform static batch size baselines, highlighting the significance of dynamic batch size schedule in large language model training.
We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim et al. 2026) and show that its frontier depends critically on choices previously fixed or studied only in isolation. In particular, optimizing batch size together with epoch counts reveals a U-shaped relationship between validation loss and batch size, favoring smaller batches than typical pretraining, likely because small-batch gradient noise provides implicit regularization. We further find that training recipes, previously optimized under a single-pass assumption, can be tuned, e.g., repacking documents into new sequences every epoch, weight tying, and positive embedding weight decay. We then relax the infinite-compute assumption. In particular, smaller-than-typical batch sizes can require prohibitively long sequential training, motivating to treat GPUs and time as separate resources. Tighter deadlines favor larger batches, and latency-oriented parallelism can trade additional GPUs for shorter training. Nevertheless, even under a fixed time budget, a smaller batch with fewer GPUs can achieve lower validation loss than a larger batch with more GPUs, challenging the common practice of increasing data parallelism to maximize GPU utilization under a deadline. Together, our results identify underexplored factors in repeated-data training and provide concrete guidance on how training recipes and resource allocation should change from single-pass pretraining.
Scaling laws relate model quality to compute budget (FLOPs), but practitioners face wall-clock time constraints, not compute budgets. We study optimal model sizing under fixed time budgets from 5 minutes to 24 hours on consumer GPUs (RTX 4090). Across 70+ runs spanning 50M--1031M parameters, we find: (1)~at each time budget a U-shaped curve emerges where too-small models overfit and too-large models undertrain; (2)~optimal model size follows N∗∝t0.60, growing faster than Chinchilla's N∗∝C0.50, with α=0.60±0.07 robustly exceeding compute-optimal across all sensitivity analyses; (3)~a dual U-shape mechanism: short-budget U-curves arise from compute bottlenecks, while long-budget U-curves emerge from data bottlenecks (overfitting), with an intermediate regime where the U-curve temporarily disappears. These findings have immediate implications for researchers training on consumer hardware, where wall-clock time -- not FLOPs -- is the binding constraint. We release all code, logs, and 70+ experimental configurations.