Nowcasting Future Performance for Population Pruning in Survival-of-the-Fittest Training

Rico Städeli; Léo Meynent; Boris Knyazev; Damian Borth

cs.LGNov 4, 2026

Nowcasting Future Performance for Population Pruning in Survival-of-the-Fittest Training

Authors: Rico Städeli, Léo Meynent, Boris Knyazev, Damian Borth

Organizations: University of St.Gallen · University of Montreal · Weight Space Labs

Abstract

Training high-performing neural networks often involves evaluating populations of candidate models with varying hyperparameters, architectures, or random seeds. Training every candidate to completion is computationally expensive, yet stopping seemingly weaker runs early risks discarding models that would ultimately perform better, as learning curves can cross. To address this challenge, we investigate whether intermediate training checkpoints can predict future model accuracy and enable more informed pruning decisions. Our nowcasters predict test accuracy five epochs ahead or at the end of training. They are trained with accuracy and loss histories, weight statistics, and SANE representations as inputs. SANE is a transformer autoencoder that encodes a checkpoint as a sequence of latent tokens. We then use the nowcasters in Survival-of-the-Fittest (SOTF) training. SOTF repeatedly ranks runs during training by predicted accuracy and stops the lowest-ranked ones. Our results show that weight statistics work best for five-epoch forecasts, and SANE representations work best for final accuracy prediction. For evaluation, we compare SOTF with pruning the population by current accuracy. In homogeneous model zoos such as the CNN zoos, pruning by current accuracy already retains a final model only slightly worse than the best. Model zoos with a larger spread in performance, such as the ResNet-18 and ViT zoos, are more difficult to prune by current accuracy alone. In these zoos, SOTF with nowcasted final accuracy almost closes the gap to the best model.

Figures & tables

Figure 1: Overview of the methodology. Input to the pipeline is a raw model zoo containing checkpoints with weights and recorded performance metrics. The pipeline produces aligned checkpoint-level datasets with weight-space representations and performance metrics. These datasets are used for descriptive learning-dynamics analysis and for training nowcasting predictors. The trained predictors are then held fixed during the Survival-of-the-Fittest training (SOTF training) on a held-out population, where weak models are removed before the end of their recorded trajectories.

Figure 2: Survival-of-the-Fittest training. Starting at epoch $e_{0}$ , the same fixed predictor, trained beforehand, estimates either performance five epochs ahead (left) or final performance (right). At intervals of $\Delta$ epochs, active models are ranked by their nowcasted test accuracy and the top fraction $\rho$ is retained. The procedure stops when one model remains or the recorded training horizon is reached.

Figure 3: Tiny ImageNet ResNet-18. The upper row shows pairwise rank changes: blue denotes the raw number of cross-overs and orange the number of

\epsilon

-significant cross-overs. The lower row shows Kendall’s

\tau

between the ranking at epoch

e

and the final ranking.

Figure 4: Distribution of test accuracy. Distribution of test accuracy over training for three model zoos.

Figure 5: Nowcasting feature ablation. Bars show median validation

R^{2}

for performance-only, representation-only, and hybrid configurations. Values below zero are clipped to zero. A performance-only token-Transformer variant is undefined because its architecture requires a latent-token sequence.

Figure 6: Predictor-family comparison. Each cell reports validation $R^{2}$ (top) and RMSE (bottom) for the selected configuration; outlines mark the lowest-RMSE learned predictor for each zoo and target.

	Final value		Horizon 5
Ranking policy	CNN	ResNet-18 / ViT	CNN	ResNet-18 / ViT
Current accuracy	0.0051 / 0.48	0.0643 / 0.11	0.0051 / 0.48	0.0629 / 0.11
LC extrapolation	0.0062 / 0.47	0.0689 / 0.05	0.0050 / 0.51	0.0490 / 0.05
LCNet	0.0056 / 0.48	0.0247 / 0.10	0.0104 / 0.43	0.0656 / 0.11
XGB (perf.)	0.0040 / 0.45	0.0051 / 0.20	0.0039 / 0.46	0.0495 / 0.09
XGB (weights)	0.0060 / 0.34	0.0018 / 0.67	0.0056 / 0.37	0.0608 / 0.09

Table 1: SOTF training results by population regime. Each cell reports the mean best-retained-model gap / mean oracle top-

K

recall over all start epochs and retention rates, separately for the six CNN zoos and the three ResNet-18 and ViT zoos. Lower gap and higher recall are better; the best value per column is in bold. “perf.” denotes accuracy and loss history features, “weights” engineered weight statistics, and “SANE” SANE latent representations. All predictors are trained once and held fixed. Results for all policies are in Appendix F .

Figure 7: SOTF training sensitivity on the Tiny ImageNet ResNet-18 zoo. The latent-token Transformer, trained beforehand and held fixed throughout SOTF training, is compared with learning-curve extrapolation for final-value nowcasting. Negative gap differences and positive recall differences favor SOTF training. Each cell additionally reports compute savings (top right) and final population size $n$ (bottom left). “Best survivor gap” denotes the best-retained-model gap defined in Equation 3 .

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset name	Image data	Architecture	Population	Models	Epochs
cifar10_cnn_large_uniform	CIFAR-10	CNN (l)	Uniform	1000	50
cifar10_cnn_large_hyp_random_seeds	CIFAR-10	CNN (l)	Hyp. random	1280	50
cifar10_cnn_large_hyp_fixed_seeds	CIFAR-10	CNN (l)	Hyp. fixed	1280	50
stl10_cnn_large_uniform	STL-10	CNN (l)	Uniform	1000	50
stl10_cnn_large_hyp_random_seeds	STL-10	CNN (l)	Hyp. random	2560	50
stl10_cnn_large_hyp_fixed_seeds	STL-10	CNN (l)	Hyp. fixed	2560	50

Appendix

Table 2: Model zoo corpus. The dataset name is used to refer to the model zoo throughout the paper. Architecture abbreviations: CNN (l) for large convolutional networks, RN-18 width for ResNet-18 populations with varying width, and ViT for Vision Transformers.

Figure 8: Accuracy distributions across all model zoos. Test-accuracy distributions at each checkpoint for all nine model zoos. Boxes summarize the population-level spread and reveal the contrast between narrow uniform zoos and heterogeneous architecture or width variants.

Figure 9: Pairwise rank flips — CIFAR-10 uniform CNN zoo. Raw and smoothed pairwise rank flips between consecutive epochs.

Figure 10: Pairwise rank flips — CIFAR-10 CNN Hyp-10-rand. Raw and smoothed pairwise rank flips between consecutive epochs.

Figure 11: Pairwise rank flips — CIFAR-10 CNN Hyp-10-fix. Raw and smoothed pairwise rank flips between consecutive epochs.

Figure 12: Pairwise rank flips — CIFAR-10 ResNet-18 width zoo. Raw and smoothed pairwise rank flips between consecutive epochs.

Figure 13: Pairwise rank flips — CIFAR-10 ViT zoo. Raw and smoothed pairwise rank flips between consecutive epochs.

Figure 14: Pairwise rank flips — STL-10 uniform CNN zoo. Raw and smoothed pairwise rank flips between consecutive epochs.

Figure 15: Pairwise rank flips — STL-10 CNN Hyp-10-rand. Raw and smoothed pairwise rank flips between consecutive epochs.

Figure 16: Pairwise rank flips — STL-10 CNN Hyp-10-fix. Raw and smoothed pairwise rank flips between consecutive epochs.

Family	Included statistics	Captured property
Norm	$\ell_{1}$ , $\ell_{2}$ , and $\ell_{\infty}$ norms	Overall parameter scale and the presence of large individual values.
Distribution	Mean, variance, percentiles, and range	Center, spread, and tails of the flattened parameter distribution.
Matrix	Frobenius norm, mean and maximum absolute value, root mean square, dimensions, aspect ratio, and row- and column-wise norms	Layer shape, scale, and variation across input and output directions.
Distance	Distance and cosine similarity to initialization and to the previous checkpoint, cumulative path length, and path straightness	Movement of the full model through parameter space during training.

Appendix

Table 3: Checkpoint-derived weight-statistic families.

Parameter	Ridge	XGBoost
Objective / penalty	Squared error $+\ \ell_{2}$	Squared error
Regularization strength	$\eta_{\mathrm{ridge}}=1.0$	$\ell_{2}$ leaf reg. $=1.0$ , $\ell_{1}$ leaf reg. $=0$
Boosting rounds	—	400
Maximum tree depth	—	6
Learning rate	—	0.05
Row / feature subsampling	—	0.8 / 0.8

Appendix

Table 4: Ridge and XGBoost configurations.

Parameter	Weight-stat. MLP	Avg.-latent MLP	Latent-window LSTM	Latent-token Transf.
Input dimension	Weight-statistic vector	128 (SANE latent)	128 (window embedding)	128 (token embedding)
Architecture	Hidden widths 256, 128, 64; LayerNorm	Hidden widths 256, 128, 64; 128, 64; or 64, 32	LSTM hidden size 64, 1 layer	Internal dim. 64, 4 heads, 1 local + 2 global layers
Local chunk size $w$	—	—	—	8, 16, 32, 128, 256, or 512
Regression head	Linear readout	Linear readout	Width 32, GELU	Width 32, GELU
Activation	GELU	GELU	—	—
Dropout	0.1	0 or 0.1	0	0

Appendix

Table 5: Feed-forward and sequence predictor configurations for the SANE representation and weight-statistic MLPs.

Figure 17: Learning-curve extrapolation baseline performance. Validation RMSE and

R^{2}

for the parametric learning-curve extrapolation baseline on final-value and horizon-5 nowcasting. The

R^{2}

axis is clipped at 0 for readability.

Figure 18: LCNet learning-curve baseline performance. Validation RMSE and

R^{2}

for the LCNet-style baseline on final-value and horizon-5 nowcasting.

Figure 19: Averaged-latent ridge performance by dataset. Validation RMSE and

R^{2}

for final-value and horizon-5 nowcasting from the mean-pooled SANE embedding.

Figure 20: Weight-statistic XGBoost performance by dataset.

Figure 21: Averaged-latent XGBoost performance by dataset. The ViT zoo is a notable failure case: even with performance features appended, the mean-pooled checkpoint vector provides little reliable information about final accuracy beyond the learning-curve position.

Figure 22: Weight-statistic MLP performance by dataset. Competitive for final-value nowcasting but selected less consistently than XGBoost.

Figure 23: Averaged-latent MLP performance by dataset.

Figure 24: Latent-window LSTM performance by dataset. Strongest on final-value prediction for the trajectory-sensitive Tiny ImageNet phase-transition and CIFAR-10 ResNet-18 width zoos.

Figure 25: Latent-token Transformer performance by dataset.

Figure 26: Predictor improvement over the LCNet baseline. Validation RMSE reduction relative to LCNet for the best weight-statistic and best SANE-representation predictor per dataset and target mode. Bars at zero indicate no improvement; labels report the selected predictor family.

Figure 27: Performance-feature ablation — RMSE view. Median validation RMSE (lower is better) for the performance-only, representation-only, and hybrid ablation configurations shown as

R^{2}

in Figure 5 . The performance-only ablation for the latent-token Transformer was not run, since that architecture requires a latent-token sequence as input.

Configuration	Predictor used for ranking
Naive current-value baseline	Current observed test accuracy.
LC Extrapolation baseline	Parametric learning-curve fit applied to the observed accuracy history.
LCNet baseline	LCNet-style predictor applied to the observed accuracy history. Final value: the predictor from the metric-prediction experiments, trained on the complete training-split learning curves of the zoo and held fixed. Horizon 5: refit at every pruning round on the history observed so far.
Weight-statistic XGBoost variants	Predictors loaded from the metric-prediction experiments, using weight-statistic features with and without observed metrics.
SANE latent variants	Predictors loaded from the metric-prediction experiments, using averaged latent, latent-window LSTM, or latent-token Transformer predictors for selected zoos.

Appendix

Table 7: SOTF training ranking configurations. Complete list of ranking policies compared in SOTF training experiments.

Model zoo	Horizon-5 start epochs	Final-value start epochs
cifar10_cnn_large_uniform	5, 10, 25	5, 10, 25
cifar10_cnn_large_hyp_random_seeds	5, 10, 25	5, 10, 25
cifar10_cnn_large_hyp_fixed_seeds	5, 10, 25	5, 10, 25
stl10_cnn_large_uniform	5, 10, 25	5, 10, 25
stl10_cnn_large_hyp_random_seeds	5, 10, 25	5, 10, 25
stl10_cnn_large_hyp_fixed_seeds	5, 10, 25	5, 10, 25

Appendix

Table 8: SOTF training model-zoo coverage and start epochs.

	Final value		Horizon 5
Ranking policy	CNN	ResNet-18 / ViT	CNN	ResNet-18 / ViT
Current accuracy	0.0051 / 0.48	0.0643 / 0.11	0.0051 / 0.48	0.0629 / 0.11
LC extrapolation	0.0062 / 0.47	0.0689 / 0.05	0.0050 / 0.51	0.0490 / 0.05
LCNet	0.0056 / 0.48	0.0247 / 0.10	0.0104 / 0.43	0.0656 / 0.11
XGB (perf.)	0.0040 / 0.45	0.0051 / 0.20	0.0039 / 0.46	0.0495 / 0.09
MLP (perf.)	0.0038 / 0.48	0.0082 / 0.22	0.0047 / 0.46	0.0494 / 0.14

Appendix

Table 9: SOTF training results for all ranking policies by population regime. Each cell reports the mean best-retained-model gap / mean oracle top-

K

recall, separately for the six CNN zoos and the three ResNet-18 and ViT zoos. Final value ranks by predicted test accuracy at the end of training; horizon 5 ranks by predicted test accuracy five epochs ahead. Lower gap and higher recall are better; the best value per column is in bold. “perf.” denotes accuracy and loss history features, “weights” engineered weight statistics, and “SANE” SANE representations.

Ranking policy	Tiny ImageNet ResNet-18	CIFAR-10 ResNet-18 width	CIFAR-10 ViT
Current accuracy	0.0387 / 0.16	0.0249 / 0.19	0.1267 / 0.00
LC extrapolation	0.0461 / 0.13	0.0223 / 0.03	0.1351 / 0.00
LCNet	0.0050 / 0.17	0.0031 / 0.04	0.0645 / 0.07
XGB (perf.)	0.0063 / 0.25	0.0037 / 0.21	0.0053 / 0.13
XGB (weights)	0.0051 / 0.26	0.0000 / 0.99	0.0001 / 0.80
XGB (perf. + weights)	0.0051 / 0.26	0.0000 / 0.99	0.0000 / 0.93

Appendix

Table 10: Final-value SOTF training results per ResNet-18 and ViT zoo. Each cell reports the mean best-retained-model gap / mean oracle top-

K

recall over start epochs and retention rates. Lower gap and higher recall are better; the best value per column is in bold.

Figure 28: TinyImageNet phase, final value: weight-statistic XGBoost with performance features; mostly robust improvements.

Figure 29: CIFAR-10 ViT, final value: latent-token Transformer; mixed results across pruning schedules.

Figure 30: TinyImageNet phase, final value: averaged-latent MLP; consistently better under meaningful pruning.

Figure 31: TinyImageNet phase, final value: averaged-latent XGBoost with performance features; robust improvements.

Figure 32: TinyImageNet phase, final value: latent-token Transformer with performance features; broadly effective.

Figure 33: TinyImageNet phase, final value: latent-window LSTM with performance features; mostly robust improvements.

Figure 34: CIFAR-10 ViT, final value: averaged-latent MLP; unstable and frequently worse than LC Extrapolation.

Figure 35: CIFAR-10 ViT, final value: averaged-latent XGBoost; strong early gains but poor transfer to later starts.

Figure 36: CIFAR-10 ViT, final value: latent-window LSTM; effective early but unreliable elsewhere.

Figure 37: CIFAR-10 ViT, final value: performance-only MLP; large early gains coexist with later degradations.

Figure 38: CIFAR-10 ViT, horizon value: averaged-latent MLP; predominantly worse than LC Extrapolation.

Figure 39: CIFAR-10 ViT, horizon value: averaged-latent XGBoost; mostly unsuccessful except at isolated settings.

Figure 40: CIFAR-10 ViT, horizon value: latent-token Transformer; consistently poor at moderate pruning rates.

Figure 41: CIFAR-10 ViT, horizon value: latent-window LSTM; generally worse despite high compute savings.

Figure 42: CIFAR-10 width-varying ResNet-18, final value: latent-window LSTM; helps mainly when pruning starts at epoch 40 or 60.

Learning Curve Prediction

See all 10 papers

Apr 6, 2025

Hyperflux: Pruning Reveals Importance

Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most methods focus on empirical results at the expense of understanding the pruning process. We introduce Hyperflux, a novel

L_0

method which models pruning as a continuously evolving system determined by flux, the gradient response to a weight's removal, and pressure, a global regularization driving weights toward pruning. By exploiting this model, Hyperflux's pruning behavior becomes understandable at both microscopic (weight regrowth/pruning) and macroscopic (sparsity convergence, etc.) levels. We also introduce a novel pressure scheduler that reliably targets desired sparsities. Hyperflux achieves competitive results with ResNet-50, VGG-19 and DeiT-T/S on CIFAR-10, CIFAR-100 and ImageNet datasets.

Eugen Barbulescu, Antonio Alexoaie, Lucian Busoniu

Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca

Sparse Neural Networks Neural Network Training Dynamics Neural Network Pruning

Ask PDF

May 23, 2026

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component

R

has small propagated logit effect

L_s \| R ψ_1(s) \|_\infty

, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive

L_2

-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge

σ_+

gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16

2{:}4{+}

ToMe reaches

83.41\%

top-1 (

-1.70

pp from dense) at

59.81\%

sparse-execution MAC reduction, with

1.388\times

best-observed A40 native-

2{:}4

backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives

2.705\times

. At structured sparsity, ViT-B/16

6{:}12

reaches

83.74\%

, ViT-L/16

8{:}16

dense+permutation reaches

85.33\%

(

-0.51

pp), and ConvNeXtV2-Base

12{:}16

reaches

86.35\%

(

-0.37

pp). For CNNs, ResNet50

8{:}16

dense+permutation reaches

75.87\%

(

-0.26

pp), and ResNet152d CAST-conv+permutation reaches

81.33\%

(

-1.53

pp) at

{\sim}50\%

MAC accounting with a

1.62\times

A40 im2col

+2{:}4

sparse-GEMM audit.

Leonid Berlyand, Theo Bourdais, Houman Owhadi +1

Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA

Structured Pruning Sparse Neural Networks Image Classification Neural Network Pruning

Ask PDF

Aug 4, 2026

Predicting Deep Neural Network Training Outcomes from Early Training Telemetry

Large hyperparameter sweeps for deep neural networks spend substantial compute on configurations that are effectively doomed from the first few epochs. We study whether a single training run's own early telemetry - per-epoch loss, training accuracy, gradient signal-to-noise ratio, weight-norm growth, and an activation-saturation snapshot - together with its sampled hyperparameters, can predict that run's eventual outcome without reference to other runs. We evaluate three prediction tasks: final test accuracy, relative performance within a domain, and training-dynamics failure, including numerical divergence. Across 23,788 training runs spanning six architecture/dataset combinations, gradient-boosted trees using only the first five epochs of telemetry achieve R^2 = 0.92-0.99 for final-accuracy regression and ROC-AUC = 0.983-0.998 for relative classification on a permanently held-out set of hyperparameter configurations. Useful prediction is already available after a single epoch. A paired ablation shows that gradient- and weight-level telemetry provides a statistically consistent improvement over loss and accuracy curves alone, although the practical gain varies by domain. Transfer is strong between similar architectures, while cross-dataset transfer is limited mainly by differences in accuracy scale rather than loss of the underlying relationship. These results suggest that early-training telemetry can provide a practical decision-support signal for compute allocation while motivating human oversight for any automated intervention.

Ranjita Naik, Anh D. Nguyen, Pankaj Kumar Singh

Georgia Institute of Technology

Learning Curve Prediction Neural Network Training Dynamics Hyperparameter Optimization

Ask PDF

Abstract

Learning Curve Prediction

Explore similar work

Hyperflux: Pruning Reveals Importance

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

Predicting Deep Neural Network Training Outcomes from Early Training Telemetry