cs.LGNov 4, 2026
SaveNowcasting Future Performance for Population Pruning in Survival-of-the-Fittest Training
Organizations: University of St.Gallen · University of Montreal · Weight Space Labs
Abstract
Training high-performing neural networks often involves evaluating populations of candidate models with varying hyperparameters, architectures, or random seeds. Training every candidate to completion is computationally expensive, yet stopping seemingly weaker runs early risks discarding models that would ultimately perform better, as learning curves can cross. To address this challenge, we investigate whether intermediate training checkpoints can predict future model accuracy and enable more informed pruning decisions. Our nowcasters predict test accuracy five epochs ahead or at the end of training. They are trained with accuracy and loss histories, weight statistics, and SANE representations as inputs. SANE is a transformer autoencoder that encodes a checkpoint as a sequence of latent tokens. We then use the nowcasters in Survival-of-the-Fittest (SOTF) training. SOTF repeatedly ranks runs during training by predicted accuracy and stops the lowest-ranked ones. Our results show that weight statistics work best for five-epoch forecasts, and SANE representations work best for final accuracy prediction. For evaluation, we compare SOTF with pruning the population by current accuracy. In homogeneous model zoos such as the CNN zoos, pruning by current accuracy already retains a final model only slightly worse than the best. Model zoos with a larger spread in performance, such as the ResNet-18 and ViT zoos, are more difficult to prune by current accuracy alone. In these zoos, SOTF with nowcasted final accuracy almost closes the gap to the best model.
Figures & tables
Figure 1: Overview of the methodology. Input to the pipeline is a raw model zoo containing checkpoints with weights and recorded performance metrics. The pipeline produces aligned checkpoint-level datasets with weight-space representations and performance metrics. These datasets are used for descriptive learning-dynamics analysis and for training nowcasting predictors. The trained predictors are then held fixed during the Survival-of-the-Fittest training (SOTF training) on a held-out population, where weak models are removed before the end of their recorded trajectories.
Figure 2: Survival-of-the-Fittest training. Starting at epoch , the same fixed predictor, trained beforehand, estimates either performance five epochs ahead (left) or final performance (right). At intervals of epochs, active models are ranked by their nowcasted test accuracy and the top fraction is retained. The procedure stops when one model remains or the recorded training horizon is reached.
Figure 3: Tiny ImageNet ResNet-18. The upper row shows pairwise rank changes: blue denotes the raw number of cross-overs and orange the number of -significant cross-overs. The lower row shows Kendall’s between the ranking at epoch and the final ranking.
Figure 4: Distribution of test accuracy. Distribution of test accuracy over training for three model zoos.
Figure 5: Nowcasting feature ablation. Bars show median validation for performance-only, representation-only, and hybrid configurations. Values below zero are clipped to zero. A performance-only token-Transformer variant is undefined because its architecture requires a latent-token sequence.
Figure 6: Predictor-family comparison. Each cell reports validation (top) and RMSE (bottom) for the selected configuration; outlines mark the lowest-RMSE learned predictor for each zoo and target.
| Final value | Horizon 5 | |||
|---|---|---|---|---|
| Ranking policy | CNN | ResNet-18 / ViT | CNN | ResNet-18 / ViT |
| Current accuracy | 0.0051 / 0.48 | 0.0643 / 0.11 | 0.0051 / 0.48 | 0.0629 / 0.11 |
| LC extrapolation | 0.0062 / 0.47 | 0.0689 / 0.05 | 0.0050 / 0.51 | 0.0490 / 0.05 |
| LCNet | 0.0056 / 0.48 | 0.0247 / 0.10 | 0.0104 / 0.43 | 0.0656 / 0.11 |
| XGB (perf.) | 0.0040 / 0.45 | 0.0051 / 0.20 | 0.0039 / 0.46 | 0.0495 / 0.09 |
| XGB (weights) | 0.0060 / 0.34 | 0.0018 / 0.67 | 0.0056 / 0.37 | 0.0608 / 0.09 |
Table 1: SOTF training results by population regime. Each cell reports the mean best-retained-model gap / mean oracle top- recall over all start epochs and retention rates, separately for the six CNN zoos and the three ResNet-18 and ViT zoos. Lower gap and higher recall are better; the best value per column is in bold. “perf.” denotes accuracy and loss history features, “weights” engineered weight statistics, and “SANE” SANE latent representations. All predictors are trained once and held fixed. Results for all policies are in Appendix F .
Figure 7: SOTF training sensitivity on the Tiny ImageNet ResNet-18 zoo. The latent-token Transformer, trained beforehand and held fixed throughout SOTF training, is compared with learning-curve extrapolation for final-value nowcasting. Negative gap differences and positive recall differences favor SOTF training. Each cell additionally reports compute savings (top right) and final population size (bottom left). “Best survivor gap” denotes the best-retained-model gap defined in Equation 3 .
Appendix figures & tables43 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset name | Image data | Architecture | Population | Models | Epochs |
|---|---|---|---|---|---|
| cifar10_cnn_large_uniform | CIFAR-10 | CNN (l) | Uniform | 1000 | 50 |
| cifar10_cnn_large_hyp_random_seeds | CIFAR-10 | CNN (l) | Hyp. random | 1280 | 50 |
| cifar10_cnn_large_hyp_fixed_seeds | CIFAR-10 | CNN (l) | Hyp. fixed | 1280 | 50 |
| stl10_cnn_large_uniform | STL-10 | CNN (l) | Uniform | 1000 | 50 |
| stl10_cnn_large_hyp_random_seeds | STL-10 | CNN (l) | Hyp. random | 2560 | 50 |
| stl10_cnn_large_hyp_fixed_seeds | STL-10 | CNN (l) | Hyp. fixed | 2560 | 50 |
Table 2: Model zoo corpus. The dataset name is used to refer to the model zoo throughout the paper. Architecture abbreviations: CNN (l) for large convolutional networks, RN-18 width for ResNet-18 populations with varying width, and ViT for Vision Transformers.
Figure 8: Accuracy distributions across all model zoos. Test-accuracy distributions at each checkpoint for all nine model zoos. Boxes summarize the population-level spread and reveal the contrast between narrow uniform zoos and heterogeneous architecture or width variants.
Figure 9: Pairwise rank flips — CIFAR-10 uniform CNN zoo. Raw and smoothed pairwise rank flips between consecutive epochs.
Figure 10: Pairwise rank flips — CIFAR-10 CNN Hyp-10-rand. Raw and smoothed pairwise rank flips between consecutive epochs.
Figure 11: Pairwise rank flips — CIFAR-10 CNN Hyp-10-fix. Raw and smoothed pairwise rank flips between consecutive epochs.
Figure 12: Pairwise rank flips — CIFAR-10 ResNet-18 width zoo. Raw and smoothed pairwise rank flips between consecutive epochs.
Figure 13: Pairwise rank flips — CIFAR-10 ViT zoo. Raw and smoothed pairwise rank flips between consecutive epochs.
Figure 14: Pairwise rank flips — STL-10 uniform CNN zoo. Raw and smoothed pairwise rank flips between consecutive epochs.
Figure 15: Pairwise rank flips — STL-10 CNN Hyp-10-rand. Raw and smoothed pairwise rank flips between consecutive epochs.
Figure 16: Pairwise rank flips — STL-10 CNN Hyp-10-fix. Raw and smoothed pairwise rank flips between consecutive epochs.
| Family | Included statistics | Captured property |
|---|---|---|
| Norm | , , and norms | Overall parameter scale and the presence of large individual values. |
| Distribution | Mean, variance, percentiles, and range | Center, spread, and tails of the flattened parameter distribution. |
| Matrix | Frobenius norm, mean and maximum absolute value, root mean square, dimensions, aspect ratio, and row- and column-wise norms | Layer shape, scale, and variation across input and output directions. |
| Distance | Distance and cosine similarity to initialization and to the previous checkpoint, cumulative path length, and path straightness | Movement of the full model through parameter space during training. |
Table 3: Checkpoint-derived weight-statistic families.
| Parameter | Ridge | XGBoost |
|---|---|---|
| Objective / penalty | Squared error | Squared error |
| Regularization strength | leaf reg. , leaf reg. | |
| Boosting rounds | — | 400 |
| Maximum tree depth | — | 6 |
| Learning rate | — | 0.05 |
| Row / feature subsampling | — | 0.8 / 0.8 |
Table 4: Ridge and XGBoost configurations.
| Parameter | Weight-stat. MLP | Avg.-latent MLP | Latent-window LSTM | Latent-token Transf. |
|---|---|---|---|---|
| Input dimension | Weight-statistic vector | 128 (SANE latent) | 128 (window embedding) | 128 (token embedding) |
| Architecture | Hidden widths 256, 128, 64; LayerNorm | Hidden widths 256, 128, 64; 128, 64; or 64, 32 | LSTM hidden size 64, 1 layer | Internal dim. 64, 4 heads, 1 local + 2 global layers |
| Local chunk size | — | — | — | 8, 16, 32, 128, 256, or 512 |
| Regression head | Linear readout | Linear readout | Width 32, GELU | Width 32, GELU |
| Activation | GELU | GELU | — | — |
| Dropout | 0.1 | 0 or 0.1 | 0 | 0 |
Table 5: Feed-forward and sequence predictor configurations for the SANE representation and weight-statistic MLPs.
Figure 17: Learning-curve extrapolation baseline performance. Validation RMSE and for the parametric learning-curve extrapolation baseline on final-value and horizon-5 nowcasting. The axis is clipped at 0 for readability.
Figure 18: LCNet learning-curve baseline performance. Validation RMSE and for the LCNet-style baseline on final-value and horizon-5 nowcasting.
Figure 19: Averaged-latent ridge performance by dataset. Validation RMSE and for final-value and horizon-5 nowcasting from the mean-pooled SANE embedding.
Figure 20: Weight-statistic XGBoost performance by dataset.
Figure 21: Averaged-latent XGBoost performance by dataset. The ViT zoo is a notable failure case: even with performance features appended, the mean-pooled checkpoint vector provides little reliable information about final accuracy beyond the learning-curve position.
Figure 22: Weight-statistic MLP performance by dataset. Competitive for final-value nowcasting but selected less consistently than XGBoost.
Figure 23: Averaged-latent MLP performance by dataset.
Figure 24: Latent-window LSTM performance by dataset. Strongest on final-value prediction for the trajectory-sensitive Tiny ImageNet phase-transition and CIFAR-10 ResNet-18 width zoos.
Figure 25: Latent-token Transformer performance by dataset.
Figure 26: Predictor improvement over the LCNet baseline. Validation RMSE reduction relative to LCNet for the best weight-statistic and best SANE-representation predictor per dataset and target mode. Bars at zero indicate no improvement; labels report the selected predictor family.
Figure 27: Performance-feature ablation — RMSE view. Median validation RMSE (lower is better) for the performance-only, representation-only, and hybrid ablation configurations shown as in Figure 5 . The performance-only ablation for the latent-token Transformer was not run, since that architecture requires a latent-token sequence as input.
| Configuration | Predictor used for ranking |
|---|---|
| Naive current-value baseline | Current observed test accuracy. |
| LC Extrapolation baseline | Parametric learning-curve fit applied to the observed accuracy history. |
| LCNet baseline | LCNet-style predictor applied to the observed accuracy history. Final value: the predictor from the metric-prediction experiments, trained on the complete training-split learning curves of the zoo and held fixed. Horizon 5: refit at every pruning round on the history observed so far. |
| Weight-statistic XGBoost variants | Predictors loaded from the metric-prediction experiments, using weight-statistic features with and without observed metrics. |
| SANE latent variants | Predictors loaded from the metric-prediction experiments, using averaged latent, latent-window LSTM, or latent-token Transformer predictors for selected zoos. |
Table 7: SOTF training ranking configurations. Complete list of ranking policies compared in SOTF training experiments.
| Model zoo | Horizon-5 start epochs | Final-value start epochs |
|---|---|---|
| cifar10_cnn_large_uniform | 5, 10, 25 | 5, 10, 25 |
| cifar10_cnn_large_hyp_random_seeds | 5, 10, 25 | 5, 10, 25 |
| cifar10_cnn_large_hyp_fixed_seeds | 5, 10, 25 | 5, 10, 25 |
| stl10_cnn_large_uniform | 5, 10, 25 | 5, 10, 25 |
| stl10_cnn_large_hyp_random_seeds | 5, 10, 25 | 5, 10, 25 |
| stl10_cnn_large_hyp_fixed_seeds | 5, 10, 25 | 5, 10, 25 |
Table 8: SOTF training model-zoo coverage and start epochs.
| Final value | Horizon 5 | |||
| Ranking policy | CNN | ResNet-18 / ViT | CNN | ResNet-18 / ViT |
| Current accuracy | 0.0051 / 0.48 | 0.0643 / 0.11 | 0.0051 / 0.48 | 0.0629 / 0.11 |
| LC extrapolation | 0.0062 / 0.47 | 0.0689 / 0.05 | 0.0050 / 0.51 | 0.0490 / 0.05 |
| LCNet | 0.0056 / 0.48 | 0.0247 / 0.10 | 0.0104 / 0.43 | 0.0656 / 0.11 |
| XGB (perf.) | 0.0040 / 0.45 | 0.0051 / 0.20 | 0.0039 / 0.46 | 0.0495 / 0.09 |
| MLP (perf.) | 0.0038 / 0.48 | 0.0082 / 0.22 | 0.0047 / 0.46 | 0.0494 / 0.14 |
Table 9: SOTF training results for all ranking policies by population regime. Each cell reports the mean best-retained-model gap / mean oracle top- recall, separately for the six CNN zoos and the three ResNet-18 and ViT zoos. Final value ranks by predicted test accuracy at the end of training; horizon 5 ranks by predicted test accuracy five epochs ahead. Lower gap and higher recall are better; the best value per column is in bold. “perf.” denotes accuracy and loss history features, “weights” engineered weight statistics, and “SANE” SANE representations.
| Ranking policy | Tiny ImageNet ResNet-18 | CIFAR-10 ResNet-18 width | CIFAR-10 ViT |
|---|---|---|---|
| Current accuracy | 0.0387 / 0.16 | 0.0249 / 0.19 | 0.1267 / 0.00 |
| LC extrapolation | 0.0461 / 0.13 | 0.0223 / 0.03 | 0.1351 / 0.00 |
| LCNet | 0.0050 / 0.17 | 0.0031 / 0.04 | 0.0645 / 0.07 |
| XGB (perf.) | 0.0063 / 0.25 | 0.0037 / 0.21 | 0.0053 / 0.13 |
| XGB (weights) | 0.0051 / 0.26 | 0.0000 / 0.99 | 0.0001 / 0.80 |
| XGB (perf. + weights) | 0.0051 / 0.26 | 0.0000 / 0.99 | 0.0000 / 0.93 |
Table 10: Final-value SOTF training results per ResNet-18 and ViT zoo. Each cell reports the mean best-retained-model gap / mean oracle top- recall over start epochs and retention rates. Lower gap and higher recall are better; the best value per column is in bold.
Figure 28: TinyImageNet phase, final value: weight-statistic XGBoost with performance features; mostly robust improvements.
Figure 29: CIFAR-10 ViT, final value: latent-token Transformer; mixed results across pruning schedules.
Figure 30: TinyImageNet phase, final value: averaged-latent MLP; consistently better under meaningful pruning.
Figure 31: TinyImageNet phase, final value: averaged-latent XGBoost with performance features; robust improvements.
Figure 32: TinyImageNet phase, final value: latent-token Transformer with performance features; broadly effective.
Figure 33: TinyImageNet phase, final value: latent-window LSTM with performance features; mostly robust improvements.
Figure 34: CIFAR-10 ViT, final value: averaged-latent MLP; unstable and frequently worse than LC Extrapolation.
Figure 35: CIFAR-10 ViT, final value: averaged-latent XGBoost; strong early gains but poor transfer to later starts.
Figure 36: CIFAR-10 ViT, final value: latent-window LSTM; effective early but unreliable elsewhere.
Figure 37: CIFAR-10 ViT, final value: performance-only MLP; large early gains coexist with later degradations.
Figure 38: CIFAR-10 ViT, horizon value: averaged-latent MLP; predominantly worse than LC Extrapolation.
Figure 39: CIFAR-10 ViT, horizon value: averaged-latent XGBoost; mostly unsuccessful except at isolated settings.
Figure 40: CIFAR-10 ViT, horizon value: latent-token Transformer; consistently poor at moderate pruning rates.
Figure 41: CIFAR-10 ViT, horizon value: latent-window LSTM; generally worse despite high compute savings.
Figure 42: CIFAR-10 width-varying ResNet-18, final value: latent-window LSTM; helps mainly when pruning starts at epoch 40 or 60.