cs.LGNov 5, 2026
SaveBUMP: Brief Unexpected Mid-Interpolation Peak
Organizations: HUN-REN Alfréd Rényi Institute of Mathematics, Budapest, Hungary · Department of Computer Science, Eötvös Loránd University, Budapest, Hungary
Abstract
We identify an intriguing phenomenon when linearly interpolating between permutation-aligned neural networks: test accuracy briefly improves near the interpolation endpoints before dropping toward the midpoint, which we term BUMP (Brief Unexpected Mid-interpolation Peak). We observe that BUMP appears consistently across datasets, architectures, training regimes, and remains robust under substantial perturbations such as early checkpoints and extreme pruning. Moreover, controlled ablations suggest that it cannot be attributed to trivial regularization effects. While similar near-endpoint gains are visible in prior interpolation plots, they have not been explicitly documented or systematically and thoroughly analyzed. Leveraging this effect, we propose a simple, novel ensembling scheme: weight-prediction mixed ensembling that combines weight mixing with prediction-space mixing. Across our experiments, the resulting ensembling method consistently outperforms standard logit ensembling in test accuracy and yields improved calibration trade-off, highlighting complementary benefits of weight- and prediction-space mixing.
Figures & tables
Figure 1: The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-10 test set, models are ResNet-20, showing model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Figure 2: The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-10 test set, models are ResNet-20, showing model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
| Accuracy gain % — Aligned pairs | ||
|---|---|---|
| Width | Fixed- | Test |
| 4x-width | ||
| 8x-width | ||
| 16x-width | ||
Table 1: Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.
| Accuracy gain % — Naïve pairs | ||
|---|---|---|
| Width | Fixed- | Test |
| 4x-width | ||
| 8x-width | ||
| 16x-width | ||
Table 2: Average accuracy gains of the BUMPs over their corresponding endpoint models on non-aligned , i.e., naïve model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.
Figure 3: BUMP gain (solid line, left axis) and average model accuracy (dashed line, right axis) as a function of training epochs for x-width ResNet-20 models on CIFAR-10. Values are averages for model pairs, bands represent standard deviations.
| Accuracy gain % — Random direction | ||
|---|---|---|
| Width | Fixed- | Test |
| 4x-width | ||
| 8x-width | ||
| 16x-width | ||
Table 3: Average accuracy differences when moving towards random directions on CIFAR-10 with ResNet-20 ( model pairs), with standard deviations denoted in parentheses.
Figure 4: Interpolation in the direction of a randomly chosen model on CIFAR-10, ResNet-20. Width multiplier is 16, the distance multiplier determines the distance of model from the random model compared to the distance of model and .
| Accuracy gain % — Randomly reflected | ||
|---|---|---|
| Width | Fixed- | Test |
| 4x-width | ||
| 8x-width | ||
| 16x-width | ||
Table 4: Average accuracy differences when moving towards models with randomly reflected weights on CIFAR-10 with ResNet-20 ( model pairs). Standard deviations are denoted in parentheses.
Figure 5: Relationship between logit values of and models on CIFAR-10, ResNet-20, width multiplier is . The red line indicates the best-fit affine relationship, highlighting the linear correspondence between the logits of the original and BUMP models.
Figure 6: Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-10, ResNet-20, width multiplier is . Right (comparing class probabilities): Kullback-Leibler divergence between the class probability distributions of the WeightMix/LogitMix models and the LogitMix (Ensemble) model, CIFAR-10, ResNet-20, width multiplier is .
Figure 7: The mean angles across the test datapoints for , CIFAR-10, ResNet-20, width multiplier is 16. Values are averages for model pairs.
Figure 8: Accuracy over for WPME obtained by interpolating model parameters in weight-space ( ) followed by logit-space mixing ( ) with endpoint model B. Results are shown for a partially trained model (left) and the fully trained model (right), CIFAR-10, ResNet-20, width multiplier is .
| Model checkpoints: @60 epochs | |||
|---|---|---|---|
| Method | Acc. | NLL | ECE |
| A (orig) | 0.89972 | 0.30911 | 0.02789 |
| B (orig) | 0.90011 | 0.31030 | 0.02893 |
| LogitMix | 0.92369 | 0.22944 | 0.01374 |
| SoftMaxMix | 0.92160 | 0.24270 | 0.01990 |
| WeightMix | 0.91268 | 0.25942 | 0.01387 |
Table 5: Comparison of Weight–Prediction Mixed Ensembling with baseline methods on CIFAR-10, ResNet-20, width-multiplier 16: accuracy (Acc.), negative log-likelihood (NLL), and expected calibration error (ECE) values for different ensembling methods of two models at two different stages of model training (epoch 60 and 250). Reported values are sample means ± standard deviations (SD). Best results within each checkpoint are shown in bold.
| Average number of datapoints | |||
|---|---|---|---|
| Width | A #1 | A #2 | Other |
| 4x-width | |||
| 8x-width | |||
| 16x-width | |||
Table 6: Comparing model with model on CIFAR-10, ResNet-20 with method Fixed- . Numbers are averages calculated for 50 model pairs.
| Average number of datapoints with changed truthiness | ||||||
|---|---|---|---|---|---|---|
| correct incorrect | incorrect correct | |||||
| #1 | #2 | Other | #1 | #2 | Other | |
| B | 10.30 | 18.94 | 11.02 | 40.72 | 13.78 | 5.92 |
| C | 10.80 | 18.18 | 11.28 | 39.26 | 15.04 | 6.12 |
Table 7: Comparing the predictions of model with predictions from model and random chosen model on CIFAR-10, 16x-width ResNet-20 (50 model pairs) with method Fixed- . In each column, we present the average number of datapoints.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Tiny-10 | |
|---|---|
| Layers | Name |
| 3 × 3 conv. 16-LN-ReLu | Layer 1 |
| 3 × 3 conv. 16-LN-ReLu | Layer 2 |
| 3 × 3 conv. 32-LN-ReLu | Layer 3 |
| 3 × 3 conv. 32-LN-ReLu | Layer 4 |
| 3 × 3 conv. 32-LN-ReLu | Layer 5 |
Table 8: The Tiny-10 architecture we used.
| Accuracy gain % — Aligned pairs at epoch 60 | ||
|---|---|---|
| Width | Fixed- | Test |
| 4x-width | ||
| 8x-width | ||
| 16x-width | ||
Table 9: Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 60 . Standard deviations are denoted in parentheses.
| Accuracy gain % — Aligned pairs at epoch 120 | ||
|---|---|---|
| Width | Fixed- | Test |
| 4x-width | ||
| 8x-width | ||
| 16x-width | ||
Table 10: Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 120 . Standard deviations are denoted in parentheses.
| Accuracy gain % — Aligned pairs at epoch 180 | ||
|---|---|---|
| Width | Fixed- | Test |
| 4x-width | ||
| 8x-width | ||
| 16x-width | ||
Table 11: Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 180 . Standard deviations are denoted in parentheses.
| Average Accuracy | ||||||||
| Width | At epoch 60 | At epoch 120 | At epoch 180 | At epoch 250 | ||||
| A | B | A | B | A | B | A | B | |
| 4x-width | 0.8729 | 0.8736 | 0.8966 | 0.8945 | 0.9126 | 0.9138 | 0.9351 | 0.9356 |
| 8x-width | 0.8869 | 0.8858 | 0.9081 | 0.9069 | 0.9206 | 0.9196 | 0.9418 | 0.9412 |
| 16x-width | 0.8980 | 0.8935 | 0.9135 | 0.9131 | 0.9251 | 0.9249 | 0.9450 | 0.9456 |
Table 12: Average model accuracy for CIFAR-10, ResNet-20 at different training epochs, comparing Model A and Model B for various model widths (100 models).
| Average Accuracies | ||||
|---|---|---|---|---|
| Width | Model A | Model B | Pruned Model A | Pruned Model B |
| 4x-width | 0.9351 | 0.9356 | 0.6195 | 0.6235 |
| 8x-width | 0.9418 | 0.9412 | 0.4613 | 0.4652 |
| 16x-width | 0.9450 | 0.9456 | 0.7433 | 0.7493 |
Table 15: Average model accuracies for CIFAR-10, ResNet-20 across widths, comparing original and pruned variants of Model A and Model B (100 models).
| Homoskedastic | Heteroskedastic | |||||
|---|---|---|---|---|---|---|
| Width | mean | std | mean | std | ||
| 4x-width | -0.0001 | 0.0006 | 0.146 | -0.0005 | 0.0007 | 1.27e-05 |
| 8x-width | -0.0001 | 0.0007 | 0.22 | -0.0007 | 0.0009 | 7.4e-07 |
| 16x-width | -0.0002 | 0.0006 | 0.0186 | -0.0009 | 0.0010 | 2.27e-07 |
Table 16: Accuracy gains (mean, std, -value) for homoskedastic and heteroskedastic noise models across widths.
Figure 9: Expected calibration error (ECE; 15 bins) over for probability-space hybrid ensembling, where weight-interpolated models are combined with endpoint predictions after softmax. Lower values indicate improved calibration.
Figure 10: The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-100 test set, models are ResNet-20, showing model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Figure 11: The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-100 test set, models are ResNet-20, showing model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Figure 12: The mean angles across the test datapoints for , CIFAR-100, ResNet-20 ( models).
Figure 13: Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-100, ResNet-20, width multiplier is . Right (comparing class probabilities): Kullback-Leibler divergence between the class probability distributions of the WeightMix/LogitMix models and the LogitMix (classicle ensemble) model, CIFAR-100, ResNet-20.
Figure 14: The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-10 test set, models are Tiny-10, showing model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Figure 15: The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-10 test set, models are Tiny-10, showing model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Figure 16: The mean angles across the test datapoints for , measured on CIFAR-10, model is Tiny-10 ( models).
Figure 17: Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-10 with model Tiny-10, width multiplier is . Right (comparing class probabilities): Kullback-Leibler divergence between the class probability distributions of the WeightMix/LogitMix models and the LogitMix (classical logit ensemble) model, dataset is CIFAR-10, model is Tiny-10, width multiplier is .