Authors: Melinda Flóra Kiss, Márton Muntag, Adrián Csiszárik, Dániel Varga
Organizations: HUN-REN Alfréd Rényi Institute of Mathematics, Budapest, Hungary · Department of Computer Science, Eötvös Loránd University, Budapest, Hungary
We identify an intriguing phenomenon when linearly interpolating between permutation-aligned neural networks: test accuracy briefly improves near the interpolation endpoints before dropping toward the midpoint, which we term BUMP (Brief Unexpected Mid-interpolation Peak). We observe that BUMP appears consistently across datasets, architectures, training regimes, and remains robust under substantial perturbations such as early checkpoints and extreme pruning. Moreover, controlled ablations suggest that it cannot be attributed to trivial regularization effects. While similar near-endpoint gains are visible in prior interpolation plots, they have not been explicitly documented or systematically and thoroughly analyzed. Leveraging this effect, we propose a simple, novel ensembling scheme: weight-prediction mixed ensembling that combines weight mixing with prediction-space mixing. Across our experiments, the resulting ensembling method consistently outperforms standard logit ensembling in test accuracy and yields improved calibration trade-off, highlighting complementary benefits of weight- and prediction-space mixing.
Figures & tables
Figure 1: The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-10 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Figure 2: The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-10 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Accuracy gain % — Aligned pairs
Width
Fixed- λ
Test
4x-width
0.053(±0.096)
0.087(±0.081)
8x-width
0.118(±0.111)
0.164(±0.117)
16x-width
0.181(±0.137)
0.227(±0.140)
Table 1: Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.
Accuracy gain % — Naïve pairs
Width
Fixed- λ
Test
4x-width
−0.041(±0.111)
0.038(±0.061)
8x-width
−0.041(±0.123)
0.051(±0.065)
16x-width
−0.073(±0.147)
0.056(±0.069)
Table 2: Average accuracy gains of the BUMPs over their corresponding endpoint models on non-aligned , i.e., naïve model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.
Figure 3: BUMP gain (solid line, left axis) and average model accuracy (dashed line, right axis) as a function of training epochs for 16 x-width ResNet-20 models on CIFAR-10. Values are averages for 50 model pairs, bands represent standard deviations.
Accuracy gain % — Random direction
Width
Fixed- λ
Test
4x-width
−0.009(±0.065)
0.028(±0.035)
8x-width
−0.012(±0.056)
0.035(±0.040)
16x-width
−0.002(±0.053)
0.031(±0.035)
Table 3: Average accuracy differences when moving towards random directions on CIFAR-10 with ResNet-20 ( 50 model pairs), with standard deviations denoted in parentheses.
Figure 4: Interpolation in the direction of a randomly chosen model on CIFAR-10, ResNet-20. Width multiplier is 16, the distance multiplier determines the distance of model A from the random model R compared to the distance of model A and B .
Accuracy gain % — Randomly reflected
Width
Fixed- λ
Test
4x-width
−0.081(±0.117)
0.025(±0.042)
8x-width
−0.094(±0.112)
0.022(±0.043)
16x-width
−0.077(±0.110)
0.028(±0.046)
Table 4: Average accuracy differences when moving towards models with randomly reflected weights on CIFAR-10 with ResNet-20 ( 50 model pairs). Standard deviations are denoted in parentheses.
Figure 5: Relationship between logit values of A and M1 models on CIFAR-10, ResNet-20, width multiplier is 16 . The red line indicates the best-fit affine relationship, highlighting the linear correspondence between the logits of the original and BUMP models.
Figure 6: Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-10, ResNet-20, width multiplier is 16 . Right (comparing class probabilities): Kullback-Leibler divergence between the class probability distributions of the WeightMix/LogitMix models and the λ=0.5 LogitMix (Ensemble) model, CIFAR-10, ResNet-20, width multiplier is 16 .
Figure 7: The mean angles across the test datapoints for g=24 , CIFAR-10, ResNet-20, width multiplier is 16. Values are averages for 50 model pairs.
Figure 8: Accuracy over (λ,β) for WPME obtained by interpolating model parameters in weight-space ( λ ) followed by logit-space mixing ( β ) with endpoint model B. Results are shown for a partially trained model (left) and the fully trained model (right), CIFAR-10, ResNet-20, width multiplier is 16 .
Model checkpoints: @60 epochs
Method
Acc. ↑
NLL ↓
ECE ↓
A (orig)
0.89972 (±0.00871)
0.30911 (±0.03136)
0.02789 (±0.00650)
B (orig)
0.90011 (±0.01077)
0.31030 (±0.04081)
0.02893 (±0.00905)
LogitMix
0.92369 (±0.00591)
0.22944 (±0.01954)
0.01374 (±0.00385)
SoftMaxMix
0.92160 (±0.00607)
0.24270 (±0.01824)
0.01990 (±0.00482)
WeightMix
0.91268 (±0.00593)
0.25942 (±0.01875)
0.01387 (±0.00514)
Table 5: Comparison of Weight–Prediction Mixed Ensembling with baseline methods on CIFAR-10, ResNet-20, width-multiplier 16: accuracy (Acc.), negative log-likelihood (NLL), and expected calibration error (ECE) values for different ensembling methods of two models at two different stages of model training (epoch 60 and 250). Reported values are sample means ± standard deviations (SD). Best results within each checkpoint are shown in bold.
Average number of datapoints
Width
A #1
A #2
Other
4x-width
9909.72
85.88
4.40
8x-width
9893.92
99.12
6.96
16x-width
9880.44
110.80
8.76
Table 6: Comparing model M1 with model A on CIFAR-10, ResNet-20 with method Fixed- λ . Numbers are averages calculated for 50 model pairs.
Average number of datapoints with changed truthiness
correct → incorrect
incorrect → correct
#1
#2
Other
#1
#2
Other
B
10.30
18.94
11.02
40.72
13.78
5.92
C
10.80
18.18
11.28
39.26
15.04
6.12
Table 7: Comparing the predictions of model M1 with predictions from model B and random chosen model C on CIFAR-10, 16x-width ResNet-20 (50 model pairs) with method Fixed- λ . In each column, we present the average number of datapoints.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Tiny-10
Layers
Name
3 × 3 conv. 16-LN-ReLu
Layer 1
3 × 3 conv. 16-LN-ReLu
Layer 2
3 × 3 conv. 32-LN-ReLu
Layer 3
3 × 3 conv. 32-LN-ReLu
Layer 4
3 × 3 conv. 32-LN-ReLu
Layer 5
Appendix
Table 8: The Tiny-10 architecture we used.
Accuracy gain % — Aligned pairs at epoch 60
Width
Fixed- λ
Test
4x-width
0.346(±0.223)
0.663(±0.528)
8x-width
0.577(±0.252)
1.009(±0.549)
16x-width
0.731(±0.309)
1.188(±0.590)
Appendix
Table 9: Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 60 . Standard deviations are denoted in parentheses.
Accuracy gain % — Aligned pairs at epoch 120
Width
Fixed- λ
Test
4x-width
0.291(±0.178)
0.506(±0.365)
8x-width
0.448(±0.191)
0.732(±0.380)
16x-width
0.614(±0.261)
0.970(±0.487)
Appendix
Table 10: Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 120 . Standard deviations are denoted in parentheses.
Accuracy gain % — Aligned pairs at epoch 180
Width
Fixed- λ
Test
4x-width
0.223(±0.148)
0.354(±0.250)
8x-width
0.358(±0.173)
0.526(±0.300)
16x-width
0.513(±0.187)
0.755(±0.348)
Appendix
Table 11: Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs . Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 180 . Standard deviations are denoted in parentheses.
Average Accuracy
Width
At epoch 60
At epoch 120
At epoch 180
At epoch 250
A
B
A
B
A
B
A
B
4x-width
0.8729
0.8736
0.8966
0.8945
0.9126
0.9138
0.9351
0.9356
8x-width
0.8869
0.8858
0.9081
0.9069
0.9206
0.9196
0.9418
0.9412
16x-width
0.8980
0.8935
0.9135
0.9131
0.9251
0.9249
0.9450
0.9456
Appendix
Table 12: Average model accuracy for CIFAR-10, ResNet-20 at different training epochs, comparing Model A and Model B for various model widths (100 models).
Table 21
Average Accuracies
Width
Model A
Model B
Pruned Model A
Pruned Model B
4x-width
0.9351
0.9356
0.6195
0.6235
8x-width
0.9418
0.9412
0.4613
0.4652
16x-width
0.9450
0.9456
0.7433
0.7493
Appendix
Table 15: Average model accuracies for CIFAR-10, ResNet-20 across widths, comparing original and pruned variants of Model A and Model B (100 models).
Homoskedastic
Heteroskedastic
Width
mean
std
p
mean
std
p
4x-width
-0.0001
0.0006
0.146
-0.0005
0.0007
1.27e-05
8x-width
-0.0001
0.0007
0.22
-0.0007
0.0009
7.4e-07
16x-width
-0.0002
0.0006
0.0186
-0.0009
0.0010
2.27e-07
Appendix
Table 16: Accuracy gains (mean, std, p -value) for homoskedastic and heteroskedastic noise models across widths.
Figure 9: Expected calibration error (ECE; 15 bins) over (λ,β) for probability-space hybrid ensembling, where weight-interpolated models are combined with endpoint predictions after softmax. Lower values indicate improved calibration.
Figure 10: The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-100 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Figure 11: The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-100 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Table 27
Figure 12: The mean angles across the test datapoints for g=24 , CIFAR-100, ResNet-20 ( 50 models).
Figure 13: Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-100, ResNet-20, width multiplier is 16 . Right (comparing class probabilities): Kullback-Leibler divergence between the class probability distributions of the WeightMix/LogitMix models and the λ=0.5 LogitMix (classicle ensemble) model, CIFAR-100, ResNet-20.
Figure 14: The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-10 test set, models are Tiny-10, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Figure 15: The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-10 test set, models are Tiny-10, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
Table 32
Figure 16: The mean angles across the test datapoints for g=24 , measured on CIFAR-10, model is Tiny-10 ( 50 models).
Figure 17: Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-10 with model Tiny-10, width multiplier is 16 . Right (comparing class probabilities): Kullback-Leibler divergence between the class probability distributions of the WeightMix/LogitMix models and the λ=0.5 LogitMix (classical logit ensemble) model, dataset is CIFAR-10, model is Tiny-10, width multiplier is 16 .
Deep neural networks that are independently trained to similar performance can be connected by low-loss parametric curves in weight space, a phenomenon known as Mode Connectivity (MC). This geometric property underpins practical techniques such as weight averaging, model ensembling, and model merging. We argue that low-loss connectivity is an incomplete geometric criterion: it controls loss only along a one-dimensional trajectory while leaving the surrounding weight-space neighborhood unconstrained, so the optimized curve may traverse sharp ridges that become fragile under distribution shift. We therefore reformulate mode connectivity as a neighborhood-robust path optimization problem, seeking a curve whose entire local neighborhood maintains low loss. We propose Sharp Mode Connectivity (SMC), which applies a first-order sharpness-aware approximation to the resulting minimax functional, enforcing flatness along the entire curve rather than only on it. We derive a practical optimization algorithm for connectivity paths under this sharpness-aware objective. Under severe blur corruptions from CIFAR-10-C, SMC achieves up to 6.09% absolute accuracy improvement over standard MC. Remarkably, SMC produces negative loss barriers, meaning that models obtained at interior points of the optimized path can outperform the average endpoint loss. These results, validated across ResNet-18, VGG16-BN, and ViT-Tiny on CIFAR-10 and ImageNet-100, establish path-wise flatness as a practical principle for robust weight-space interpolation.
In resource-constrained and low-latency settings, uncertainty estimates must be efficiently obtained. Deep Ensembles provide robust epistemic uncertainty (EU) but require training multiple full-size models. BatchEnsemble aims to deliver ensemble-like EU at far lower parameter and memory cost by applying learned rank-1 perturbations to a shared base network. We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline in terms of accuracy, calibration and out-of-distribution (OOD) detection on CIFAR10/10C/SVHN. A controlled study on MNIST finds members are near-identical in function and parameter space, indicating limited capacity to realize distinct predictive modes. Thus, BatchEnsemble behaves more like a single model than a true ensemble.
We introduce Leap+Verify, a framework that applies speculative execution -- predicting future model weights and validating predictions before acceptance -- to accelerate neural network training. Inspired by speculative decoding in language model inference and by the Automatically Scalable Computation (ASC) architecture for program execution, Leap+Verify decomposes training into three dynamically detected regimes (chaotic, transition, stable) using activation-space cosine similarity as a real-time Lyapunov proxy signal. Within each regime, analytic weight predictors (momentum, linear, quadratic extrapolation) attempt to forecast model parameters K training steps ahead; predictions are accepted only when validated against a held-out loss criterion. We evaluate Leap+Verify on GPT-2 124M and Qwen 2.5-1.5B trained on WikiText-103 across five random seeds, sweeping prediction depth K in {5, 10, 25, 50, 75, 100}. Momentum-based prediction (Adam moment extrapolation) fails catastrophically at both scales, with predicted losses exceeding actuals by 100-10,000x -- a universal norm explosion in optimizer-state extrapolation. Finite-difference predictors (linear, quadratic) succeed where momentum fails: at 124M, they achieve 24% strict acceptance at K=5 in stable regimes; at 1.5B, they achieve 37% strict acceptance in transition regimes. The scale-dependent finding is in regime distribution: GPT-2 124M spends 34% of training in stable regime, while Qwen 1.5B spends 64% in chaotic regime and reaches stable in only 0-2 of 40 checkpoints. Larger models are more predictable when predictable, but less often predictable -- the practical bottleneck shifts from predictor accuracy to regime availability. Cross-seed results are highly consistent (less than 1% validation loss variance), and the three-regime framework produces identical phase boundaries (plus or minus 50 steps) across seeds.