cs.LGOct 27, 2026

Measuring Deep Learning Performance - An Empirical Study of Performance Distributions Across Architectures and Tasks

Authors: Kevin L Coakley, Odd Erik Gundersen

Organizations: Department of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway. · San Diego Supercomputer Center, University of California San Diego, La Jolla, CA, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Context: Non-determinism in deep learning algorithm design and implementation leads to performance variation, meaning model performance is not a single value, but rather a distribution. These model performance distributions are underexplored despite their impact on robustness. Objective: We investigate the robustness of deep learning performance to sources of non-determinism, specifically focusing on how performance distributions differ across various architectures and tasks. Method: We conducted 186 experiments on image classification and time series forecasting architectures. Each experiment was run 100 times with different random seeds to generate performance distributions, resulting in 18,600 runs. Robustness was quantified using metrics for spread, symmetry, and tail risk. Results: Performance distributions are frequently non-Gaussian, particularly in time series forecasting. Model size does not systematically affect robustness -- larger image classification models show fewer outliers but not lower spread, while smaller time series models show lower spread but more extreme underperformers. Training duration does not scale linearly; early stopping effectively balances performance and robustness. Mean performance does not predict robustness - time series forecasting shows moderate correlation while image classification shows none. Time series models produce nearly three times more underperforming outliers than image classification models, indicating substantially higher tail risk. Conclusions: Tail risk poses serious concerns for Trustworthy AI in high-stakes applications. Models performing well on average may exhibit long tails and extreme outliers revealed only through distributional analysis. Mean performance alone should not guide model selection; assessment of spread, symmetry, and tail risk is essential for reliable model assessment where consistent performance is critical.

Figures & tables

Figure 1: Kernel density estimates of top-1 accuracy for image classification models on CIFAR-10 (left) and CIFAR-100 (right). Top row shows ResNet architectures (ResNet20, ResNet56, ResNet110); bottom row shows Vision Transformers (ViT-S-8, ViT-B-8). Each curve represents the distribution across 100 random seeds for a single model-dataset pair.
Figure 2: Kernel density estimates of MAE for time series forecasting models on ETTh1 across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.
Robustness# of Models More RobustEqually# ofp-value
MeasureLarger ModelsSmaller ModelsRobustTrialsLarger ModelsSmaller Models
Intra-architecture Binomial Comparison
Range12140260.72140.4225
Standard Deviation9170260.96220.0843
Skewness14120260.42250.7214
Lower Fence Outliers12104220.41590.7383
Table 1: Binomial test results for robustness comparisons between image classification models with different parameter counts. Rows show counts of pairwise comparisons favoring larger models, smaller models, or ties, with corresponding p-values. Hypothesized probability p=0.5p=0.5 for all tests.
Robustness# of Models More RobustEqually# ofp-value
MeasureLarger ModelsSmaller ModelsRobustTrialsLarger ModelsSmaller Models
Cross-architecture Binomial Comparison
Range2711101381.01.36×10−13\boldsymbol{1.36\times 10^{-13}}
Standard Deviation2711101381.01.36×10−13\boldsymbol{1.36\times 10^{-13}}
Skewness835501380.01060.9934
Upper Fence Outliers7847131250.00350.998
Table 2: Binomial test results for robustness comparisons between time series forecasting models with different parameter counts. Rows show counts of pairwise comparisons favoring larger models, smaller models, or ties, with corresponding p-values. Hypothesized probability p=0.5p=0.5 for all tests.
Figure 3: Mean and range of top-1 accuracy across training epochs for ResNet20 (blue) and ViT-S-8 (orange) on CIFAR-10. Solid lines show mean accuracy (left y-axis); dashed lines show range (right y-axis). Each point computed from 100 seed replicates.
Figure 4: Mean MAE (solid lines, left y-axis) and range (dashed lines, right y-axis) across training epochs for time series forecasting models on ETTh1 with forecast horizon 96. Top panel shows Autoformer (epochs 1-10); bottom panel shows iTransformer and NLinear (epochs 1-10, bottom x-axis) and TSMixer (epochs 10-100, top x-axis). Each point computed from 100 seed replicates trained for the specified number of epochs.
ResNet20ViT-S-8
EpochsMeanRangeEpochsMeanRange
1075.94%11.04%1056.23%3.29%
2083.22%4.81%2065.74%2.26%
3085.90%3.33%3070.64%1.75%
4087.28%2.75%4073.47%1.83%
5088.20%2.18%5075.32%1.49%
Table 3: Mean top-1 accuracy and range for ResNet20 and ViT-S-8 on CIFAR-10 across training epochs. Each row computed from 100 seed replicates. Pearson correlation coefficients show the relationship between epochs and range (robustness).
AutoformeriTransformerNLinearTSMixer
EpochsMeanRangeEpochsMeanRangeEpochsMeanRangeEpochsMeanRange
10.45930.087610.41120.005410.40670.0191100.40430.0083
20.46450.099920.40580.005320.40490.0349200.39300.0034
30.48160.147730.40510.005130.39830.0143300.39200.0036
40.49360.178740.40490.004240.39510.0095400.39190.0041
50.50270.238750.40490.004750.39380.0061500.39230.0046
Table 4: Mean MAE and range for Autoformer, iTransformer, NLinear, and TSMixer on ETTh1 across training epochs. Each row computed from 100 seed replicates. Pearson correlation coefficients show the relationship between epochs and range (robustness).
Figure 5: Relationship between mean performance and robustness (range) for image classification (top) and time series forecasting (bottom). Each point represents a model-dataset pair computed from 100 seed replicates. Performance measured as top-1 error for image classification and MAE for time series forecasting. Marker shapes indicate model architecture; colors indicate dataset.
TaskPearsonp-valueSpearmanp-value
r ρ\rho 
Image Classification0.20530.29460.37500.0493
Time Series Forecasting0.41583.08×10−5\boldsymbol{3.08\times 10^{-5}}0.42112.38×10−5\boldsymbol{2.38\times 10^{-5}}
Table 5: Pearson and Spearman rank correlation coefficients and p-values for the relationship between mean performance and range (robustness) across tasks. Performance measured as top-1 error (image classification) or MAE (time series forecasting), computed from 100 seed replicates per model-dataset pair. Statistically significant results ( p<0.05p<0.05 ) in bold.
 Image ClassificationTime Series Forecastingp-values
Outlier TypeOutliersDatasetMeanOutliersDatasetMeanImageTime Series
  ReplicatesOutliers ReplicatesOutliersClassificationForecasting
Underperforming22221.0266942.831.56×10−5\boldsymbol{1.56\times 10^{-5}}1.0
Overperforming19220.8634940.360.980.02
Total41221.86300943.193.26×10−3\boldsymbol{3.26\times 10^{-3}}1.0
Table 6: Comparison of outlier counts between image classification and time series forecasting models. Each row shows total outliers, number of dataset replicates, mean outliers per replicate, and p-values for underperforming, overperforming, and total outliers. Welch’s t-test used for unequal variances; Student’s t-test used for equal variances. Statistically significant results ( p<0.05p<0.05 ) in bold.
Figure 6: Annual publication counts (1990-2024) for deep learning and neural network papers in image classification/segmentation (blue) versus time series forecasting/prediction (orange) from Scopus. Results show substantially higher research volume in image classification.
Figure 7: Publication counts for deep learning papers on regularization methods in image classification/segmentation (blue) versus time series forecasting/prediction (orange) from Scopus. All regularization techniques show higher research volume for image classification, with data augmentation being the most studied method.
ModelL1 RegL2 Reg/WDDropoutEarly StopBatch NormData AugAdv TrainLabel SmoothMTLNoise Inj
ResNet-SGD--✓✓----
ViT-SGD✓--✓----
AutoFormer--✓✓------
iTransformer--✓✓------
Nlinear---✓------
TSMixer--✓✓✓-----
Table 7: Regularization techniques used in image classification and time series forecasting models. Column abbreviations: L1 Reg (L1 regularization), L2 Reg/WD (L2 regularization/Weight Decay), Early Stop (Early Stopping), Batch Norm (Batch Normalization), Data Aug (Data Augmentation), Adv Train (Adversarial Training), Label Smooth (Label Smoothing), MTL (Multi-task Learning), Noise Inj (Noise Injection). Checkmarks indicate technique is applied. SGD: models trained with the Stochastic Gradient Descent optimizer use weight decay, models trained with the Adam optimizer do not.
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

ModelVariationParameters
ResNet200.27M
560.85M
1101.73M
1811.18M
5023.52M
10142.52M
Appendix
Table 8: The number of parameters for each of the Image Classification models.
ModelDatasetLearning RateBatch SizeOptimizer
ResNetCIFAR-100.1128SGD
CIFAR-1000.1128SGD
Oxford Flowers0.00164Adam
UC Merced0.00132Adam
ViT (Random Weights)CIFAR-100.0001128Adam
CIFAR-1000.0001128Adam
Appendix
Table 9: Hyperparameters selected for training based on a grid search.
DatasetHorizonAutoformeriTransformerNLinearTSMixer
ETTh19610.54M0.84M0.01M0.04M
19210.54M0.86M0.02M0.05M
33610.54M0.90M0.03M0.06M
72010.54M1.00M0.07M0.10M
ETTh29610.54M0.22M0.01M0.04M
19210.54M0.24M0.02M0.05M
Appendix
Table 10: The number of parameters for each of the Time Series models.
    SpreadSymmetryTail Risk
ModelDataset# ParaMeanRangeSDSkewnessShapiro WShapiro pLFOUFO
32x32x3 Pixel Images & Random Weight Initialization 
ResNet20CIFAR-100.27M91.19%1.18%0.22-0.41760.98020.138230
ResNet560.85M92.05%3.19%0.61-0.93790.94750.000650
ResNet1101.73M93.56%1.03%0.170.37840.97830.098102
ViT-S-821.67M78.16%1.31%0.27-0.40260.97470.050700
Appendix
Table 11: Performance and robustness metrics for image classification models. Each row shows results for 100 seed replicates. Columns: # Para (parameter count), Mean (mean top-1 accuracy), Range, SD (standard deviation), Skewness, Shapiro W (Shapiro-Wilk W statistic), Shapiro p (Shapiro-Wilk p-value), LFO (lower fence outliers), UFO (upper fence outliers). Models grouped by image resolution and initialization strategy.
Robustness MeasureWp-valuerMore RobustEffect Size
Intra-architecture Wilcoxon Comparison
Range155.50.6204−- 0.114—Small
Standard Deviation135.00.3153−- 0.231—Small
Skewness163.00.7644−- 0.071—Negligible
Lower Fence Outliers107.00.5240−- 0.154—Small
Upper Fence Outliers63.00.5301−- 0.177—Small
Appendix
Table 12: Wilcoxon signed-rank test results for robustness comparisons between image classification models with different parameter counts. W Statistic and p-value report the Wilcoxon signed-rank test statistic and corresponding p-value for paired comparisons on raw metric values. r denotes the rank-biserial correlation coefficient. More Robust indicates which model size exhibited lower metric values on the robustness measure; — indicates no statistically significant difference. Effect Size reports the rank-biserial correlation magnitude using the thresholds: negligible ( ∣r∣<0.1|r|<0.1 ), small ( 0.1≤∣r∣<0.30.1\leq|r|<0.3 ), medium ( 0.3≤∣r∣<0.50.3\leq|r|<0.5 ), and large ( ∣r∣≥0.5|r|\geq 0.5 ). Statistically significant results ( p<0.05p<0.05 ) in bold.
     SpreadSymmetryTail Risk
ModelDatasetHorizon# ParaMeanRangeSDSkewnessShapiro WShapiro pLFOUFO
AutoformerETTh19610.54M0.45760.06440.01410.74420.94940.000805
iTransformer0.84M0.40490.0040.00090.25210.97680.075300
NLinear0.01M0.39520.02690.0043.4470.64924.09×10−44.09\times 10^{-4}05
TSMixer0.04M0.39210.00330.00080.19370.97750.08500
AutoformerETTh119210.54M0.47780.0580.01160.16480.98840.536210
Appendix
Table 13: Performance and robustness metrics for time series forecasting models on ETTh1 and ETTh2. Each row shows results for 100 seed replicates. Columns: # Para (parameter count), Mean (mean MAE), Range, SD (standard deviation), Skewness, Shapiro W (Shapiro-Wilk W statistic), Shapiro p (Shapiro-Wilk p-value), LFO (lower fence outliers), UFO (upper fence outliers). Models grouped by dataset and forecast horizon.
     SpreadSymmetryTail Risk
ModelDatasetHorizon# ParaMeanRangeSDSkewnessShapiro WShapiro pLFOUFO
AutoformerETTm19610.54M0.46690.08020.0171.16130.9031.95×10−61.95\times 10^{-6}04
iTransformer0.22M0.37660.00610.00130.02080.98880.571800
NLinear0.01M0.37590.00380.00060.96460.88162.13×10−72.13\times 10^{-7}58
TSMixer0.04M0.3630.00630.00140.27460.88160.402200
AutoformerETTm119210.54M0.49330.05370.01270.39450.95850.003200
Appendix
Table 14: Performance and robustness metrics for time series forecasting models on ETTm1 and ETTm2. Each row shows results for 100 seed replicates. Columns: # Para (parameter count), Mean (mean MAE), Range, SD (standard deviation), Skewness, Shapiro W (Shapiro-Wilk W statistic), Shapiro p (Shapiro-Wilk p-value), LFO (lower fence outliers), UFO (upper fence outliers). Models grouped by dataset and forecast horizon.
     SpreadSymmetryTail Risk
ModelDatasetHorizon# ParaMeanRangeSDSkewnessShapiro WShapiro pLFOUFO
AutoformerTraffic9614.91M0.3930.07240.01471.29060.89911.29×10−61.29\times 10^{-6}06
iTransformer6.41M0.26860.00250.0005-0.04720.99150.780900
NLinear0.01M0.38830.00560.0010.93920.94710.000502
TSMixer6.27M0.31350.01380.00250.82020.96010.004102
AutoformerTraffic19214.91M0.40730.10070.02320.56180.95420.001600
Appendix
Table 15: Performance and robustness metrics for time series forecasting models on Traffic and Weather. Each row shows results for 100 seed replicates. Columns: # Para (parameter count), Mean (mean MAE), Range, SD (standard deviation), Skewness, Shapiro W (Shapiro-Wilk W statistic), Shapiro p (Shapiro-Wilk p-value), LFO (lower fence outliers), UFO (upper fence outliers). Models grouped by dataset and forecast horizon. Traffic horizons 336 and 720 were not completed for iTransformer due to GPU memory constraints on the NVIDIA A100s.
Robustness MeasureWp-valuerMore RobustEffect Size
Cross-architecture Wilcoxon Comparison
Range1163.01.17×10−14\boldsymbol{1.17\times 10^{-14}}−- 0.758SmallerLarge
Standard Deviation934.02.29×10−16\boldsymbol{2.29\times 10^{-16}}−- 0.805SmallerLarge
Skewness3203.00.0007−- 0.332LargerMedium
Upper Fence Outliers2505.50.0004−- 0.364LargerMedium
Appendix
Table 16: Wilcoxon signed-rank test results for robustness comparisons between time series forecasting models with different parameter counts. W Statistic and p-value report the Wilcoxon signed-rank test statistic and corresponding p-value for paired comparisons on raw metric values. r denotes the rank-biserial correlation coefficient. More Robust indicates which model size exhibited lower metric values on the robustness measure; — indicates no statistically significant difference. Effect Size reports the rank-biserial correlation magnitude using the thresholds: negligible ( ∣r∣<0.1|r|<0.1 ), small ( 0.1≤∣r∣<0.30.1\leq|r|<0.3 ), medium ( 0.3≤∣r∣<0.50.3\leq|r|<0.5 ), and large ( ∣r∣≥0.5|r|\geq 0.5 ). Statistically significant results ( p<0.05p<0.05 ) in bold.
Figure 8: Kernel density estimates of top-1 accuracy for image classification models on Oxford Flowers (left) and UC Merced (right). Top row shows ResNet architectures (ResNet18, ResNet50, ResNet101); middle row shows Vision Transformers with random initialization (ViT-Tiny-16, ViT-S-16, ViT-B-16); bottom row shows Vision Transformers with pretrained weights (ViT-Tiny-16, ViT-S-16, ViT-B-16). Each curve represents the distribution across 100 random seeds for a single model-dataset pair.
Figure 9: Kernel density estimates of MAE for time series forecasting models on ETTh2 across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.
Figure 10: Kernel density estimates of MAE for time series forecasting models on ETTm1 across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.
Figure 11: Kernel density estimates of MAE for time series forecasting models on ETTm2 across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.
Figure 12: Kernel density estimates of MAE for time series forecasting models on Traffic across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.
Figure 13: Kernel density estimates of MAE for time series forecasting models on Weather across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.
Figure 14: Mean gradient L2 norm (left) and within-epoch gradient norm variance (right) across 200 training epochs for ResNet110 and ViT-B-8 on CIFAR-10, computed over 100 seeds. Shaded bands indicate the interquartile range across seeds. Dashed vertical lines mark learning rate decay events at epochs 100 and 150.
Figure 15: Perturbation-based sharpness at convergence for ResNet110 and ViT-B-8 on CIFAR-10, computed over 100 seeds. Each box shows the median, interquartile range, and 1.5 x IQR whiskers; points beyond the whiskers are plotted individually.
More in this topic

ML Reproducibility

Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.

See all 138 papers

Explore similar work

CardsList

  1. Estimating Performance for Safe Deployment of Machine Learning Models

    Oct 21, 2026Zeju Li, Konstantinos Kamnitsas, Mobarakol Islam +4Distribution ShiftModel Calibration

    College of Biomedical Engineering, Fudan University, Shanghai, China · Oxford Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK · Department of Engineering Scienc, University of Oxford, Oxford, UK +4

  2. A New Framework to Analyse the Distributional Robustness of Deep Neural Networks

    May 20, 2026Divij Khaitan, Subhashis BanerjeeDistribution Shift RobustnessNeural Network Generalization

    Microsoft · Ashoka University

  3. Performance Variation in Deep Reinforcement Learning

    Jun 4, 2026Haruto Tanaka, A. Rupam MahmoodReinforcement Learning

    Department of Computing Science University of Alberta Alberta Machine Intelligence Institute (Amii)