Measuring Deep Learning Performance - An Empirical Study of Performance Distributions Across Architectures and Tasks

Kevin L Coakley; Odd Erik Gundersen

cs.LGOct 27, 2026

Measuring Deep Learning Performance - An Empirical Study of Performance Distributions Across Architectures and Tasks

Authors: Kevin L Coakley, Odd Erik Gundersen

Organizations: Department of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway. · San Diego Supercomputer Center, University of California San Diego, La Jolla, CA, USA.

Abstract

Context: Non-determinism in deep learning algorithm design and implementation leads to performance variation, meaning model performance is not a single value, but rather a distribution. These model performance distributions are underexplored despite their impact on robustness. Objective: We investigate the robustness of deep learning performance to sources of non-determinism, specifically focusing on how performance distributions differ across various architectures and tasks. Method: We conducted 186 experiments on image classification and time series forecasting architectures. Each experiment was run 100 times with different random seeds to generate performance distributions, resulting in 18,600 runs. Robustness was quantified using metrics for spread, symmetry, and tail risk. Results: Performance distributions are frequently non-Gaussian, particularly in time series forecasting. Model size does not systematically affect robustness -- larger image classification models show fewer outliers but not lower spread, while smaller time series models show lower spread but more extreme underperformers. Training duration does not scale linearly; early stopping effectively balances performance and robustness. Mean performance does not predict robustness - time series forecasting shows moderate correlation while image classification shows none. Time series models produce nearly three times more underperforming outliers than image classification models, indicating substantially higher tail risk. Conclusions: Tail risk poses serious concerns for Trustworthy AI in high-stakes applications. Models performing well on average may exhibit long tails and extreme outliers revealed only through distributional analysis. Mean performance alone should not guide model selection; assessment of spread, symmetry, and tail risk is essential for reliable model assessment where consistent performance is critical.

Figures & tables

Figure 1: Kernel density estimates of top-1 accuracy for image classification models on CIFAR-10 (left) and CIFAR-100 (right). Top row shows ResNet architectures (ResNet20, ResNet56, ResNet110); bottom row shows Vision Transformers (ViT-S-8, ViT-B-8). Each curve represents the distribution across 100 random seeds for a single model-dataset pair.

Figure 2: Kernel density estimates of MAE for time series forecasting models on ETTh1 across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.

Robustness	# of Models More Robust		Equally	# of	p-value
Measure	Larger Models	Smaller Models	Robust	Trials	Larger Models	Smaller Models
Intra-architecture Binomial Comparison
Range	12	14	0	26	0.7214	0.4225
Standard Deviation	9	17	0	26	0.9622	0.0843
Skewness	14	12	0	26	0.4225	0.7214
Lower Fence Outliers	12	10	4	22	0.4159	0.7383

Table 1: Binomial test results for robustness comparisons between image classification models with different parameter counts. Rows show counts of pairwise comparisons favoring larger models, smaller models, or ties, with corresponding p-values. Hypothesized probability

p=0.5

for all tests.

Robustness	# of Models More Robust		Equally	# of	p-value
Measure	Larger Models	Smaller Models	Robust	Trials	Larger Models	Smaller Models
Cross-architecture Binomial Comparison
Range	27	111	0	138	1.0	$\boldsymbol{1.36\times 10^{-13}}$
Standard Deviation	27	111	0	138	1.0	$\boldsymbol{1.36\times 10^{-13}}$
Skewness	83	55	0	138	0.0106	0.9934
Upper Fence Outliers	78	47	13	125	0.0035	0.998

Table 2: Binomial test results for robustness comparisons between time series forecasting models with different parameter counts. Rows show counts of pairwise comparisons favoring larger models, smaller models, or ties, with corresponding p-values. Hypothesized probability

p=0.5

for all tests.

Figure 3: Mean and range of top-1 accuracy across training epochs for ResNet20 (blue) and ViT-S-8 (orange) on CIFAR-10. Solid lines show mean accuracy (left y-axis); dashed lines show range (right y-axis). Each point computed from 100 seed replicates.

Figure 4: Mean MAE (solid lines, left y-axis) and range (dashed lines, right y-axis) across training epochs for time series forecasting models on ETTh1 with forecast horizon 96. Top panel shows Autoformer (epochs 1-10); bottom panel shows iTransformer and NLinear (epochs 1-10, bottom x-axis) and TSMixer (epochs 10-100, top x-axis). Each point computed from 100 seed replicates trained for the specified number of epochs.

ResNet20			ViT-S-8
Epochs	Mean	Range	Epochs	Mean	Range
10	75.94%	11.04%	10	56.23%	3.29%
20	83.22%	4.81%	20	65.74%	2.26%
30	85.90%	3.33%	30	70.64%	1.75%
40	87.28%	2.75%	40	73.47%	1.83%
50	88.20%	2.18%	50	75.32%	1.49%

Table 3: Mean top-1 accuracy and range for ResNet20 and ViT-S-8 on CIFAR-10 across training epochs. Each row computed from 100 seed replicates. Pearson correlation coefficients show the relationship between epochs and range (robustness).

Autoformer			iTransformer			NLinear			TSMixer
Epochs	Mean	Range	Epochs	Mean	Range	Epochs	Mean	Range	Epochs	Mean	Range
1	0.4593	0.0876	1	0.4112	0.0054	1	0.4067	0.0191	10	0.4043	0.0083
2	0.4645	0.0999	2	0.4058	0.0053	2	0.4049	0.0349	20	0.3930	0.0034
3	0.4816	0.1477	3	0.4051	0.0051	3	0.3983	0.0143	30	0.3920	0.0036
4	0.4936	0.1787	4	0.4049	0.0042	4	0.3951	0.0095	40	0.3919	0.0041
5	0.5027	0.2387	5	0.4049	0.0047	5	0.3938	0.0061	50	0.3923	0.0046

Table 4: Mean MAE and range for Autoformer, iTransformer, NLinear, and TSMixer on ETTh1 across training epochs. Each row computed from 100 seed replicates. Pearson correlation coefficients show the relationship between epochs and range (robustness).

Figure 5: Relationship between mean performance and robustness (range) for image classification (top) and time series forecasting (bottom). Each point represents a model-dataset pair computed from 100 seed replicates. Performance measured as top-1 error for image classification and MAE for time series forecasting. Marker shapes indicate model architecture; colors indicate dataset.

Task	Pearson	p-value	Spearman	p-value
Task	r		$\rho$
Image Classification	0.2053	0.2946	0.3750	0.0493
Time Series Forecasting	0.4158	$\boldsymbol{3.08\times 10^{-5}}$	0.4211	$\boldsymbol{2.38\times 10^{-5}}$

Table 5: Pearson and Spearman rank correlation coefficients and p-values for the relationship between mean performance and range (robustness) across tasks. Performance measured as top-1 error (image classification) or MAE (time series forecasting), computed from 100 seed replicates per model-dataset pair. Statistically significant results (

p<0.05

) in bold.

	Image Classification			Time Series Forecasting			p-values
Outlier Type	Outliers	Dataset	Mean	Outliers	Dataset	Mean	Image	Time Series
		Replicates	Outliers		Replicates	Outliers	Classification	Forecasting
Underperforming	22	22	1.0	266	94	2.83	$\boldsymbol{1.56\times 10^{-5}}$	1.0
Overperforming	19	22	0.86	34	94	0.36	0.98	0.02
Total	41	22	1.86	300	94	3.19	$\boldsymbol{3.26\times 10^{-3}}$	1.0

Table 6: Comparison of outlier counts between image classification and time series forecasting models. Each row shows total outliers, number of dataset replicates, mean outliers per replicate, and p-values for underperforming, overperforming, and total outliers. Welch’s t-test used for unequal variances; Student’s t-test used for equal variances. Statistically significant results (

p<0.05

) in bold.

Figure 6: Annual publication counts (1990-2024) for deep learning and neural network papers in image classification/segmentation (blue) versus time series forecasting/prediction (orange) from Scopus. Results show substantially higher research volume in image classification.

Figure 7: Publication counts for deep learning papers on regularization methods in image classification/segmentation (blue) versus time series forecasting/prediction (orange) from Scopus. All regularization techniques show higher research volume for image classification, with data augmentation being the most studied method.

Model	L1 Reg	L2 Reg/WD	Dropout	Early Stop	Batch Norm	Data Aug	Adv Train	Label Smooth	MTL	Noise Inj
ResNet	-	SGD	-	-	✓	✓	-	-	-	-
ViT	-	SGD	✓	-	-	✓	-	-	-	-
AutoFormer	-	-	✓	✓	-	-	-	-	-	-
iTransformer	-	-	✓	✓	-	-	-	-	-	-
Nlinear	-	-	-	✓	-	-	-	-	-	-
TSMixer	-	-	✓	✓	✓	-	-	-	-	-

Table 7: Regularization techniques used in image classification and time series forecasting models. Column abbreviations: L1 Reg (L1 regularization), L2 Reg/WD (L2 regularization/Weight Decay), Early Stop (Early Stopping), Batch Norm (Batch Normalization), Data Aug (Data Augmentation), Adv Train (Adversarial Training), Label Smooth (Label Smoothing), MTL (Multi-task Learning), Noise Inj (Noise Injection). Checkmarks indicate technique is applied. SGD: models trained with the Stochastic Gradient Descent optimizer use weight decay, models trained with the Adam optimizer do not.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Variation	Parameters
ResNet	20	0.27M
	56	0.85M
	110	1.73M
	18	11.18M
	50	23.52M
	101	42.52M

Appendix

Table 8: The number of parameters for each of the Image Classification models.

Model	Dataset	Learning Rate	Batch Size	Optimizer
ResNet	CIFAR-10	0.1	128	SGD
	CIFAR-100	0.1	128	SGD
	Oxford Flowers	0.001	64	Adam
	UC Merced	0.001	32	Adam
ViT (Random Weights)	CIFAR-10	0.0001	128	Adam
	CIFAR-100	0.0001	128	Adam

Appendix

Table 9: Hyperparameters selected for training based on a grid search.

Dataset	Horizon	Autoformer	iTransformer	NLinear	TSMixer
ETTh1	96	10.54M	0.84M	0.01M	0.04M
	192	10.54M	0.86M	0.02M	0.05M
	336	10.54M	0.90M	0.03M	0.06M
	720	10.54M	1.00M	0.07M	0.10M
ETTh2	96	10.54M	0.22M	0.01M	0.04M
	192	10.54M	0.24M	0.02M	0.05M

Appendix

Table 10: The number of parameters for each of the Time Series models.

				Spread		Symmetry			Tail Risk
Model	Dataset	# Para	Mean	Range	SD	Skewness	Shapiro W	Shapiro p	LFO	UFO
32x32x3 Pixel Images & Random Weight Initialization
ResNet20	CIFAR-10	0.27M	91.19%	1.18%	0.22	-0.4176	0.9802	0.1382	3	0
ResNet56		0.85M	92.05%	3.19%	0.61	-0.9379	0.9475	0.0006	5	0
ResNet110		1.73M	93.56%	1.03%	0.17	0.3784	0.9783	0.0981	0	2
ViT-S-8		21.67M	78.16%	1.31%	0.27	-0.4026	0.9747	0.0507	0	0

Appendix

Table 11: Performance and robustness metrics for image classification models. Each row shows results for 100 seed replicates. Columns: # Para (parameter count), Mean (mean top-1 accuracy), Range, SD (standard deviation), Skewness, Shapiro W (Shapiro-Wilk W statistic), Shapiro p (Shapiro-Wilk p-value), LFO (lower fence outliers), UFO (upper fence outliers). Models grouped by image resolution and initialization strategy.

Robustness Measure	W	p-value	r	More Robust	Effect Size
Intra-architecture Wilcoxon Comparison
Range	155.5	0.6204	$-$ 0.114	—	Small
Standard Deviation	135.0	0.3153	$-$ 0.231	—	Small
Skewness	163.0	0.7644	$-$ 0.071	—	Negligible
Lower Fence Outliers	107.0	0.5240	$-$ 0.154	—	Small
Upper Fence Outliers	63.0	0.5301	$-$ 0.177	—	Small

Appendix

Table 12: Wilcoxon signed-rank test results for robustness comparisons between image classification models with different parameter counts. W Statistic and p-value report the Wilcoxon signed-rank test statistic and corresponding p-value for paired comparisons on raw metric values. r denotes the rank-biserial correlation coefficient. More Robust indicates which model size exhibited lower metric values on the robustness measure; — indicates no statistically significant difference. Effect Size reports the rank-biserial correlation magnitude using the thresholds: negligible (

|r|<0.1

), small (

0.1\leq|r|<0.3

), medium (

0.3\leq|r|<0.5

), and large (

|r|\geq 0.5

). Statistically significant results (

p<0.05

) in bold.

					Spread		Symmetry			Tail Risk
Model	Dataset	Horizon	# Para	Mean	Range	SD	Skewness	Shapiro W	Shapiro p	LFO	UFO
Autoformer	ETTh1	96	10.54M	0.4576	0.0644	0.0141	0.7442	0.9494	0.0008	0	5
iTransformer			0.84M	0.4049	0.004	0.0009	0.2521	0.9768	0.0753	0	0
NLinear			0.01M	0.3952	0.0269	0.004	3.447	0.6492	$4.09\times 10^{-4}$	0	5
TSMixer			0.04M	0.3921	0.0033	0.0008	0.1937	0.9775	0.085	0	0
Autoformer	ETTh1	192	10.54M	0.4778	0.058	0.0116	0.1648	0.9884	0.5362	1	0

Appendix

Table 13: Performance and robustness metrics for time series forecasting models on ETTh1 and ETTh2. Each row shows results for 100 seed replicates. Columns: # Para (parameter count), Mean (mean MAE), Range, SD (standard deviation), Skewness, Shapiro W (Shapiro-Wilk W statistic), Shapiro p (Shapiro-Wilk p-value), LFO (lower fence outliers), UFO (upper fence outliers). Models grouped by dataset and forecast horizon.

					Spread		Symmetry			Tail Risk
Model	Dataset	Horizon	# Para	Mean	Range	SD	Skewness	Shapiro W	Shapiro p	LFO	UFO
Autoformer	ETTm1	96	10.54M	0.4669	0.0802	0.017	1.1613	0.903	$1.95\times 10^{-6}$	0	4
iTransformer			0.22M	0.3766	0.0061	0.0013	0.0208	0.9888	0.5718	0	0
NLinear			0.01M	0.3759	0.0038	0.0006	0.9646	0.8816	$2.13\times 10^{-7}$	5	8
TSMixer			0.04M	0.363	0.0063	0.0014	0.2746	0.8816	0.4022	0	0
Autoformer	ETTm1	192	10.54M	0.4933	0.0537	0.0127	0.3945	0.9585	0.0032	0	0

Appendix

Table 14: Performance and robustness metrics for time series forecasting models on ETTm1 and ETTm2. Each row shows results for 100 seed replicates. Columns: # Para (parameter count), Mean (mean MAE), Range, SD (standard deviation), Skewness, Shapiro W (Shapiro-Wilk W statistic), Shapiro p (Shapiro-Wilk p-value), LFO (lower fence outliers), UFO (upper fence outliers). Models grouped by dataset and forecast horizon.

					Spread		Symmetry			Tail Risk
Model	Dataset	Horizon	# Para	Mean	Range	SD	Skewness	Shapiro W	Shapiro p	LFO	UFO
Autoformer	Traffic	96	14.91M	0.393	0.0724	0.0147	1.2906	0.8991	$1.29\times 10^{-6}$	0	6
iTransformer			6.41M	0.2686	0.0025	0.0005	-0.0472	0.9915	0.7809	0	0
NLinear			0.01M	0.3883	0.0056	0.001	0.9392	0.9471	0.0005	0	2
TSMixer			6.27M	0.3135	0.0138	0.0025	0.8202	0.9601	0.0041	0	2
Autoformer	Traffic	192	14.91M	0.4073	0.1007	0.0232	0.5618	0.9542	0.0016	0	0

Appendix

Table 15: Performance and robustness metrics for time series forecasting models on Traffic and Weather. Each row shows results for 100 seed replicates. Columns: # Para (parameter count), Mean (mean MAE), Range, SD (standard deviation), Skewness, Shapiro W (Shapiro-Wilk W statistic), Shapiro p (Shapiro-Wilk p-value), LFO (lower fence outliers), UFO (upper fence outliers). Models grouped by dataset and forecast horizon. Traffic horizons 336 and 720 were not completed for iTransformer due to GPU memory constraints on the NVIDIA A100s.

Robustness Measure	W	p-value	r	More Robust	Effect Size
Cross-architecture Wilcoxon Comparison
Range	1163.0	$\boldsymbol{1.17\times 10^{-14}}$	$-$ 0.758	Smaller	Large
Standard Deviation	934.0	$\boldsymbol{2.29\times 10^{-16}}$	$-$ 0.805	Smaller	Large
Skewness	3203.0	0.0007	$-$ 0.332	Larger	Medium
Upper Fence Outliers	2505.5	0.0004	$-$ 0.364	Larger	Medium

Appendix

Table 16: Wilcoxon signed-rank test results for robustness comparisons between time series forecasting models with different parameter counts. W Statistic and p-value report the Wilcoxon signed-rank test statistic and corresponding p-value for paired comparisons on raw metric values. r denotes the rank-biserial correlation coefficient. More Robust indicates which model size exhibited lower metric values on the robustness measure; — indicates no statistically significant difference. Effect Size reports the rank-biserial correlation magnitude using the thresholds: negligible (

|r|<0.1

), small (

0.1\leq|r|<0.3

), medium (

0.3\leq|r|<0.5

), and large (

|r|\geq 0.5

). Statistically significant results (

p<0.05

) in bold.

Figure 8: Kernel density estimates of top-1 accuracy for image classification models on Oxford Flowers (left) and UC Merced (right). Top row shows ResNet architectures (ResNet18, ResNet50, ResNet101); middle row shows Vision Transformers with random initialization (ViT-Tiny-16, ViT-S-16, ViT-B-16); bottom row shows Vision Transformers with pretrained weights (ViT-Tiny-16, ViT-S-16, ViT-B-16). Each curve represents the distribution across 100 random seeds for a single model-dataset pair.

Figure 9: Kernel density estimates of MAE for time series forecasting models on ETTh2 across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.

Figure 10: Kernel density estimates of MAE for time series forecasting models on ETTm1 across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.

Figure 11: Kernel density estimates of MAE for time series forecasting models on ETTm2 across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.

Figure 12: Kernel density estimates of MAE for time series forecasting models on Traffic across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.

Figure 13: Kernel density estimates of MAE for time series forecasting models on Weather across forecast horizons of 96, 192, 336, and 720. Each panel shows distributions for Autoformer, iTransformer, NLinear, and TSMixer. Each curve represents the distribution across 100 random seeds for a single model-horizon pair.

Figure 14: Mean gradient L2 norm (left) and within-epoch gradient norm variance (right) across 200 training epochs for ResNet110 and ViT-B-8 on CIFAR-10, computed over 100 seeds. Shaded bands indicate the interquartile range across seeds. Dashed vertical lines mark learning rate decay events at epochs 100 and 150.

Figure 15: Perturbation-based sharpness at convergence for ResNet110 and ViT-B-8 on CIFAR-10, computed over 100 seeds. Each box shows the median, interquartile range, and 1.5 x IQR whiskers; points beyond the whiskers are plotted individually.

ML Reproducibility

Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.

See all 138 papers

Oct 21, 2026

Estimating Performance for Safe Deployment of Machine Learning Models

The main challenge in applying deep neural networks to real-world scenarios lies in distributional discrepancies between training and test datasets, often caused by variations in data acquisition conditions. Assessing the performance of pre-trained models without labels is crucial for safety, but this is complicated by deep models' overly optimistic probabilistic outputs, often exacerbated by class imbalances, leading to unreliable predictions. To address these issues, we introduce MOVAL, a novel MOdel eVALuation tool that estimates performance using unlabeled test data. MOVAL incorporates class-wise confidence calibration to adjust model outputs, allowing for the estimation of metrics beyond accuracy and mitigating the detrimental effects of class imbalance on confidence calibration. Additionally, we propose an ensemble approach that combines multiple calibrated confidence scores for a more robust solution. We validate MOVAL on natural and medical image datasets with various acquisition shifts, covering tasks from classification to segmentation. It achieves state-of-the-art results across evaluation metrics such as accuracy, F1-Score, AUC, and more. Furthermore, we demonstrate that MOVAL provides accurate performance estimations at the class, instance, and pixel levels, ensuring the transparent and safe deployment of machine learning models. We provide MOVAL as a deployment tool for monitoring and estimating performance under a range of domain shifts.

Zeju Li, Konstantinos Kamnitsas, Mobarakol Islam +4

College of Biomedical Engineering, Fudan University, Shanghai, China · Oxford Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK · Department of Engineering Scienc, University of Oxford, Oxford, UK +4

Distribution Shift Model Calibration

Ask PDF

May 20, 2026

A New Framework to Analyse the Distributional Robustness of Deep Neural Networks

Deep neural networks have achieved impressive performance on a variety of tasks, but their brittleness to distributional shifts remains a significant barrier to real-world deployment. In this paper, we propose a framework to analyse and quantify the distributional robustness of neural networks by studying the interactions between layer weights and activations. We model these interactions using Bernoulli distributions, using the separation between classes as a diagnostic proxy for robustness. We demonstrate the usefulness of this framework through models trained on CIFAR-10 and ImageNet. We show that our proposed metrics can distinguish between networks that have memorised their training data and those that have not. We also perform analogous experiments in the activation space and find that the same properties do not hold up. Additionally, we investigate the behaviour of our metrics under various distribution shifts and show that these shifts reduce separation under our path-based diagnostics. Our results suggest that this framework provides useful model-level diagnostics of representation structure and robustness.

Divij Khaitan, Subhashis Banerjee

Microsoft · Ashoka University

Distribution Shift Robustness Neural Network Generalization Neural Network Robustness

Ask PDF

Jun 4, 2026

Performance Variation in Deep Reinforcement Learning

Deep reinforcement learning (RL) algorithms often suffer from low run-to-run robustness, manifesting as significant performance variation across independent runs of identically configured agents. Although this issue poses a spectrum of challenges across research and practice, relatively few studies develop methods to evaluate it; RL research instead often reports uncertainty in the estimated mean performance. In this paper, we outline the limitations of conventional uncertainty and variation estimates, particularly their misalignment with purpose and the risk of underreporting. We then propose an alternative percentile-based statistic and visualization method, min-max IPR and run-wise percentile highlighting, respectively. These percentile-based tools are easy to interpret and rely on standard properties of sample percentiles, providing rich information about run-to-run performance variation. We demonstrate this through three case studies. First, we show that LayerNorm and penultimate-layer normalizations narrow performance variation in PPO, whereas the variation is mostly unchanged in SAC. Second, we compare PPO, SAC, TD-MPC, and TD-MPC2, and show TD-MPC exhibits the least variation while being the most data efficient among the four. Finally, in a comparison of DQN and Rainbow on five Atari environments, we show that both algorithms exhibit similar levels of performance variation.

Haruto Tanaka, A. Rupam Mahmood

Department of Computing Science University of Alberta Alberta Machine Intelligence Institute (Amii)

Reinforcement Learning

Ask PDF

Abstract

ML Reproducibility

Explore similar work

Estimating Performance for Safe Deployment of Machine Learning Models

A New Framework to Analyse the Distributional Robustness of Deep Neural Networks

Performance Variation in Deep Reinforcement Learning