The Impact of Stochasticity on the Rashomon Effect in Machine Learning

Andrea Apicella; Francesco Isgrò; Andrea Pollastro; Roberto Prevete

cs.LGSep 26, 2026

The Impact of Stochasticity on the Rashomon Effect in Machine Learning

Authors: Andrea Apicella, Francesco Isgrò, Andrea Pollastro, Roberto Prevete

Organizations: Department of Information Engineering, Electrical Engineering, and Applied Mathematics (DIEM), University of Salerno, Fisciano (SA), Italy · Department of Electrical Engineering and Information Technology (DIETI), University of Naples Federico II, Naples, Italy

Abstract

Neural network training is inherently stochastic, with factors such as weight initialization leading to distinct models despite comparable predictive performance. This phenomenon is commonly associated with the Rashomon effect, which describes the existence of multiple near-optimal models for the same task. Although the Rashomon effect has received increasing attention, it remains unclear whether different sources of training stochasticity contribute similarly or differently to its manifestations. In this work, we present an empirical study of the Rashomon phenomenon along three complementary dimensions: solution-space multiplicity, predictive multiplicity, and decision-basis multiplicity. These dimensions are quantified through the size of the empirical Rashomon set, predictive ambiguity, and agreement between XAI attribution maps, respectively. By independently controlling three standard sources of stochasticity, namely weight initialization, mini-batch data ordering, and dropout, we isolate their respective contributions to each dimension of the Rashomon phenomenon. Experiments on tabular and image classification benchmarks reveal that these sources affect the three dimensions in different ways. In particular, larger empirical Rashomon sets do not necessarily correspond to greater predictive disagreement or lower explanation agreement, indicating that solution-space, predictive, and decision-basis multiplicity capture complementary rather than interchangeable aspects of the Rashomon effect. Overall, our results show that training stochasticity influences not only predictive performance but also the stability of predictions and explanations, highlighting the importance of identifying the specific sources of stochasticity responsible for different manifestations of the Rashomon phenomenon when assessing the reliability, reproducibility, and interpretability of neural network models.

Figures & tables

Figure 1: A recap of the investigated Rashomon effect dimensions. See text for further details.

Hyperparameter	Tabular datasets (MLP)	Image datasets (ResNet-20)
Architecture	MLP (2 hidden layers)	ResNet-20
Hidden units	256, 256	–
Activation function	ReLU	ReLU
Optimizer	SGD	SGD
Momentum $\mu$	0.9	0.9
Learning rate $\eta$	0.001	0.1

Table 1: Hyperparameter values for the adopted architectures.

Number of models	Median SD	75th percentile SD	Median absolute deviation
10	0.097	0.170	0.143
20	0.074	0.137	0.082
30	0.062	0.118	0.054
50	0.046	0.095	0.033
75	0.038	0.079	0.019
100	0.032	0.066	0.012

Table 2: Variance-stabilisation analysis for the empirical Rashomon-set fraction. Results aggregate 80 configurations across five datasets, four batch sizes, four stochasticity conditions, and ten performance tolerances. Each configuration was evaluated using 500 bootstrap resamples. SD denotes the bootstrap standard deviation; absolute deviation is computed relative to the estimate obtained from the complete observed pool.

		MLP			ResNet-20
Source	BS	ISOLET	Optical	Waveform	CIFAR-10	F-MNIST
All	32	$86.4\pm 1.5$	$95.4\pm 0.6$	$84.3\pm 1.2$	$86.7\pm 0.4$	$93.1\pm 0.2$
	64	$86.4\pm 1.6$	$95.6\pm 0.5$	$84.2\pm 1.1$	$86.4\pm 0.3$	$93.0\pm 0.2$
	128	$86.0\pm 1.7$	$95.6\pm 0.5$	$84.2\pm 1.1$	$86.0\pm 0.4$	$92.9\pm 0.2$
	256	$86.2\pm 1.8$	$95.6\pm 0.5$	$84.2\pm 1.1$	$85.7\pm 0.5$	$92.8\pm 0.3$
Init	32	$86.2\pm 1.6$	$95.4\pm 0.6$	$84.5\pm 1.3$	$86.7\pm 0.3$	$93.0\pm 0.2$

Table 3: Mean test accuracy (%)

\pm

standard deviation across datasets, batch sizes, and stochasticity sources.

Panel of Figure 2 — Figure 2: Rashomon set size as a function of $\epsilon$ for tabular datasets (MLP). Rows correspond to datasets, columns to batch sizes.

Panel of Figure 3 — Figure 3: Rashomon set size as a function of $\epsilon$ for image datasets (ResNet-20). Rows correspond to datasets, columns to batch sizes.

Panel of Figure 4 — Figure 4: Ambiguity as a function of $\epsilon$ for tabular datasets (MLP), broken down by stochasticity source. Rows correspond to datasets, columns to batch sizes.

Panel of Figure 5 — Figure 5: Ambiguity as a function of $\epsilon$ for image datasets (ResNet-20), broken down by stochasticity source. Rows correspond to datasets, columns to batch sizes.

Panel of Figure 6 — Figure 6: Top- $k$ attribution overlap obtained using Vanilla Gradients (VG) as a function of the performance tolerance $\epsilon$ for the tabular datasets and MLP architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains separate panels for Top-5%, Top-10%, Top-20%, and Top-25% overlap. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the corresponding empirical Rashomon set contains fewer than two models, since pairwise attribution agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.

Panel of Figure 7 — Figure 7: Top- $k$ attribution overlap obtained using Vanilla Gradients (VG) as a function of the performance tolerance $\epsilon$ for the image datasets and ResNet-20 architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains separate panels for Top-5%, Top-10%, Top-20%, and Top-25% overlap. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the corresponding empirical Rashomon set contains fewer than two models, since pairwise attribution agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.

Panel of Figure 8 — Figure 8: Per-instance predictive disagreement at $\epsilon=0.03$ across batch sizes. For each stochasticity condition $s$ , the colour of the cell associated with test instance $x_{i}$ represents $d_{i}^{(s)}=\frac{1}{\binom{K_{s}}{2}}\sum_{m<n}\mathbf{1}\!\left[\hat{y}_{m}(x_{i})\neq\hat{y}_{n}(x_{i})\right],$ namely, the fraction of pairs among the $K_{s}$ models in the corresponding empirical Rashomon set that predict different classes for that instance. Black indicates complete or near-complete agreement, while lighter colours indicate greater pairwise disagreement. Within each dataset and batch-size configuration, instances are ordered from left to right according to their mean $d_{i}^{(s)}$ across the four stochasticity conditions. The same instance therefore occupies the same column in all four rows of a given panel, with the instances exhibiting the greatest average disagreement appearing on the right. The horizontal axis reports the actual rank in this ordering, from 1 to the total number of test instances.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Top- $k$ attribution overlap obtained using Integrated Gradients (IG) as a function of the performance tolerance $\epsilon$ for the tabular datasets and MLP architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.

Panel of Figure 10 — Figure 10: Top- $k$ attribution overlap obtained using Integrated Gradients (IG) as a function of the performance tolerance $\epsilon$ for the image datasets and ResNet-20 architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.

Panel of Figure 11 — Figure 11: Top- $k$ attribution overlap obtained using Layer-wise Relevance Propagation (LRP) as a function of the performance tolerance $\epsilon$ for the tabular datasets and MLP architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.

Panel of Figure 12 — Figure 12: Top- $k$ attribution overlap obtained using Layer-wise Relevance Propagation (LRP) as a function of the performance tolerance $\epsilon$ for the image datasets and ResNet-20 architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.

Panel of Figure 13 — Figure 13: Top- $k$ attribution overlap obtained using Occlusion as a function of the performance tolerance $\epsilon$ for the tabular datasets and MLP architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.

Panel of Figure 14 — Figure 14: Top- $k$ attribution overlap obtained using Occlusion as a function of the performance tolerance $\epsilon$ for the image datasets and ResNet-20 architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 520 papers

Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity
Sep 1, 2026Sinjini Banerjee, Tim Marrinan, Anand D. SarwateSelective Prediction Ensemble Learning
Department of Electrical and Computer Engineering, Rutgers University · Pacific Northwest National Lab
Resolving Predictive Multiplicity for the Rashomon Set
Jan 14, 2026Parian Haghighat, Hadis Anahideh, Cynthia Rudin
University of Illinois Chicago, Chicago, IL, USA · Duke University, Durham, NC, USA
Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning
Jun 4, 2026Helge Spieker, Jørn Eirik Betten, Arnaud GotliebMetamorphic Testing Explainability Evaluation
Simula Research Laboratory Oslo, Norway

Sep 1, 2026J/K move · Enter open · S save

Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity

Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate

Department of Electrical and Computer Engineering, Rutgers University · Pacific Northwest National Lab

Selective Prediction Ensemble Learning LLM Auditing

Open paper AskPDF arXiv

The Rashomon effect is a machine learning phenomenon where equally accurate models produce different predictions for the same inputs (predictive multiplicity). Existing work primarily focuses on multiplicity within individual models, but in more complex decision systems, the impact of the Rashomon effect is less well understood. In this work, we study multiplicity from the perspective of auditing incorrect ensemble predictions, where the decision to divert an instance for human review is based on a consistency criterion that combines the ensemble margin with a measure of local prediction variability for each constituent model. With mild assumptions about stability and smoothness, we show that the consistency scores of finite ensembles converge to the corresponding consistency score of the expected model from the Rashomon set as the ensemble size and the number of samples used to measure local prediction variability increase. To demonstrate the efficacy of the proposed criterion, we evaluate the framework with respect to transformer models applied to natural language understanding tasks and parameter-efficient fine-tuning of large language models used for tabular data classification tasks. Our experiments show that ensembling models from the Rashomon set substantially reduces the risk of incorrect predictions going unchecked compared with auditing a single model, while incurring only a moderate increase in the number of diversions. Moreover, the auditing behavior of the full Rashomon set can be closely approximated by finite ensembles of relatively modest size, with the risk approaching zero for some datasets. We further demonstrate that the proposed measure exhibits stronger agreement with established predictive multiplicity metrics than existing consistency measures, providing a more reliable way to capture multiplicity in the Rashomon set.

Abstract

Neural Network Training Dynamics

Explore similar work

Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity

Resolving Predictive Multiplicity for the Rashomon Set

Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning