The Impact of Stochasticity on the Rashomon Effect in Machine Learning
Authors: Andrea Apicella, Francesco Isgrò, Andrea Pollastro, Roberto Prevete
Organizations: Department of Information Engineering, Electrical Engineering, and Applied Mathematics (DIEM), University of Salerno, Fisciano (SA), Italy · Department of Electrical Engineering and Information Technology (DIETI), University of Naples Federico II, Naples, Italy
Neural network training is inherently stochastic, with factors such as weight initialization leading to distinct models despite comparable predictive performance. This phenomenon is commonly associated with the Rashomon effect, which describes the existence of multiple near-optimal models for the same task. Although the Rashomon effect has received increasing attention, it remains unclear whether different sources of training stochasticity contribute similarly or differently to its manifestations. In this work, we present an empirical study of the Rashomon phenomenon along three complementary dimensions: solution-space multiplicity, predictive multiplicity, and decision-basis multiplicity. These dimensions are quantified through the size of the empirical Rashomon set, predictive ambiguity, and agreement between XAI attribution maps, respectively. By independently controlling three standard sources of stochasticity, namely weight initialization, mini-batch data ordering, and dropout, we isolate their respective contributions to each dimension of the Rashomon phenomenon. Experiments on tabular and image classification benchmarks reveal that these sources affect the three dimensions in different ways. In particular, larger empirical Rashomon sets do not necessarily correspond to greater predictive disagreement or lower explanation agreement, indicating that solution-space, predictive, and decision-basis multiplicity capture complementary rather than interchangeable aspects of the Rashomon effect. Overall, our results show that training stochasticity influences not only predictive performance but also the stability of predictions and explanations, highlighting the importance of identifying the specific sources of stochasticity responsible for different manifestations of the Rashomon phenomenon when assessing the reliability, reproducibility, and interpretability of neural network models.
Figures & tables
Figure 1: A recap of the investigated Rashomon effect dimensions. See text for further details.
Hyperparameter
Tabular datasets (MLP)
Image datasets (ResNet-20)
Architecture
MLP (2 hidden layers)
ResNet-20
Hidden units
256, 256
–
Activation function
ReLU
ReLU
Optimizer
SGD
SGD
Momentum μ
0.9
0.9
Learning rate η
0.001
0.1
Table 1: Hyperparameter values for the adopted architectures.
Number of models
Median SD
75th percentile SD
Median absolute deviation
10
0.097
0.170
0.143
20
0.074
0.137
0.082
30
0.062
0.118
0.054
50
0.046
0.095
0.033
75
0.038
0.079
0.019
100
0.032
0.066
0.012
Table 2: Variance-stabilisation analysis for the empirical Rashomon-set fraction. Results aggregate 80 configurations across five datasets, four batch sizes, four stochasticity conditions, and ten performance tolerances. Each configuration was evaluated using 500 bootstrap resamples. SD denotes the bootstrap standard deviation; absolute deviation is computed relative to the estimate obtained from the complete observed pool.
MLP
ResNet-20
Source
BS
ISOLET
Optical
Waveform
CIFAR-10
F-MNIST
All
32
86.4±1.5
95.4±0.6
84.3±1.2
86.7±0.4
93.1±0.2
64
86.4±1.6
95.6±0.5
84.2±1.1
86.4±0.3
93.0±0.2
128
86.0±1.7
95.6±0.5
84.2±1.1
86.0±0.4
92.9±0.2
256
86.2±1.8
95.6±0.5
84.2±1.1
85.7±0.5
92.8±0.3
Init
32
86.2±1.6
95.4±0.6
84.5±1.3
86.7±0.3
93.0±0.2
Table 3: Mean test accuracy (%) ± standard deviation across datasets, batch sizes, and stochasticity sources.
Figure 2: Rashomon set size as a function of ϵ for tabular datasets (MLP). Rows correspond to datasets, columns to batch sizes.
Figure 3: Rashomon set size as a function of ϵ for image datasets (ResNet-20). Rows correspond to datasets, columns to batch sizes.
Figure 4: Ambiguity as a function of ϵ for tabular datasets (MLP), broken down by stochasticity source. Rows correspond to datasets, columns to batch sizes.
Figure 5: Ambiguity as a function of ϵ for image datasets (ResNet-20), broken down by stochasticity source. Rows correspond to datasets, columns to batch sizes.
Figure 6: Top- k attribution overlap obtained using Vanilla Gradients (VG) as a function of the performance tolerance ϵ for the tabular datasets and MLP architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains separate panels for Top-5%, Top-10%, Top-20%, and Top-25% overlap. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the corresponding empirical Rashomon set contains fewer than two models, since pairwise attribution agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.
Figure 7: Top- k attribution overlap obtained using Vanilla Gradients (VG) as a function of the performance tolerance ϵ for the image datasets and ResNet-20 architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains separate panels for Top-5%, Top-10%, Top-20%, and Top-25% overlap. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the corresponding empirical Rashomon set contains fewer than two models, since pairwise attribution agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.
Figure 8: Per-instance predictive disagreement at ϵ=0.03 across batch sizes. For each stochasticity condition s , the colour of the cell associated with test instance xi represents di(s)=(2Ks)1∑m<n1[y^m(xi)=y^n(xi)], namely, the fraction of pairs among the Ks models in the corresponding empirical Rashomon set that predict different classes for that instance. Black indicates complete or near-complete agreement, while lighter colours indicate greater pairwise disagreement. Within each dataset and batch-size configuration, instances are ordered from left to right according to their mean di(s) across the four stochasticity conditions. The same instance therefore occupies the same column in all four rows of a given panel, with the instances exhibiting the greatest average disagreement appearing on the right. The horizontal axis reports the actual rank in this ordering, from 1 to the total number of test instances.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Top- k attribution overlap obtained using Integrated Gradients (IG) as a function of the performance tolerance ϵ for the tabular datasets and MLP architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.
Figure 10: Top- k attribution overlap obtained using Integrated Gradients (IG) as a function of the performance tolerance ϵ for the image datasets and ResNet-20 architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.
Figure 11: Top- k attribution overlap obtained using Layer-wise Relevance Propagation (LRP) as a function of the performance tolerance ϵ for the tabular datasets and MLP architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.
Figure 12: Top- k attribution overlap obtained using Layer-wise Relevance Propagation (LRP) as a function of the performance tolerance ϵ for the image datasets and ResNet-20 architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.
Figure 13: Top- k attribution overlap obtained using Occlusion as a function of the performance tolerance ϵ for the tabular datasets and MLP architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.
Figure 14: Top- k attribution overlap obtained using Occlusion as a function of the performance tolerance ϵ for the image datasets and ResNet-20 architecture. Rows correspond to datasets and columns to batch sizes. Each plot contains the results for Top-5%, Top-10%, Top-20%, and Top-25%. Lower overlap indicates stronger attribution-based evidence of decision-basis multiplicity. Error bars report the standard deviation across model-pair–instance comparisons. Points are omitted when the empirical Rashomon set contains fewer than two models, since pairwise agreement is then undefined. Small horizontal offsets are used only to make overlapping markers and error bars visible.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
The Rashomon effect is a machine learning phenomenon where equally accurate models produce different predictions for the same inputs (predictive multiplicity). Existing work primarily focuses on multiplicity within individual models, but in more complex decision systems, the impact of the Rashomon effect is less well understood. In this work, we study multiplicity from the perspective of auditing incorrect ensemble predictions, where the decision to divert an instance for human review is based on a consistency criterion that combines the ensemble margin with a measure of local prediction variability for each constituent model. With mild assumptions about stability and smoothness, we show that the consistency scores of finite ensembles converge to the corresponding consistency score of the expected model from the Rashomon set as the ensemble size and the number of samples used to measure local prediction variability increase. To demonstrate the efficacy of the proposed criterion, we evaluate the framework with respect to transformer models applied to natural language understanding tasks and parameter-efficient fine-tuning of large language models used for tabular data classification tasks. Our experiments show that ensembling models from the Rashomon set substantially reduces the risk of incorrect predictions going unchecked compared with auditing a single model, while incurring only a moderate increase in the number of diversions. Moreover, the auditing behavior of the full Rashomon set can be closely approximated by finite ensembles of relatively modest size, with the risk approaching zero for some datasets. We further demonstrate that the proposed measure exhibits stronger agreement with established predictive multiplicity metrics than existing consistency measures, providing a more reliable way to capture multiplicity in the Rashomon set.
Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate
Department of Electrical and Computer Engineering, Rutgers University · Pacific Northwest National Lab
The existence of multiple, equally accurate models for a given predictive task leads to predictive multiplicity, where a Rashomon set of models achieve similar accuracy but diverge in their individual predictions. This inconsistency undermines trust in high-stakes applications where we want consistent predictions. We propose three approaches to reduce inconsistency among predictions for the members of the Rashomon set. The first approach is outlier correction. An outlier has a label that none of the good models are capable of predicting correctly. Outliers can cause the Rashomon set to have high variance predictions in a local area, so fixing them can lower variance. Our second approach is local patching. In a local region around a test point, models may disagree with each other because some of them are biased. We can detect and fix such biases using a validation set, which also reduces multiplicity. Our third approach is pairwise reconciliation, where we find pairs of models that disagree on a region around the test point. We modify predictions that disagree, making them less biased. These three approaches can be used together or separately, and they each have distinct advantages. The reconciled predictions can then be distilled into a single interpretable model for real-world deployment. In experiments across multiple datasets, our methods reduce disagreement metrics while maintaining competitive accuracy.
Parian Haghighat, Hadis Anahideh, Cynthia Rudin
University of Illinois Chicago, Chicago, IL, USA · Duke University, Durham, NC, USA
Multiple machine learning models can achieve near-equivalent predictive performance on the same task, yet provide divergent feature-based explanations. This is called the Rashomon effect of (explainable) machine learning, and it raises the question of which explanations, if any, are trustworthy. We propose a framework based on metamorphic testing that assesses explanation faithfulness without requiring ground-truth labels by exploring attributed feature importance from post-hoc explanation methods. Five metamorphic relations formalize expected consistency properties between model behavior and feature attributions. We apply this general framework to two tabular regression datasets and two post-hoc explainers (SHAP and LIME) to demonstrate the approach. The framework offers a practical, model-agnostic tool for selecting accurate models with reliable and trustworthy explanations.