CASHomon Sets: Efficient Rashomon Sets Across Multiple Model Classes and their Hyperparameters

Fiona Katharina Ewald; Martin Binder; Matthias Feurer; Bernd Bischl; Giuseppe Casalicchio

cs.LGMar 16, 2026

CASHomon Sets: Efficient Rashomon Sets Across Multiple Model Classes and their Hyperparameters

Authors: Fiona Katharina Ewald, Martin Binder, Matthias Feurer, Bernd Bischl, Giuseppe Casalicchio

Organizations: Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany

Abstract

Rashomon sets are model sets within one model class that perform nearly as well as a reference model from the same model class. They reveal the existence of alternative well-performing models, which may support different interpretations. This enables selecting models that match domain knowledge, hidden constraints, or user preferences. However, efficient construction methods currently exist for only a few model classes. Applied machine learning usually searches many model classes, and the best class is unknown beforehand. We therefore study Rashomon sets in the combined algorithm selection and hyperparameter optimization (CASH) setting and call them CASHomon sets. We propose TruVaRImp, a model-based active learning algorithm for level set estimation with an implicit threshold, and provide convergence guarantees. On synthetic and real-world datasets, TruVaRImp reliably identifies CASHomon sets members and matches or outperforms naive sampling, Bayesian optimization, classical and implicit level set estimation methods, and other baselines. Our analyses of predictive multiplicity and feature-importance variability across model classes question the common practice of interpreting data through a single model class.

Figures & tables

\begin{split}\hskip-6.45831pt\text{find }\lambda_{t}&\leftarrow\mathop{\mathrm{arg\,max}}_{\lambda\in\tilde{\bm{\Lambda}}}\Big\{\Big[\Delta_{i}(U_{t-1},\sigma_{t-1}^{2},1)-\Delta_{i}(U_{t-1},\sigma_{t-1|\lambda}^{2},1)+\\ &\Delta_{i}(M_{t-1},\sigma_{t-1}^{2},1{+}\varepsilon_{\mathrm{rel}})-\Delta_{i}(M_{t-1},\sigma_{t-1|\lambda}^{2},1{+}\varepsilon_{\mathrm{rel}})\Big]\Big/\operatorname{cost}(\lambda)\Big\}\\ \end{split}

Algorithm 1 Truncated Variance Reduction for Implicitly Defined LSE ( TruVaRImp )

Figure 1: Illustration of the TruVaRImp algorithm. The true (unknown) function is shown in green, with known function values

c(\lambda)

as black dots, through which a GP model is fit (purple line with transparent confidence region).

M_{t}

is the set of potential minimizers (purple strip at the bottom), for which the confidence interval crosses the “pessimistic” (i.e., highest possible within confidence region) minimum

c_{\mathrm{min},t}^{\mathrm{pes}}

(upper red dashed line). Configuration points

\lambda

are classified as belonging to the lower or upper set (

L_{t}

and

H_{t}

, not shown), or remain unclassified (

U_{t}

, blue strip at the bottom), depending on whether their confidence interval crosses the region of plausible threshold values

[h_{t}^{\mathrm{opt}},h_{t}^{\mathrm{pes}}]

(blue dashed lines). In this illustration, these are relative to the minimum with

\varepsilon_{\mathrm{rel}}=2,\varepsilon_{\mathrm{abs}}=0

. TruVaRImp selects points based on how much evaluating them will reduce the variance for candidates in both

M_{t}

and

U_{t}

that goes beyond their target confidence values

\eta_{(i)}

the most (green and purple ribbons around the GP mean).

Figure 2: Mean CASHomon set algorithm progress in terms of F1 score of the surrogate model predicting CASHomon set membership. The iteration count does not include the initial sample of 30 points per model class. The left facet shows the mean values over all datasets. The middle and right show individual performance on two example datasets.

Panel of Figure 3 — Figure 3: Test performance and RC for tasks BC, CR, and CS (binarized) across the TreeFARMS Rashomon set and our CASHomon set. Top row: Brier score distributions. Bottom row: RC (y-axis) versus Brier score (x-axis); horizontal boxplots summarize test score distributions; triangles indicate the reference model.

Figure 4: VICs for our CASHomon set (left) and for the TreeFARMS Rashomon set (right) on task CS . For each feature, PFI values across models are shown as a boxplot together with a point cloud colored by model class, where identical values are vertically jittered. PFI values are scaled: the maximal PFI value for a model equals 1. In our CASHomon set, most models are xgb models (178), followed by nnet (66) and cart (33) models, but all model classes are present.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visualization of Rashomon sets on a simple regression problem. Every coordinate on the plot corresponds to a linear model with given coefficients. The “true” model $f^{\star}$ lies at $(1,1)$ and has RMSE $0.5$ . The black ellipse indicates the Rashomon set when taking $f_{\text{ref}}=f^{\star}$ , $\varepsilon=0.15$ , and the true risk $\mathcal{R}$ . The red cross indicates $\hat{f^{\star}}$ , a model that minimizes the empirical risk $\mathcal{R}_{\text{emp}}$ , and the red ellipse represents the Rashomon set when using this model as reference. The shaded blue area represents $\mathcal{H}^{\mathcal{D}}_{\text{HPO}}$ as defined in Eq. ( 1 ), the models resulting from fitting an elastic net model (using the glmnet R-package [ 29 ] ) and parameterized with the lambda and alpha HPs, on the specific available dataset. The shade of blue indicates the value of alpha ; lambda is not shown (though models with larger lambda values lie closer to the origin, corresponding to the regularizing effect of lambda ). The shaded area within the red ellipse is the intersection of the Rashomon set and the models achievable through HP variation.

Name	Description	$N$	#Num	#Cat	Type	Reference
BC	Breast Cancer dataset	683	9	0	classification	[ 27 ]
CR	Car dataset	1728	15	0	classification	[ 6 ]
CS	COMPAS dataset	6172	5	6	classification	[ 1 ]
CS (binarized)	binarized CS	6907	12	0	classification	[ 1 ]
FC	FICO dataset	10459	17	0	classification	footnote 6
GC	German Credit dataset	1000	3	18	classification	[ 15 ]

Appendix

Table 1: Overview of the datasets used in the experiments. We abbreviate the columns as #Num for number of numeric features, #Cat for number of categorical features, and

N

for total number of samples.

HP	Type	Range
alpha	continuous	[ $10^{-3},10^{3}$ ] (log)
colsample_bylevel	continuous	[0.1, 1]
colsample_bytree	continuous	[0.1, 1]
eta	continuous	[ $10^{-4},1$ ] (log)
lambda	continuous	[ $10^{-3},10^{3}$ ] (log)
max_depth	integer	[1, 20] (log)

Appendix

Table 2: HP search space for xgb .

HP	Type	Range
cp	continuous	[ $10^{-4},0.2$ ] (log)
minbucket	integer	[1, 64] (log)
minsplit	integer	[2, 128] (log)

Appendix

Table 3: HP search space for cart .

HP	Type	Range
decay	continuous	[ $10^{-6},1$ ] (log)
size	integer	[8, 512] (log)
skip	categorical	{TRUE, FALSE}

Appendix

Table 4: HP search space for nnet .

HP	Type	Range
alpha	continuous	[0, 1]
lambda	continuous	[ $10^{-4},10^{3}$ ] (log)

Appendix

Table 5: HP search space for glmnet .

HP	Type	Range
cost	continuous	[ $10^{-4},10^{4}$ ] (log)
gamma	continuous	[ $10^{-4},10^{4}$ ] (log)
kernel	categorical	{linear, radial}

Appendix

Table 6: HP search space for svm .

HP	Type	Range
regularization	continuous	[ $10^{-4},0.49$ ] (log)
balance	categorical	{TRUE, FALSE}
depth_budget	integer	[1, 100] (log)

Appendix

Table 7: HP search space for gosdt .

	glmnet	gosdt	nnet	svm.linear	svm.radial	cart	xgb
BC	234*	0	130	0	0	0	0
BS	0	0	0	0	0	0	346*
CR	0	0	20*	0	0	0	0
CS	40	0	9	0	0	2	297*
CS (binarized)	22	19	66	5	15	33	178*
FC	33	0	2	5	1	10	288*

Appendix

Table 8: Number of models of each model class within a CASHomon set found by TruVaRImp per task. “*” marks the reference model class.

Figure 6: Mean F1 score of the surrogate model predicting CASHomon set membership.

Panel of Figure 7 — Figure 7: (a) VICs for CASHomon set (left) and TreeFARMS Rashomon set (right) for task MK . For each feature, a point cloud colored by model class and a boxplot are displayed. If several models yield the same importance value, the points in the cloud scatter vertically. PFI values are scaled: the maximal PFI value for a model equals 1. (b) RC (y-axis) versus Brier score (x-axis); horizontal boxplots summarize test score distributions; triangles indicate the reference model.

Panel of Figure 8 — Figure 8: VICs for CASHomon sets (left) and TreeFARMS Rashomon sets (right) for tasks CR , BC , and FC . For each feature, a point cloud colored by model class and a boxplot are displayed. If several models yield the same importance value, the points in the cloud scatter vertically. PFI values are scaled: the maximal PFI value for a model equals 1.

Hyperparameter Optimization

Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.

See all 161 papers

Parameter-Efficient Construction of the Rashomon Slice for Concept Bottleneck Models
Nov 24, 2025Shihan Feng, Cheng Zhang, Michael Xi +3Concept Bottleneck Models Classification
UNC Chapel Hill · Rutgers University · Duke University
From Rashomon Theory to PRAXIS: Efficient Decision Tree Rashomon Sets
May 29, 2026Zakk Heile, Hayden McTavish, Varun Babbar +2Interpretable ML Decision Tree Learning
Department of Computer Science, Duke University, Durham, USA · Department of Computer Science, University of British Columbia, Vancouver, Canada.
ArborEnum: Decision Tree Rashomon Sets over Continuous Features
Aug 5, 2026Zakk Heile, Hayden McTavish, Margo Seltzer +1Decision Tree Learning
Department of Computer Science, Duke University, Durham, USA · Department of Computer Science, University of British Columbia, Vancouver, Canada

Nov 24, 2025J/K move · Enter open · S save

Parameter-Efficient Construction of the Rashomon Slice for Concept Bottleneck Models

Shihan Feng, Cheng Zhang, Michael Xi, Ethan Hsu+2

UNC Chapel Hill · Rutgers University · Duke University

Concept Bottleneck Models Classification

Open paper AskPDF arXiv

In many machine learning problems, there may exist multiple models that achieve nearly identical predictive performance while relying on fundamentally different internal logic. However, standard training procedures produce a single model, offering no practical way to explore alternatives that may better suit downstream needs. The set of these equally accurate models is known as the Rashomon set. Exploring the Rashomon set is particularly challenging in large and complex hypothesis spaces, such as Concept Bottleneck Models (CBMs), which are widely used in computer vision to make predictions through intermediate, human-understandable concepts. In this paper, we provide a method for efficiently exploring the Rashomon set of CBMs. Our framework introduces a specialized parallel parameter-efficient adaptation module, combined with a checkpointing scheme and a concept diversity objective, to generate multiple equally accurate CBMs from a single training process. Empirical results show that our method finds models with better diversity than baselines while using much less memory. We further demonstrate that access to these diverse yet accurate CBMs enables trustworthy model selection, resolution of inter-class confusion, and reliable abstention in decision-making.

Abstract

Hyperparameter Optimization

Explore similar work

Parameter-Efficient Construction of the Rashomon Slice for Concept Bottleneck Models

From Rashomon Theory to PRAXIS: Efficient Decision Tree Rashomon Sets

ArborEnum: Decision Tree Rashomon Sets over Continuous Features