Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations

Marcin Kostrzewa; Maciej Zięba

cs.LGSep 25, 2026

Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations

Authors: Marcin Kostrzewa, Maciej Zięba

Organizations: Department of Artificial Intelligence Wrocław University of Science and Technology

Abstract

Robust counterfactual explanations promise recourse that still works after the model behind it changes. Whether they keep that promise depends on what the change is. A small perturbation of the parameters, retraining on new data, and a new architecture are different events, and each existing method is evaluated against the one it was built for. Reported robustness scores, therefore, answer different questions and cannot be compared. We propose a unified cross-family evaluation protocol that holds factual instances and generated counterfactuals fixed while testing every method against the same eight types of model change. The benchmark compares six robust methods and two standard baselines on four tabular datasets. It characterizes every changed classifier through its outputs and reports empirical robustness together with coverage, base validity, and proximity. We find that relative performance and failure modes vary across change families. Bounded parameter perturbations change 0.95% of test predictions on average, compared with 4.9% for bootstrap retraining. Methods with guarantees for these perturbations do not necessarily transfer to other changes. RobX transfers most consistently in our experiments, although greater stability can require larger interventions. We argue that robust CFE methods should be evaluated through a common protocol that specifies the model changes, measures their realized behavioral magnitude, and keeps generation performance separate from robustness.

Figures & tables

Method	Representation of model change	Robustness formulation	Changed models in original evaluation
ROAR-LIME	Bounded coefficients and intercept of a local linear surrogate	Objective under the most unfavorable allowed change	Retrained on temporally or geographically shifted data
RBR	Wasserstein ambiguity set around a Gaussian mixture	Objective under the most unfavorable allowed distribution	Retrained on shifted data
RobX	Local input stability used as a proxy for retraining	Stability threshold	Retrained after small changes to the training data
RNCE	Interval-bounded parameter changes	Validity certificate for every allowed change	Incremental, leave-one-out, and full retraining
AP $\Delta$ S	Random bounded parameter perturbations	Probabilistic certificate	One model retrained on added data
BetaRCE	User-defined distribution of admissible classifiers	Probabilistic bound	Retrained with new data, hyperparameters, or architectures

Table 1: Model-change assumptions and robustness formulations of the methods evaluated in this work, and the changed models used in their original evaluations.

Family	Construction
New initialization	Same data, architecture, and training configuration but different seed
Bootstrap	Bootstrap resampling of the training set
Data deletion	Random deletion of 1%, 5%, or 10% of the training observations
Data addition	Addition of 25%, 50%, 75%, or 100% of the reserved update pool
Label update	Random label flips for 1%, 5%, or 10% of the training observations
Training configuration	Prespecified combinations of optimizer, learning rate, and weight decay

Table 2: Construction of the held-out model variants. Each row contains 25 variants per base classifier. All families except new initialization reuse the base model’s initialization seed, so they isolate the data or training change from initialization noise.

Figure 1: Empirical robustness across held-out model changes. Panels correspond to datasets, rows to CFE methods, and columns to the eight change families. Each cell aggregates 25 variants for each of five independently trained base models and measures changed-model validity among base-valid CFEs.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$n$	$p$	Split sizes	Factuals	Test balanced accuracy
Breast Cancer	569	30	284/57/57/171	312 (60–64)	0.967/0.973/0.988
Diabetes	768	8	384/76/77/231	281 (50–61)	0.712/0.721/0.736
Wine Quality	6,497	11	3248/649/650/1950	1,250 (250)	0.735/0.741/0.749
HELOC	8,291	20	4145/829/829/2488	1,250 (250)	0.727/0.730/0.735

Appendix

Table 3: Dataset and base-model summary.

n

and

p

denote the numbers of observations and input features. Split sizes are reported as training/update/validation/test. “Factuals” gives the total number of adverse test predictions evaluated across the five base models, followed by the per-model range in parentheses. Test balanced accuracy is reported as minimum/mean/maximum across the five models.

Change family	Hard disagreement (%)	Probability MAE
Bootstrap	4.92 $\pm$ 1.25	0.039 $\pm$ 0.010
Architecture	4.69 $\pm$ 1.54	0.038 $\pm$ 0.017
New initialization	4.57 $\pm$ 0.93	0.036 $\pm$ 0.005
Label update	3.69 $\pm$ 1.99	0.045 $\pm$ 0.025
Training configuration	3.10 $\pm$ 1.99	0.028 $\pm$ 0.019
Data addition	2.18 $\pm$ 0.96	0.017 $\pm$ 0.007

Appendix

Table 4: Output difference from the base classifier: mean

\pm

standard deviation over the 125 variants per dataset, macro-averaged across datasets.

Method	Mean $\pm$ SD	Median	90th percentile	Timeouts
AP $\Delta$ S	3.909 $\pm$ 4.643	2.612	7.542	43
BetaRCE	0.0198 $\pm$ 0.0134	0.0159	0.0359	0
KD-tree	0.0037 $\pm$ 0.0022	0.0042	0.0050	0
RBR	1.754 $\pm$ 0.909	1.860	2.900	0
RNCE	0.262 $\pm$ 4.200	0.0007	0.0008	0
ROAR-LIME	0.086 $\pm$ 0.060	0.074	0.151	0

Appendix

Table 5: Per-factual CFE generation time in seconds, reported as mean

\pm

standard deviation across requests. AP

\Delta

S timeouts are unsuccessful requests that reach its 30-second limit.

Method	Coverage $\uparrow$	Base valid $\uparrow$	Emp. robust. $\uparrow$	E2E robust. $\uparrow$	Distance $\downarrow$
Breast Cancer
AP $\Delta$ S	90.4 $\pm$ 19.8	100.0 $\pm$ 0.0	58.5 $\pm$ 3.6	53.1 $\pm$ 12.8	0.689 $\pm$ 0.201
BetaRCE	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	97.0 $\pm$ 0.8	97.0 $\pm$ 0.8	1.860 $\pm$ 0.052
KD-tree	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	94.8 $\pm$ 2.4	94.8 $\pm$ 2.4	1.554 $\pm$ 0.006
RBR	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	77.1 $\pm$ 2.4	77.1 $\pm$ 2.4	1.097 $\pm$ 0.022
RNCE	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	98.5 $\pm$ 0.8	98.5 $\pm$ 0.8	1.571 $\pm$ 0.010

Appendix

Table 6: Full five-seed results, reported as mean

\pm

standard deviation over the five base models.

Explore similar work

A Probabilistic Consensus-Driven Approach for Robust Counterfactual Explanations
Apr 19, 2026Marcin Kostrzewa, Maciej Zięba, Jerzy StefanowskiCounterfactual Explanation Explainability
Wrocław University of Science and Technology, Wrocław 50-347, Poland · Tooploox, Wrocław 53-601, Poland · Poznań University of Technology, Poznań 61-131, Poland
Counterfactual Marginalisation: Framework for Evaluating Robustness to Nuisance Variables
Sep 12, 2026Yasin Ibrahim, Hermione Warr, Robin J. Evans +1Minority Worst-Case
Department of Engineering Science, University of Oxford, Oxford, UK · Department of Statistics, University of Oxford, Oxford, UK
CEL: Comprehensive Counterfactual Explanations Library and Benchmark
Jul 24, 2026Oleksii Furman, Łukasz Lenkiewicz, Marcel Musiałek +1Counterfactual Explanation Distributional Information
Wrocław University of Science and Technology, Wrocław, Poland

Apr 19, 2026, cs.LGJ/K move · Enter open · S save

A Probabilistic Consensus-Driven Approach for Robust Counterfactual Explanations

Marcin Kostrzewa, Maciej Zięba, Jerzy Stefanowski

Wrocław University of Science and Technology, Wrocław 50-347, Poland · Tooploox, Wrocław 53-601, Poland · Poznań University of Technology, Poznań 61-131, Poland

Counterfactual Explanation Explainability Generative Models Normalizing Flows

Open paper AskPDF arXiv

Counterfactual explanations (CFEs) are essential for interpreting black-box models, yet they often become invalid when models are slightly changed. Existing methods for generating robust CFEs are often limited to specific types of models, require costly tuning, or inflexible robustness controls. We propose a novel approach that jointly models the data distribution and the space of plausible model decisions to ensure robustness to model changes. Using a probabilistic consensus over a model ensemble, we train a conditional normalizing flow that captures the data density under varying levels of classifier agreement. At inference time, a single interpretable parameter controls the robustness level; it specifies the minimum fraction of models that should agree on the target class without retraining the generative model. Our method effectively pushes CFEs toward regions that are both plausible and stable across model changes. Experimental results demonstrate that our approach achieves superior empirical robustness while also maintaining good performance across other evaluation measures.

Abstract

Explore similar work

A Probabilistic Consensus-Driven Approach for Robust Counterfactual Explanations

Counterfactual Marginalisation: Framework for Evaluating Robustness to Nuisance Variables

CEL: Comprehensive Counterfactual Explanations Library and Benchmark