"Where Can I Trust You?": Boundary-Aware Evaluation of Surrogate Fidelity

Jackson Eshbaugh

cs.LGSep 26, 2026

"Where Can I Trust You?": Boundary-Aware Evaluation of Surrogate Fidelity

Authors: Jackson Eshbaugh

Organizations: Department of Computer Science Lafayette College Easton, PA, USA

Abstract

Surrogate models are commonly evaluated by how often they agree with their teacher model over an evaluation set. Local variation in this agreement is well known, but its structure and consequences are less clear. We ask whether disagreement is systematically concentrated near the teacher's decision boundary and whether retaining that structure provides information beyond a single global score. Across several datasets and surrogate model classes, we find substantially lower fidelity near teacher decision boundaries under two different methods of identifying near-boundary examples. Moreover, conditioning agreement on confidence-defined regions improves prediction of teacher--surrogate agreement when evaluation-set composition changes, relative to the global score alone. Yet surrogates that agree equally well with the teacher both globally and near the decision boundary can respond very differently to changes selected using the surrogate itself. Finally, we show that independently trained deep teachers can agree on most predictions while identifying different examples as lying near their decision boundaries, complicating the use of those boundaries as stable reference regions for evaluating surrogates. Together, these results show that surrogate fidelity depends not only on how often a surrogate agrees with its teacher, but also on where that agreement holds and, for deep models, how stable the teacher's decision boundary is across training runs.

Figures & tables

Figure 1: Global-to-boundary fidelity gaps, in percentage points, as the boundary fraction varies under confidence-based and approximate segment-crossing measures. Points show mean gaps across random seeds after averaging over decision-tree depths within each seed; error bars show approximate 95% confidence intervals across seeds.

\toprule		Global fidelity (%)	Confidence-based boundary		Segment-crossing boundary
\cmidrule (lr)4-5 \cmidrule (lr)6-7 Dataset	Surrogate	Global fidelity (%)	Gap (pp) mean $\pm$ 95% CI	Pos.	Gap (pp) mean $\pm$ 95% CI	Pos.
\midrule Breast Cancer	Decision tree	94.0	11.9 $\pm$ 2.2	10/10	10.1 $\pm$ 2.8	10/10
	Logistic regression	98.2	5.2 $\pm$ 1.6	10/10	5.2 $\pm$ 1.6	10/10
	$k$ -NN	96.4	10.6 $\pm$ 3.4	10/10	9.9 $\pm$ 3.6	10/10
	MLP	98.8	3.4 $\pm$ 1.7	8/10	3.2 $\pm$ 1.6	8/10
\midrule Diabetes	Decision tree	73.6	18.6 $\pm$ 4.2	10/10	13.9 $\pm$ 4.6	9/10

Table 1: Boundary-local fidelity degradation across surrogate families at

\alpha=0.25

. Global fidelity and boundary gaps are averaged over ten seeds; gaps are reported in percentage points with approximate 95% confidence intervals. “Pos.” counts seeds for which global fidelity exceeds boundary-local fidelity.

Figure 2: Confidence-stratified fidelity improves prediction when evaluation composition changes. Mean absolute error in predicting target-set fidelity under shifts toward lower- and higher-confidence regions. Predictions using a three-bin confidence-stratified fidelity profile are compared with a single global-fidelity baseline. Error bars indicate 95% confidence intervals across random seeds, after averaging over surrogate depths within each seed.

Figure 3: Similar observational agreement can correspond to substantial differences in counterfactual transfer. Each point represents one surrogate configuration, with global fidelity on the $x$ -axis and counterfactual transfer on the $y$ -axis. The vertical spread at similar global fidelity indicates that aggregate agreement does not uniquely determine counterfactual behavior. The highlighted Breast Cancer pair has identical global and boundary-local fidelity, but counterfactual transfer rates of 73.1% and 24.1%.

Figure 4: Boundary regions are only partially stable across independently trained deep teachers. Left: mean Jaccard overlap between teachers’ lowest-margin examples as the boundary fraction varies. Right: the drop from global pairwise predictive agreement to agreement over the union of the two teachers’ boundary regions. Results are shown for independently trained teachers on MNIST and CIFAR-10.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

\toprule Dataset	Examples	Features	Binary task
\midrule Breast Cancer Wisconsin [ 21 ]	569	30	Original binary labels
Wine [ 1 ]	178	13	Class 0 vs. remaining classes
Iris [ 6 ]	150	4	Class 0 vs. remaining classes
Diabetes [ 4 ]	442	10	Target $\geqslant$ median vs. target $<$ median
\bottomrule

Appendix

Table 2: Tabular datasets used in the experiments and the binary classification tasks constructed from them. Wine and Iris are converted to one-vs-rest tasks using class 0 as the positive class. The Diabetes target is binarized at its median.

\toprule Surrogate family	Configuration
\midrule Decision tree	maximum depth $5$
Logistic regression	$C=1$ , liblinear solver
$k$ -NN	$k=5$
MLP	one hidden layer, width $32$ , ReLU, lbfgs
\bottomrule

Appendix

Table 3: Representative surrogate configurations used in the cross-family robustness analyses.

\toprule Dataset	Depth	Global fidelity (%)	Confidence gap (pp)	Segment-crossing gap (pp)
\midrule Breast Cancer	1	91.6	12.3 $\pm$ 3.2	13.7 $\pm$ 2.5
	2	93.3	11.2 $\pm$ 3.0	10.8 $\pm$ 3.9
	3	93.4	11.8 $\pm$ 2.4	11.8 $\pm$ 3.0
	5	94.0	11.9 $\pm$ 2.2	10.1 $\pm$ 2.8
	7	94.1	12.9 $\pm$ 2.2	10.6 $\pm$ 2.4
	10	94.2	12.5 $\pm$ 2.0	10.9 $\pm$ 2.6

Appendix

Table 4: Decision-tree capacity sweep at boundary fraction

\alpha=0.25

. Global fidelity and boundary gaps are averaged over ten random seeds. Boundary gaps are defined as

F_{\mathrm{global}}-F_{\mathrm{boundary}}

; gap values are shown as means with approximate 95% confidence intervals.

Figure 5: Surrogate-family robustness across boundary fractions. Global-to-boundary fidelity gaps are shown for representative decision-tree, logistic-regression, $k$ -NN, and MLP surrogates under confidence-based (top) and approximate segment-crossing (bottom) measures of boundary proximity. Points show means over ten random seeds; error bars indicate approximate 95% confidence intervals.

\toprule Dataset	Global baseline (pp)	3-bin profile (pp)	5-bin profile (pp)
\midrule Breast Cancer	4.30	2.93	2.92
Diabetes	13.62	5.92	6.04
Wine	5.09	5.02	6.10
\midrule Overall	7.67	4.62	5.02
\bottomrule

Appendix

Table 5: Sensitivity of target-fidelity prediction to the number of confidence bins. Values are mean absolute prediction errors across surrogate depths, random seeds, and target compositions. The global-fidelity baseline is unchanged by the number of profile bins.

\toprule Scope	Surrogate	Global baseline (pp)	3-bin profile (pp)	Improvement (pp)
\midrule Overall	Decision tree	8.00	5.06	2.95
	Logistic	6.78	2.96	3.82
	$k$ -NN	7.26	4.07	3.19
	MLP	5.13	2.81	2.32
\addlinespace Breast Cancer	Decision tree	4.56	3.63	0.93
	Logistic	1.93	1.44	0.49

Appendix

Table 6: Fidelity-prediction robustness across surrogate families. Values are mean absolute target-fidelity prediction errors. “Improvement” is computed before rounding as global baseline error minus profile error, so positive values favor the confidence-stratified profile.

\toprule Dataset	Depth	Global fidelity (%)	Boundary fidelity (%)	Transfer rate (%)
\midrule Breast Cancer	2	92.4 $\pm$ 2.1	79.7 $\pm$ 4.1	50.2 $\pm$ 12.5
	3	93.8 $\pm$ 1.7	83.1 $\pm$ 3.8	51.9 $\pm$ 11.1
	5	94.6 $\pm$ 1.5	84.8 $\pm$ 4.6	49.0 $\pm$ 14.2
	7	94.3 $\pm$ 1.7	84.5 $\pm$ 3.9	45.3 $\pm$ 13.5
	10	94.5 $\pm$ 1.7	84.5 $\pm$ 4.4	44.4 $\pm$ 13.9
\addlinespace Diabetes	2	73.9 $\pm$ 2.5	48.2 $\pm$ 5.5	50.2 $\pm$ 11.1

Appendix

Table 7: Counterfactual-transfer results across decision-tree capacities. Fidelity and transfer rates are reported as means across ten random seeds, with

\pm

denoting approximate 95% confidence-interval half-widths. Boundary-local fidelity uses

\alpha=0.25

\toprule Dataset	Seed	Depths	Global fidelity (%)	Boundary fidelity (%)	Transfer rates (%)	Gap (pp)
\midrule Breast Cancer	3	3 / 7	94.7 / 94.7	82.8 / 82.8	73.1 / 24.1	49.1
Diabetes	7	5 / 10	72.7 / 72.7	54.5 / 54.5	56.2 / 37.5	18.8
Wine	4	3 / 10	100.0 / 100.0	100.0 / 100.0	100.0 / 61.1	38.9
\bottomrule

Appendix

Table 8: Matched-fidelity surrogate pairs with differing counterfactual-transfer behavior. Each row compares two decision-tree surrogates trained under the same dataset and random seed. The transfer gap is the difference between the two transfer rates in percentage points.

\toprule Dataset	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4
\midrule MNIST	99.09	99.27	99.33	99.18	99.39
CIFAR-10	95.30	95.27	95.35	95.42	95.63
\bottomrule

Appendix

Table 9: Test accuracy (%) of independently trained deep teachers. Each teacher is evaluated on the common 10,000-example test set for its dataset.

\toprule Dataset	$\alpha$	Global agr. (%)	Jaccard (%)	Union agr. (%)	Intersection agr. (%)	Global $-$ union (pp)
\midrule MNIST	0.10	99.18	39.39	94.31	86.63	4.87
	0.20	99.18	43.13	97.08	93.32	2.10
	0.25	99.18	45.01	97.63	94.77	1.55
\addlinespace CIFAR-10	0.10	96.05	44.32	72.09	58.39	23.96
	0.20	96.05	56.81	84.50	74.63	11.54
	0.25	96.05	58.53	87.47	79.50	8.58

Appendix

Table 10: Pairwise stability of independently trained deep teachers. Values are means across the ten unique teacher pairs for each dataset.

Decision Boundary Analysis

See all 8 papers

Bound to Disagree: Generalization Bounds via Certifiable Surrogates
Feb 26, 2026Mathieu Bazinet, Valentina Zantedeschi, Pascal GermainGeneralization Bounds
1Département d’informatique et génie logiciel, Université Laval, Québec, Qc, Canada · 2ServiceNow Research, Montreal, Qc, Canada
When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design
Aug 2, 2026Shuangxiu, Ma, Wenhe +1Surrogate Modeling Surrogate-Assisted Optimization
Max · William G. Lowrie Department of Chemical and Biomolecular Engineering, The Ohio State University, Columbus, Ohio 43210, United States · Zachary +1
Inferential Evaluation of Surrogate-Derived Models under Covariate Shift
Aug 16, 2026Longtian Shi, Molei Liu, Doudou ZhouDistribution Shift Robustness Covariate Shift
Department of Statistics and Data Science, National University of Singapore · Department of Biostatistics, Peking University Health Science Center, Peking University, Beijing, China · Beijing International Center for Mathematical Research, Peking University, Beijing, China

Feb 26, 2026J/K move · Enter open · S save

Bound to Disagree: Generalization Bounds via Certifiable Surrogates

Mathieu Bazinet, Valentina Zantedeschi, Pascal Germain

1Département d’informatique et génie logiciel, Université Laval, Québec, Qc, Canada · 2ServiceNow Research, Montreal, Qc, Canada

Generalization Bounds

Open paper AskPDF arXiv

Generalization bounds for deep learning models are typically vacuous, not computable or restricted to specific model classes. In this paper, we tackle these issues by providing new disagreement-based certificates for the gap between the true risk of any two predictors. We then bound the true risk of the predictor of interest via a surrogate model that enjoys tight generalization guarantees, and by evaluating our disagreement bound on an unlabeled dataset.We empirically demonstrate the tightness of the obtained certificates and showcase the versatility of the approach by training surrogate models leveraging three different frameworks: sample compression, model compression and PAC-Bayes theory. Importantly, such guarantees are achieved without modifying the target model, nor adapting the training procedure to the generalization framework.

Abstract

Decision Boundary Analysis

Explore similar work

Bound to Disagree: Generalization Bounds via Certifiable Surrogates

When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design

Inferential Evaluation of Surrogate-Derived Models under Covariate Shift