BRIDGE: An Acceptance Gate to Block False Improvements in Model Updates, with a Retrospective Clinical Evaluation

Dwi Sunaryono

cs.LGOct 21, 2026

BRIDGE: An Acceptance Gate to Block False Improvements in Model Updates, with a Retrospective Clinical Evaluation

Authors: Dwi Sunaryono

Organizations: Department of Informatics, Faculty of Intelligent Electrical and Informatics Technology, Institut Teknologi Sepuluh Nopember, Surabaya 60111, Indonesia

Abstract

Models are updated repeatedly, and an update that looks better on validation data but not on new data is a false improvement that can misguide clinical decisions. The naive rule accepts every positive validation gain and is optimistic when the same data choose and judge the update. BRIDGE is a model-agnostic acceptance gate that decides each update from predictions and the available labels, with fixed settings. Each fold of the data judges the update chosen on the other folds (cross-fitting) to reduce selection bias; acceptance requires the gain's lower bootstrap bound to exceed a margin. The four settings are (A) 11 new datasets, (B) synthetic tasks, (C) 12 retrospective clinical tasks, and (D) 39 supporting datasets. The false-improvement rate falls from 19.1% (naive) to 1.1% (BRIDGE) in (A), 20.7% to 1.7% (B), 20.0% to 0% (C), and 12.5% to 0.5% (D); every 95% confidence interval of the reduction over datasets or tasks excludes zero (not over the clinical sources). An equal-size random draw from the naive acceptances contains more false improvements in (A)-(E) (post hoc, p<0.001). On the OpenML collection (E) of 14 new datasets, the false-improvement rate falls from 25.6% to 1.0%. BRIDGE removes 97% to 100% of the false improvements that the naive rule accepts, and its mean gain per deployed update is 1.8 to 3.1 times the naive rule's. The 95% upper bound on the false-improvement rate is below 5.5%: formal in (B) for independent tasks (3.56%), and conditional on the real data tested (0.33% to 5.40%).

Figures & tables

Figure 1: False improvements accepted by the naive rule, grouped by validation gain, in collections (A)–(E).

Method	What it controls	Accounts for selection	Labels needed at decision	Bound covers
Nested CV and bias corrections [ 13 , 14 , 35 ]	error-estimate optimism	yes (inside estimate)	existing labels only	selection procedure
McNemar / $5\times 2$ cv / corrected $t$ [ 21 , 22 , 23 ]	fixed-pair difference	no	existing labels only	prespecified comparison
Learn then Test [ 30 ]	policy risk	yes	reserved calibration set	selected policy
Update approval [ 19 , 24 ]	bio-creep	yes (within policy)	labeled deployment stream	approval policy
No-regression certificates [ 26 , 27 ]	per-slice regression	yes (within slices)	certification set (Zhang); disagreements only (Balachandran)	deployed model
BRIDGE	false-improvement rate among accepted updates	yes (per-fold separation)	existing training labels only	acceptance procedure, population-level

Table 1: Positioning of BRIDGE against related gates and methods.

Parameter	Value
Cross-fit folds $F$	$3$
Bootstrap replicates $N_{\mathrm{boot}}$	$20{,}000$
Lower quantile $q$	$5\%$
Margin $\varepsilon$	$0.005$
Base learner	HGB, library defaults with at most 300 iterations and early stopping
Bagged base models	$32$ bags of $80\%$ of SEARCH

Table 2: Frozen parameters of BRIDGE and of the evaluation protocol.

Symbol / term	Definition
Task	One dataset, or one draw from the synthetic population $\mathcal{Q}$ .
Trial	One re-randomization of the training rows, the SEARCH/POOL split, and the bag draw under a fresh seed; produces one decision; trials per task: $100$ (real), $200$ (clinical), $60$ (multiclass), $1$ ( $\mathcal{Q}$ ).
Decision; index $j$	Accept or reject in one trial; $j$ runs over trials on real data and over tasks on $\mathcal{Q}$ ; $J$ is the number of trials or tasks.
Source	Clinical: data provider (six sources; the four National Health and Nutrition Examination Survey (NHANES) tasks form one source, the four Behavioral Risk Factor Surveillance System (BRFSS) tasks another). Non-clinical: the dataset.
Locked reference $R_{j}$	Rows set aside once per task with a pre-registered seed; used only as the ground truth to judge the deployed update. Size: Appendix B .
TRAIN $=$ SEARCH $\sqcup$ POOL	SEARCH fits the base models; POOL holds the evidence for selection and gating. Size: Appendix B .

Table 3: Data units, partition, and predictors.

Figure 2: Two-level separation: outer train/reference split, inner three-fold cross-fit within POOL; colors mark subset roles.

Symbol / term	Definition
$n_{y,\tilde{b},\tilde{c}}$	Cross-fitted triple counts: label, baseline prediction, and candidate prediction on each held-out fold, pooled over the $F=3$ folds.
$\Delta$ , $\Delta^{*}$	Metric difference (candidate minus baseline) from the pooled counts; $\Delta^{*}$ is one of $N_{\mathrm{boot}}=20{,}000$ multinomial bootstrap draws.
Accept rule	Accept if and only if $Q_{0.05}(\Delta^{*})>\varepsilon$ ; otherwise retain $b$ .
Margin $\varepsilon$	$0.005$ on the scale of the selected metric, not rescaled; for net benefit ( 10 ), five additional net true positives per $1{,}000$ examples.
$\mathcal{M}$	Any higher-is-better metric computed from a confusion table (weighted $F_{1}$ , balanced accuracy, macro- $F_{1}$ , $F_{2}$ , and, in the binary clinical protocol, net benefit); $\mathcal{M}$ is the metric, distinct from the accepted count $M=\sum_{j}A_{j}$ (Section III-D ).

Table 4: Quantities of the bootstrap gate.

Symbol / term	Definition
$\Delta_{\mathrm{ref},j}$	Metric difference between the deployed configuration $c_{j}$ and $b_{j}$ on the locked reference.
False improvement, $H_{j}$	$H_{j}=\mathbf{1}\{\Delta_{\mathrm{ref},j}\leq 0\}$ : the deployed configuration does not improve the selected metric on the locked reference; ties count as false improvements.
Clearly worse / reference-uncertain	From the $95\%$ Bayesian-bootstrap interval of $\Delta_{\mathrm{ref},j}$ [ 66 ] : a Dirichlet posterior over the confusion cells with pseudocount $0.5$ (the Jeffreys prior) on the binary and clinical protocols and over the positive cells on multiclass, $20{,}000$ draws, quantiles $2.5\%$ and $97.5\%$ ; entirely below $0$ = clearly worse; entirely above $0$ = clearly improving; otherwise reference-uncertain.
$A_{j}$ ; false-improvement rate $\hat{r}$	$A_{j}=1$ if decision $j$ accepts; $\hat{r}=\sum_{j}A_{j}H_{j}/\sum_{j}A_{j}$ , defined when $\sum_{j}A_{j}>0$ .
Yield; total gain	Accepted updates per $100$ decisions; $\sum_{j}A_{j}\Delta_{\mathrm{ref},j}$ .
$r_{\mathcal{Q}}$	False-improvement rate among accepted updates, $\mathbb{E}[A_{j}H_{j}]/\mathbb{E}[A_{j}]$ , on the i.i.d. task population $\mathcal{Q}$ .

Table 5: Outcome and risk quantities.

Symbol / term	Definition
Naive rule	Accept whenever the configuration selected on the whole of POOL improves the validation metric, i.e., ( 1 ) with a positive gain.
McNemar	Exact one-sided binomial test on discordant pairs (candidate correct and baseline wrong, or the reverse) on POOL; tests per-example accuracy.
$5\times 2$ cv	Dietterich’s paired $t$ statistic, $5$ d.f., on the fixed predictions of the final configuration over five $2$ -fold splits of POOL, without refitting.
Corrected $t$	Ten $2/3$ – $1/3$ splits of POOL, Nadeau–Bengio variance correction, $9$ d.f., final configuration; with re-selection: the candidate is re-chosen inside each split.
Fixed margin $t$	Accept when the validation gain on POOL is at least $t$ ; $t^{*}=0.05$ pre-registered from development, other $t$ post hoc.
Random control	Accept at random with BRIDGE’s yield.

Table 6: Comparator rules and the pre-registered criterion.

Setting	Naive false % $\downarrow$	BRIDGE false % $\downarrow$	Reduction (pp) $\uparrow$	False removed % $\uparrow$	Gain per update $\uparrow$
(A) New datasets	19.1	1.1	18.0	99	2.5 $\times$
(B) Synthetic	20.7	1.7	19.0	97	2.1 $\times$
(C) Clinical	20.0	0.0	20.0	100	1.8 $\times$
(D) Supporting	12.5	0.5	12.0	99	2.1 $\times$
(E) OpenML †	25.6	1.0	24.6	99	3.1 $\times$

Table 7: Summary of main results: the naive rule against BRIDGE, (A)–(E).

No.	Test ( $H_{0}$ )	Result	Decision on $H_{0}$ ( $\alpha=0.05$ ) and meaning
1	T1 ( 14 )	BRIDGE lower by $12.0$ to $24.6$ pp ( $p<0.04$ )	Rejected in (A)–(E): BRIDGE accepts fewer false improvements than the naive rule in every collection
2	T1 ( 14 )	BRIDGE lower by $2.0$ to $33.1$ pp wherever $H_{0}$ is rejected	Rejected in (A), (B), (D), and against the $5\times 2$ cv test in (C): BRIDGE accepts fewer false improvements
3	T1 ( 14 )	These rules have $0.0\%$ to $3.0\%$ false improvements; BRIDGE is $1.0$ pp lower than the re-selection test in (D)	Not rejected (except that test in (D)): these rules share the principle of BRIDGE, separating the data that choose from the data that judge; BRIDGE accepts $1.1$ to $1.9$ times as many updates as the holdouts (Section IV-C )
4	T1 ( 14 )	BRIDGE lower by $1.1$ to $1.8$ pp in (A), (B), (D); no difference in (C)	Rejected in (A), (B), (D): the margin lowers the false-improvement rate; in (C) both versions accept almost no false improvement
5	T2 ( 15 )	All $12$ comparisons within $\pm 2$ pp ( $p\leq 0.0498$ )	Rejected in (A)–(D): rejecting this $H_{0}$ means that the margins are equivalent, so the false-improvement rate is stable across margins in this range
6	T3 ( 16 ), post hoc	Random draw $49$ to $240$ false improvements, BRIDGE $0$ to $8$ ( $p<10^{-23}$ )	Rejected in (A)–(E): BRIDGE picks fewer false improvements than a random choice, also within single datasets in (D) and (E)

Table 8: Hypothesis tests: results and decisions on questions 1–6.

Rule	(A)	(B)	(C)	(D)	(E)
Naive (validation improves)	19.1 ∗ $\times$	20.7 ∗ $\times$	20.0 ∗ $\times$	12.5 ∗ $\times$	25.6 ∗ $\times$
McNemar, same POOL	12.0 ∗ $\times$	34.8 ∗ $\times$	0.0 ∘ $\times$	6.7 ∗ $\times$	–
McNemar, cross-fit	8.9 $\times$	26.0 $\times$	0.0 ∘ $\times$	2.5 ∗ $\checkmark$	–
$5\times 2$ cv paired $t$	6.3 ∗ $\times$	5.6 ∗ $\checkmark$	2.5 ∗ $\checkmark$	3.6 ∗ $\checkmark$	–
Corrected resampled $t$	6.3 ∗ $\times$	5.7 ∗ $\checkmark$	2.6 ∘ $\checkmark$	4.2 ∗ $\checkmark$	–
Corrected $t$ , re-selection	2.3 ∘ $\checkmark$	3.0 ∘ $\checkmark$	0.4 ∘ $\checkmark$	1.5 ∗ $\checkmark$	–

Table 9: Acceptance rules on collections (A)–(E): false-improvement rate of the accepted updates (%).

	(A) 11 new datasets			(B) Synthetic			(C) Clinical			(D) Supporting
Rule	False % $\downarrow$	Yield $\uparrow$	${>}\varepsilon$ $\uparrow$	False % $\downarrow$	Yield $\uparrow$	${>}\varepsilon$ $\uparrow$	False % $\downarrow$	Yield $\uparrow$	${>}\varepsilon$ $\uparrow$	False % $\downarrow$	Yield $\uparrow$	${>}\varepsilon$ $\uparrow$
Corrected $t$ , re-selection	2.3 ∘ $\checkmark$	31.7	29.8	3.0 ∘ $\checkmark$	37.9	35.9	0.4 ∘ $\checkmark$	58.4	56.4	1.5 ∗ $\checkmark$	39.6	35.2
Holdout 2/3 + 1/3	1.3	13.9	13.4	1.9 $\checkmark$	18.6	18.0	0.0 ∘ $\checkmark$	41.1	41.0	0.3 ∘ $\checkmark$	19.5	18.8
Holdout 1/2 + 1/2	3.2	17.3	16.2	2.1 $\checkmark$	24.2	23.3	0.0 ∘ $\checkmark$	46.0	46.0	0.8 ∘ $\checkmark$	22.3	21.2
Fixed margin, dataset held out †	5.8 $\times$	31.5	28.7	4.2 $\checkmark$	49.4	45.4	–	–	–	–	–	–
BRIDGE	1.1 $\checkmark$	25.8	24.9	1.7 $\checkmark$	33.9	32.8	0.0 $\checkmark$	50.0	49.9	0.5 $\checkmark$	30.1	29.1

Table 10: Strongest comparators: false-improvement rate, yield, and updates above the margin, (A)–(D).

Figure 3: Acceptance rules compared at matched acceptance counts on (A) and (B).

Rule	Accepted	False	Bound % $\downarrow$	Yield/100 $\uparrow$	Criterion
Naive (validation improves)	1,323	274	23.52	94.50	$\times$
McNemar on same POOL	135	47	45.19	9.64	$\times$
McNemar on cross-fit	50	13	42.65	3.57	$\times$
$5\times 2$ cv paired $t$	610	34	8.14	43.57	$\checkmark$
Corrected resampled $t$	774	44	7.95	55.29	$\checkmark$
Corrected $t$ , re-selection	531	16	5.19	37.93	$\checkmark$

Table 11: Finite-sample bound on the synthetic tasks (B) for each acceptance rule.

Co-primary (pre-registered)
Metric	Naive false impr. % $\downarrow$	BRIDGE accepted	BRIDGE false impr. % $\downarrow$	Bound % $\downarrow$	LOCO- source % $\downarrow$
Net benefit, $p_{t}=0.2$	20.0	1,199	0.0	0.33	0.50
Balanced accuracy	0.3	2,273	0.4	0.82	1.25
Secondary
Weighted $F_{1}$	23.6	435	2.1	5.12	7.84
$F_{2}$	0.0	2,379	0.0	0.16	0.24

Table 12: Clinical application (C): results by metric.

Known as	Cross-fit	Bootstrap bound	Margin	(A)		(B)		(C)		(D)		(E)
Known as	Cross-fit	Bootstrap bound	Margin	False % $\downarrow$	Bound % $\downarrow$	False % $\downarrow$	Bound % $\downarrow$	False % $\downarrow$	Bound % $\downarrow$	False % $\downarrow$	Bound % $\downarrow$	False % $\downarrow$	Bound % $\downarrow$
Naive	–	–	–	19.08	22.90 $\times$	20.71	23.52 $\times$	19.99	22.68 $\times$	12.49	14.19 $\times$	27.65	32.13 $\times$
Fixed margin	–	–	$\checkmark$	10.20	14.17 $\times$	16.42	19.19 $\times$	2.65	4.05 $\checkmark$	6.18	7.68 $\checkmark$	16.74	22.48 $\times$
Same-POOL bootstrap	–	$\checkmark$	–	6.96	10.70 $\times$	6.61	8.95 $\checkmark$	2.67	4.06 $\checkmark$	4.87	6.33 $\checkmark$	13.73	19.16 $\times$
Same-POOL cell	–	$\checkmark$	$\checkmark$	2.65	5.85 $\checkmark$	3.18	5.14 $\checkmark$	0.00	0.31 $\checkmark$	1.08	2.06 $\checkmark$	3.37	8.17 $\times^{\ddagger}$
Cross-fit, point	$\checkmark$	–	–	10.81	15.00 $\times$	12.07	14.72 $\times$	7.69	9.84 $\times$	7.28	8.89 $\checkmark$	18.98	24.82 $\times$

Table 13: Component analysis on (A)–(E): ablation of the three components of the gate.

Figure 4: Component analysis: naive and cross-fitted gain estimates against the reference truth on (A).

Figure 5: Component analysis: false-improvement rate against candidate-space size for (A)–(E).

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Source	N	Min. %	Ref.
(A) 11 new datasets
pc3	Software	OpenML 1050	1,563 †	10.24	468
MIC	Medical	OpenML 45648	1,649 †	15.59	494
Marketing_Campaign	Marketing	OpenML 46940	2,240 †	14.91	672
Pumpkin_Seeds	Agriculture	OpenML 46951	2,500 †	48.00	750
seismic-bumps	Mining	OpenML 46956	2,584 †	6.58	775

Appendix

Table B1: Data partition for each dataset or task in (A) and (C)–(E).

Task	Population	Outcome	Predictors / timing	Hypothetical action
mammography	Screening mammograms (OpenML 310)	Malignant calcification	Image-derived features at reading	Recall for biopsy
actg175	HIV-1 trial participants (UCI 890)	Recorded event: CD4 decline $\geq 50\%$ , AIDS, or death	Baseline covariates only	Intensify monitoring
nhanes_*	US adults $\geq$ 20 y, NHANES 2017–18	Self-reported: hypertension, diabetes, asthma, CHD	Exam & questionnaire, same visit	Refer for testing
brfss_*	US adults, BRFSS 2015	Self-reported: stroke, asthma, kidney disease, depression	Survey responses, same interview	Refer for screening
covid_mx	Confirmed COVID-19 patients, Mexico	In-hospital/registered death	Admission data (comorbidities, triage)	Escalate care
wids_icu	ICU admissions, GOSSIS (WiDS 2020)	Hospital death	First-24 h ICU measurements	Escalate care

Appendix

Table B2: Clinical tasks.

MINI-02 $\cdot$ 34 (D) $\times$ 100
Check	Acc./ false	Bound % $\downarrow$	LOCO % $\downarrow$	Yield
frozen gate, threshold-only HGB	586 / 1	1.24	1.45	17.2 a
MINI-04 $\cdot$ 34 (D) $\times$ 100
threshold-only KNN, cross-fit gate	150 / 7	14.04	19.95	4.4
MINI-09 $\cdot$ 18 dev. $\times$ 100
third member KNN	218 / 3	5.71	8.70	12.1

Appendix

Table C1: Runs cited in the text, by run number.

Run	Fixed beforehand	End.	Outcome
01–19	mechanism studies; thresholded HGB, HGB–DT ensembles, the $173$ space	Beh	validation–test independence hypothesis not supported
04	threshold-only KNN with the cross-fit gate	Crt	fails (bound $14.0\%$ )
09	third ensemble member: KNN or SVC	Yld	SVC chosen: higher yield at a comparable false-improvement rate
12, 14	Optuna-tuned HGB on SEARCH	Dsc	small gains; clearly worse in up to $12\%$ of trials, so the fixed configuration is frozen
17, 20	$1{,}000$ / $2{,}430$ / $6{,}000$ training rows (17 dev.); fresh seeds	Crt	pass (bound $3.04$ / $1.05$ / $1.37\%$ )
21	$11$ new binary datasets, weighted- $F_{1}$	Crt	pass (bound $4.23\%$ , LOCO $5.95\%$ )

Appendix

Table C2: Chronology of the pre-registered runs, ordered by run number.

Choice	Made with	Assessed on
Model type and hyperparameters	fixed a priori , no tuning (rationale: Section III-A )	—
Candidate space ( $173$ binary default) and gate parameters	$18$ development datasets, frozen	families of new data never seen in development (new binary, new multiclass, clinical, fresh synthetic); the $34$ confirmation datasets of (D) with fresh seeds
Preprocessing (imputation, scaling, encoding)	fitted per bag, SEARCH only; WiDS category codes are listed from all rows before the split (covariates only, no labels)	POOL and reference
Base models (up to $32$ bag draws, kept bags averaged); early stopping	SEARCH (internal bag validation)	POOL and reference
Candidate selection inside the gate	$2/3$ of POOL	the held-out $1/3$ of POOL
Accept / reject	lower bootstrap quantile of the cross-fit counts	—

Appendix

Table D1: What each choice is made with, and where it is assessed.

Parameter	Value	Set	Tested in
Cross-fit folds $F$	$3$	Dev	$F\in\{2,3,5\}$ on $\mathcal{Q}$ (Table D7 )
Replicates $N_{\mathrm{boot}}$	$20{,}000$ , multinomial over the triple counts	Dev	MC error, $20$ seeds; $2{,}000$ (Table C1 )
Lower quantile $q$	$5\%$ , decision statistic (not the error level $\delta=0.05$ )	Dev	$2.5$ / $5$ / $10\%$ (Tables D7 , D5 )
Margin $\varepsilon$	$0.005$	Dev	$0.0025$ – $0.01$ (Tables D3 , D4 )
HGB base learner	lr $0.1$ , $\leq 300$ iter., $\leq 31$ leaves, $\geq 20$ /leaf, no L2; early stop: $10$ iter., $10\%$ of bag	Lib	tuned HGB (Tables H13 , D6 )
Bags	$\leq 32$ draws of $80\%$ of SEARCH; kept if every class $\geq 10$ (multiclass $\geq 2$ ); averaged	Dev	$16$ , $32$ , $64$ bags on $\mathcal{Q}$ (Table C1 )

Appendix

Table D2: Frozen parameters of BRIDGE and of the evaluation protocol: value and basis of each.

Setting	$\varepsilon=0.0025$	$\varepsilon=0.005$ (default)	$\varepsilon=0.0075$	$\varepsilon=0.01$
(A), weighted- $F_{1}$	357/4: 3.83 / 5.10 $\checkmark$	284/3: 4.23 / 5.95 $\checkmark$	234/2: 4.32 / 6.36 $\checkmark$	209/1: 3.74 / 5.67 $\checkmark$
(A), balanced accuracy	856/1: 0.80 / 0.91 $\checkmark$	848/1: 0.81 / 0.92 $\checkmark$	836/0: 0.46 / 0.53 $\checkmark$	825/0: 0.47 / 0.54 $\checkmark$
(A), macro- $F_{1}$	852/0: 0.45 / 0.52 $\checkmark$	829/0: 0.47 / 0.53 $\checkmark$	813/0: 0.48 / 0.55 $\checkmark$	794/0: 0.49 / 0.56 $\checkmark$
(A), feature selection, KNN	263/5: 6.06 / 9.75 $\checkmark$	225/1: 3.45 / 6.00 $\checkmark$	193/0: 2.30 / 4.20 $\checkmark$	169/0: 2.66 / 5.32 $\checkmark$
(A), feature selection, SVC	296/10: 8.20 / 11.52 $\times$	253/7: 7.74 / 11.41 $\times$	211/5: 7.73 / 12.22 $\times$	171/3: 7.42 / 12.61 $\times$
(D) 26, 1,000 rows	648/12: 4.06 / 4.71 $\checkmark$	558/6: 3.04 / 3.59 $\checkmark$	468/4: 2.91 / 3.52 $\checkmark$	404/3: 2.93 / 3.64 $\checkmark$

Appendix

Table D3: Parameter sensitivity: margin sweep.

Margin	Acc./false	$\Delta$ , pp [95% CI]	TOST $p$	Acc. ratio
(A)
0.0025	357 / 4	+0.06 [-0.29; +0.35]	0.011	1.26
0.0075	234 / 2	-0.20 [-0.88; +1.78]	0.0498	0.82
0.01	209 / 1	-0.58 [-1.40; +0.04]	0.0498	0.74
(B)
0.0025	529 / 10	+0.21 [-0.22; +0.81]	$<10^{-4}$	1.11

Appendix

Table D4: Parameter sensitivity: each margin tested against the default

0.005

$q$	$\varepsilon=0.0025$	$\varepsilon=0.005$	$\varepsilon=0.0075$	$\varepsilon=0.01$
(A)
2.5%	299/3, 3.99 eq.	243/2, 4.15 n.e.	202/1, 3.89 n.e.	164/0, 2.76 n.e.
5%	357/4, 3.83 eq.	284/3, 4.23 def.	234/2, 4.32 n.e.	209/1, 3.74 n.e.
10%	422/11, 5.94 n.e.	357/4, 3.83 eq.	297/3, 4.02 eq.	251/2, 4.00 n.e.
(D)
2.5%	1,180/7, 1.52 eq.	972/5, 1.52 eq.	799/2, 1.17 eq.	669/0, 0.61 eq.

Appendix

Table D5: Parameter sensitivity: quantile level

q

against margin

\varepsilon

on the real data.

Design	Naive	BRIDGE	Bound $\downarrow$	LOCO $\downarrow$	LOCO 0 $\downarrow$	Crit.
(A)
Paper	1,064 / 203 (19.1)	284 / 3 (1.1)	4.23	5.95	2.11	$\checkmark$
Tuned HGB	1,061 / 246 (23.2)	211 / 5 (2.4)	7.73	10.66	3.02	$\times$
baseline change, $F_{1}$ points: +0.30 [+0.04; +0.61]
Gate tuned	1,064 / 203 (19.1)	299 / 2 (0.7)	3.30	4.59	1.97	$\checkmark$
(D)

Appendix

Table D6: Parameter sensitivity: tests of tuning on the real data.

Variant	Acc.	False	$U$ , % $\downarrow$	Cons., % $\downarrow$	Yield $\uparrow$	Gain $\uparrow$
$F{=}2$ , $q{=}2.5\%$	382	2	2.00	2.45	27.3	13.81
$F{=}2$ , $q{=}5\%$	445	4	2.47	4.41	31.8	15.09
$F{=}2$ , $q{=}10\%$	548	10	3.59	5.50	39.1	16.89
$F{=}3$ , $q{=}2.5\%$	406	4	2.70	3.08	29.0	14.34
$F{=}3$ , $q{=}5\%$ (standard)	475	8	3.56	4.70	33.9	15.61
$F{=}3$ , $q{=}10\%$	563	16	4.90	6.24	40.2	17.12

Appendix

Table D7: Parameter sensitivity on the synthetic tasks (B).

	MIC	Mktg	Census	Homes.	Sant.
POOL rows	741	741	1,800	1,800	1,800
Features	104	25	41	297	369
Fit bags on SEARCH (s)	1.39	0.73	1.30	6.19	4.24
Naive selection (s) $\downarrow$	0.006	0.006	0.006	0.007	0.007
Cross-fitted selection (s) $\downarrow$	0.017	0.017	0.019	0.019	0.019
Bootstrap (s) $\downarrow$	0.011	0.011	0.010	0.009	0.006

Appendix

Table D8: Wall-clock cost of BRIDGE.

Dataset	Naive	$5\times 2$	Corr.	Re-sel.	No-m.	BRIDGE
Census-KDD	100/0	60/0	86/0	40/0	55/0	17/0
Homesite	99/3	49/0	71/0	41/0	53/0	30/0
KDD98	83/54	7/3	8/6	3/3	2/1	0/0
LICD	100/0	79/0	90/0	79/0	86/0	72/0
MIC	100/1	75/0	96/1	72/1	82/1	71/1
Marketing	100/0	78/0	98/0	81/0	87/0	76/0

Appendix

Table E1: Per-dataset results on the 11 new datasets.

Real data (11 new datasets)
Comparator	$\Delta$ , pp [95% CI]	$p$	Holm $p$
naive	$-18.0\ [-30.2;\,-7.1]$	$<$ 0.0002 †	$<$ 0.0012 †
McNemar, same POOL	$-11.0\ [-19.2;\,-1.5]$	0.0026	0.0078
$5\times 2$ cv $t$	$-5.2\ [-12.2;\,-1.4]$	0.0006	0.0024
corrected $t$	$-5.2\ [-11.7;\,-1.6]$	0.0002	0.0010
BRIDGE without margin	$-1.7\ [-4.8;\,-0.3]$	0.0046	0.0092

Appendix

Table E2: Paired comparisons: the difference BRIDGE minus each comparator.

Learner	AUC $\uparrow$	Log-loss $\downarrow$	Extreme, % $\downarrow$	wF1 $\uparrow$	Rounds	Fit, s $\downarrow$
HGB	0.578	0.675	0.00	0.552	16.67	0.028
XGBoost, default	0.566	0.810	1.68	0.559	300.00	0.350
XGBoost, regul.	0.579	0.673	0.00	0.520	7.50	0.032

Appendix

Table E3: Audit of the base learners.

Source	Identity key	Rows	Unique keys	Dup.
WiDS 2020 (GOSSIS)	patient_id	91,713	91,713	0
one row per patient
unique encounter_id: 91,713
hospitals: 147
COVID-19 Mexico	id (record id)	566,602	563,201	3,401
duplicates are excess rows

Appendix

Table F1: Patient-identity audit of the clinical sources.

Net benefit, $p_{t}=0.2$ (co-primary)
Source or subset	BRIDGE	Unc.	Bound $\downarrow$	Cons. $\downarrow$	LOCO $\downarrow$	Naive
ACTG 175	198/0	2	–	–	–	200/0
BRFSS (4 tasks)	210/0	0	–	–	–	684/282
COVID-19 Mexico	200/0	0	–	–	–	200/0
Mammography	10/0	1	–	–	–	200/0
NHANES (4 tasks)	401/0	1	–	–	–	782/171

Appendix

Table F2: Patient-identity audit: clinical results by source.

Dataset	Source	N	K	Feat.	Naive	BRIDGE
(E) $14$ OpenML datasets
airline-satisfaction	OML 46920	129,880	2	21	99/6	22/0
KDDCup09-churn	OML 1112	50,000	2	230	80/62	0/0
bank-marketing	OML 1461	45,211	2	16	100/0	53/0
pulsar-HTRU2	OML 45558	17,898	2	8	97/45	0/0
jm1	OML 1053	10,885	2	21	100/2	63/0

Appendix

Table G1: Datasets of the OpenML collection (E) and its extension (E2), with per-dataset decisions and curation flags.

Setting	Decisions	Naive: accepted	Naive % false [95% CI] $\downarrow$	BRIDGE: accepted	BRIDGE % false [95% CI] $\downarrow$	Reduction, pp [95% CI] $\uparrow$	Upper bound $\hat{U}$ or $U$ (%) $\downarrow$	False removed % [95% CI] $\uparrow$
(A) 11 new datasets	1,100	1,064	$19.1\ [8.1;\,31.6]$	284	$1.1\ [0.0;\,4.0]$	$18.0\ [7.1;\,30.2]$	4.23	99 [95; 100]
(B) Synthetic tasks	1,400	1,323	$20.7\ [18.5;\,22.9]$	475	$1.7\ [0.6;\,2.9]$	$19.0\ [16.8;\,21.3]$	3.56	97 [95; 99]
(C) Clinical tasks, net benefit	2,400	2,266	$20.0\ [4.3;\,40.4]$	1,199	0.0	$20.0\ [4.5;\,40.3]$	0.33	100 [100; 100]
(D) 39 supporting datasets	3,700	3,562	$12.5\ [7.4;\,18.6]$	1,113	$0.5\ [0.1;\,1.2]$	$12.0\ [7.2;\,17.9]$	1.47	99 [97; 100]
(E) OpenML collection, 14 datasets	1,280	1,236	$25.6\ [15.0;\,37.8]$	192	$1.0\ [0.0;\,4.0]$	$24.6\ [14.0;\,36.0]$	5.40	99 [98; 100]

Appendix

Table H1: Main results in detail, (A)–(E).

Test	Comparison (T1: BRIDGE minus)	(A)	(B)	(C)	(D)	(E)
T1 ( 14 )	Naive	$-18.0$ ( $p<0.0012^{*}$ )	$-19.0$ ( $p<0.0012^{*}$ )	$-20.0$ ( $p=0.034^{*}$ )	$-12.0$ ( $p=0.0009^{*}$ )	$-24.6$ ( $p<0.0002^{*}$ )
	McNemar, same POOL	$-11.0$ ( $p=0.0078^{*}$ )	$-33.1$ ( $p<0.0012^{*}$ )	$0.0$ ( $p=1.00$ )	$-6.1$ ( $p=0.0009^{*}$ )	–
	McNemar, cross-fit	–	–	$0.0$ ( $p=1.00$ )	$-2.0$ ( $p=0.018^{*}$ )	–
	$5\times 2$ cv paired $t$	$-5.2$ ( $p=0.0024^{*}$ )	$-3.9$ ( $p<0.0012^{*}$ )	$-2.5$ ( $p=0.034^{*}$ )	$-3.1$ ( $p=0.0009^{*}$ )	–
	Corrected resampled $t$	$-5.2$ ( $p=0.0010^{*}$ )	$-4.0$ ( $p<0.0012^{*}$ )	$-2.6$ ( $p=0.10$ )	$-3.6$ ( $p=0.0009^{*}$ )	–
	Corrected $t$ , re-selection	$-1.2$ ( $p=0.32$ )	$-1.3$ ( $p=0.051$ )	$-0.4$ ( $p=0.34$ )	$-1.0$ ( $p=0.0009^{*}$ )	–

Appendix

Table H2: Hypothesis tests: estimates and

p

-values, collection by collection.

Rule	Acc- epted	False (%) $\downarrow$	Bound % $\downarrow$	LOCO % $\downarrow$	Yield /100 $\uparrow$	Gain ${>}\varepsilon$ /100 $\uparrow$	95% CI, pp
(A) 11 new datasets, 1,100 decisions
Naive	1,064	203 (19.1)	22.90	25.27	96.7	59.5	[ $-30.2$ ; $-7.1$ ] ∗
McNemar, POOL	183	22 (12.0)	24.52	30.68	16.6	12.5	[ $-19.2$ ; $-1.5$ ] ∗
McNemar, cross-fit	45	4 (8.9)	41.14	54.35	4.1	3.2	–
$5\times 2$ cv $t$	461	29 (6.3)	10.99	13.43	41.9	36.2	[ $-12.2$ ; $-1.4$ ] ∗
Corrected $t$	608	38 (6.2)	10.09	12.15	55.3	46.8	[ $-11.7$ ; $-1.6$ ] ∗

Appendix

Table H3: Acceptance rules on (A)–(E): counts, bounds, yield, gain, and paired differences, in one format.

		Accepted (% false)		Random draw (T3): false, expected vs BRIDGE		False removed	Gain per update
Setting	Decisions	Naive	BRIDGE	Pooled	Within dataset	(%) $\uparrow$	BRIDGE / naive $\uparrow$
(A)	1,100	1,064 (19.1)	284 (1.1)	54.2 vs 3 (p $<10^{-25}$ )	5.5 vs 3 (p 0.15)	99 [95; 100]	0.0385 / 0.0156
(B)	1,400	1,323 (20.7)	475 (1.7)	98.4 vs 8 (p $<10^{-47}$ )	–	97 [95; 99]	0.0329 / 0.0157
(C)	2,400	2,266 (20.0)	1,199 (0.0)	239.7 vs 0 (p $<10^{-176}$ )	0.7 vs 0 (p 0.45)	100 [100; 100]	0.0416 / 0.0233
(D)	3,700	3,562 (12.5)	1,113 (0.5)	139.0 vs 6 (p $<10^{-67}$ )	27.4 vs 6 (p $<10^{-9}$ )	99 [97; 100]	0.0239 / 0.0115
(E)	1,280	1,236 (25.6)	192 (1.0)	49.2 vs 2 (p $<10^{-23}$ )	12.9 vs 2 (p $<10^{-5}$ )	99 [98; 100]	0.0197 / 0.0063

Appendix

Table H4: Acceptance rules: random-selection control and gain per update on (A)–(E).

Accepted updates gaining at most $\varepsilon$ (%) $\downarrow$
Rule	(A)	(B)	(C)	(D)	(E)
Naive	38.4 ∗	37.3 ∗	33.9 ∗	39.7 ∗	57.4 ∗
McNemar, same POOL	24.6 ∗	53.3 ∗	47.1 ∘	26.1 ∗	–
McNemar, cross-fit	22.2 ∘	38.0 ∗	25.0 ∗	14.2 ∗	–
$5\times 2$ cv paired $t$	13.7 ∗	13.8 ∗	8.5 ∗	18.1 ∗	–
Corrected resampled $t$	15.3 ∗	14.0 ∗	10.2 ∗	21.4 ∗	–

Appendix

Table H5: Acceptance rules on (A)–(E): accepted updates below and above the margin on the reference (post hoc).

Setting	Acc- epted	False	Bound % $\downarrow$	LOCO % $\downarrow$	Yield /100 $\uparrow$	Crit.
(A) weighted $F_{1}$	284	3	4.23	5.95	25.82	$\checkmark$
(A) balanced accuracy	848	1	0.81	0.92	77.09	$\checkmark$
(A) macro- $F_{1}$	829	0	0.47	0.53	75.36	$\checkmark$
(A) feat. selection + threshold (KNN)	225	1	3.45	6.00	20.45	$\checkmark$
(A) feat. selection + threshold (SVC)	253	7	7.74	11.41	23.00	$\times$
(D) 26 datasets, 1,000 rows	558	6	3.04	3.59	21.49	$\checkmark$

Appendix

Table H6: Conditions of use: the metrics, the training sizes, the multiclass tasks, and feature selection.

		$173$ (default)		$1{,}825$		$11{,}249$
Data	Rule	False % $\downarrow$	Bound % $\downarrow$	False % $\downarrow$	Bound % $\downarrow$	False % $\downarrow$	Bound % $\downarrow$
(A)	Naive	19.08	22.90 $\times$	20.44	24.23 $\times$	19.46	23.16 $\times$
	Same-POOL	2.65	5.85 $\checkmark$	3.61	7.02 $\checkmark$	3.80	7.17 $\checkmark$
	BRIDGE	1.06	4.23 $\checkmark$	1.45	5.07 $\checkmark$	1.37	4.75 $\checkmark$
(B)	Naive	20.71	23.52 $\times$	17.36	19.93 $\times$	16.88	19.43 $\times$
	Same-POOL	3.18	5.14 $\checkmark$	3.62	5.48 $\checkmark$	4.84	6.84 $\checkmark$

Appendix

Table H7: Component analysis: candidate-space size on the collections (A)–(E).

Row	Condition	Setting	Evidence	Status
1	Metrics	(A)	H6	met
2	Training sizes	(D)	H6	met
3	Multiclass tasks	(D)	H6	met
4	Feature selection	(A)	H6	conditional
5	Supporting datasets	(D)	H1	met
6	OpenML collections	(E), (E2)	H1 , App. G	conditional

Appendix

Table H8: Conditions of use: setting, evidence, and status.

Minority prev.	Naive: acc. / false (%)	BRIDGE: acc. / false (%)	Bound % $\downarrow$	LOCO % $\downarrow$	LOCO 0 % $\downarrow$	Yield $\uparrow$	Crit.
weighted- $F_{1}$
20%	494 / 29 (5.9)	138 / 1 (0.7)	5.83	11.78	6.63	27.6	$\times$
10%	474 / 95 (20.0)	100 / 0 (0.0)	4.76	14.77	14.77	20.0	$\times$
5%	403 / 139 (34.5)	36 / 0 (0.0)	16.42	48.81	48.81	7.2	$\times$
balanced accuracy
20%	500 / 0 (0.0)	477 / 0 (0.0)	0.80	1.01	1.01	95.4	$\checkmark$

Appendix

Table H9: Conditions of use: class imbalance on five datasets of (A).

Base learner setting	Acc./ false	$U$ % $\downarrow$	Cons. % $\downarrow$	Yield $\uparrow$	Gain $\uparrow$	Naive % $\downarrow$	Skill $\uparrow$	Crit.
HGB, default (early stop.)	462/7	3.35	4.54	33.00	15.95	22.00	0.009	$\checkmark$
XGBoost, regularized	893/5	1.41	1.74	63.79	61.28	9.23	0.012	$\checkmark$
XGBoost, default (300 rounds)	146/15	17.48	23.21	10.43	1.96	31.87	-0.190	$\times$
+ early stop.	963/1	0.59	1.15	68.79	71.04	6.75	0.011	$\checkmark$
+ min_child_weight 5	138/18	21.08	26.99	9.86	1.77	31.59	-0.146	$\times$
+ 31-leaf trees	125/9	14.21	21.24	8.93	1.69	32.34	-0.194	$\times$

Appendix

Table H10: Conditions of use: all tested base-learner settings on (B).

Paper’s design
Design	(A)	(B)	(C)	(D)	(E)
Default (DT, SVC)	1.1 $\checkmark$	1.7 $\checkmark$	0.0 $\checkmark$	0.5 $\checkmark$	1.0 $\checkmark$
Members replaced (base HGB)
LR and NB	2.3 $\checkmark$	0.5 $\checkmark$	0.2 $\checkmark$	0.9 $\checkmark$	2.5 $\times$
MLP and KNN	1.2 $\checkmark$	1.5 $\checkmark$	0.0 $\checkmark$	1.9 $\checkmark$	4.1 $\times$
Base learner replaced (members DT, SVC)

Appendix

Table H11: Conditions of use: ensemble members and base learners across (A)–(E).

Base learner	Baseline mean	Final mean	Final minus HGB, $F_{1}$ points [95% CI]	Better / equal / worse
(A) 11 new datasets
HGB	0.855	0.864	+0.000 [+0.000; +0.000]	0 / 11 / 0
XGB0	0.863	0.864	-0.016 [-0.235; +0.210]	5 / 0 / 6
XGBES	0.849	0.866	+0.203 [-0.050; +0.521]	4 / 1 / 6
MLPES	0.823	0.862	-0.217 [-0.727; +0.367]	3 / 0 / 8
(D) 39 supporting datasets

Appendix

Table H12: Conditions of use: final model quality of four base learners.

Learner	Naive acc./ false (%)	BRIDGE acc./ false (%)	Bound % $\downarrow$	LOCO % $\downarrow$	LOCO 0 % $\downarrow$	Yield $\uparrow$	Crit.
(A) 11 new datasets
HGB	526 / 97 (18.4)	86 / 6 (7.0)	23.86	43.42	10.49	7.8	$\times$
DT	677 / 120 (17.7)	171 / 6 (3.5)	10.88	20.29	5.46	15.5	$\times$
SVC	503 / 92 (18.3)	163 / 6 (3.7)	11.48	24.84	6.00	14.8	$\times$
KNN	571 / 169 (29.6)	39 / 4 (10.3)	49.17	59.92	33.67	3.5	$\times$
(B) Synthetic tasks

Appendix

Table H13: Conditions of use: tuning treated as an update on (A), (B), and (D).

		Per $1{,}000$			Net benefit
Task	Acc.	Flagged	TP	FP	Before	After	All
BRFSS depression	197	88 $\to$ 291	65.3	137.8	0.0495	0.0803	-0.0116
NHANES diabetes	200	37 $\to$ 390	100.3	253.4	0.0133	0.0502	-0.0467
NHANES asthma	1	0 $\to$ 85	22.2	62.9	0.0000	0.0064	-0.0620
BRFSS asthma	13	1 $\to$ 124	31.0	92.2	0.0002	0.0082	-0.0812
ICU (WiDS)	180	9 $\to$ 112	35.8	67.3	0.0059	0.0248	-0.1421

Appendix

Table H14: Clinical application (C): consequences for each

1{,}000

observations.

Cross-Fitting

See all 9 papers

Sep 12, 2026

Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback

An updated model can improve an aggregate metric while degrading a slice that matters to a downstream user. We study checkpoint selection subject to non-regression tolerances relative to a retained incumbent. The central distinction is between failing to detect harm and certifying non-inferiority: the former can release harmful updates with high probability when evaluation is noisy. We give a reproducible release procedure that separates candidate search from independent, paired evaluation and returns the exact incumbent when certification fails. Applying established intersection-union and Learn-then-Test principles, we state finite-sample guarantees for one frozen candidate, a finite candidate library, and a prespecified testing order. A joint release decision does not require a slice-count Bonferroni penalty, although certification power can still decrease with the number of slices. In bounded-score simulations, a no-detected-harm gate releases a harmful candidate in 99.7% of trials in one 32-slice setting, compared with 2.6% for an exact non-inferiority gate at a 5% target. A constructed two-block family yields larger certified utility than a scalar path under matched candidate counts. Public digits experiments, including a subsequent continuation that improves average aggregate accuracy, return the incumbent in every run because certification is underpowered. These results establish an auditable protocol and its limitations; they do not establish benefits on foundation-model or multilingual translation upgrades.

Shengwei Zhang, Tao Wu, Fei Qian

University of Pennsylvania · Alibaba International Digital Commerce

Model Selection Automated Evaluation Statistical Hypothesis Testing

Ask PDF

Sep 10, 2026

Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation

Continual world models must decide whether new data justify changing the model. Fixed replay schedules and prediction-error triggers specify when to update, but neither reveals the value of an individual update: one deployment run cannot show how the same model would have performed at that moment had it held its parameters. We introduce the fork ledger, which branches a deployment stream at pre-registered decision points into matched update and hold continuations under common random numbers. It evaluates both continuations on the same episodes and records

ΔR = R_{\mathrm{update}} - R_{\mathrm{hold}}

. Always applying one fixed update mechanism lowers return on all three simulated control tasks: CartPole (

-144.0

; checkpoint-bootstrap

95\%

[-185.4,-116.1]

, against a converged return near

650

), Walker (

-82.8

;

[-101.1,-61.7]

) and Cheetah (

-18.6

;

[-29.0,-6.6]

). Divergence is an outcome of applying the update, so the estimand counts every attempted fork; restricted to the

693

720

that did not collapse, CartPole and Walker are unchanged in sign (

-113.4

and

-82.1

) and Cheetah becomes unresolved (

-3.9

;

[-17.5,+13.0]

). The task is the unit of inference: each contributes

240

attempted forks over five pretrained checkpoints crossed with two drift directions. The ledger makes counterfactual utility observable for a fixed mechanism, allowing triggers to be judged by the updates they select rather than by surprise detection alone.

Anqi Peter Li, Kaden Kim

Substrate Labs · UC Berkeley

Counterfactual Evaluation World Model Learning Continual RL Model-Based RL

Ask PDF

Oct 2, 2026

Large language models exhibit unreliable updating of clinical judgment as patient evidence evolves

Large language models (LLMs) are increasingly explored for clinical reasoning, but whether they appropriately revise judgments as patient evidence evolves remains unclear. We evaluated longitudinal belief updating using matched intensive-care trajectories from electronic health records. Across diverse LLMs, conditioning on a preceding judgment more often increased than reduced prediction error when estimates changed, replicated for a second endpoint. Controlled interventions revealed two failure modes. First, with preceding assessment fixed, models responded more strongly to worsening than matched improving respiratory evidence; this asymmetry persisted after headroom normalization at moderate and strong evidence levels. Second, with current evidence fixed, increasing prior risk from 10% to 90% shifted estimates by 26.2 percentage points, demonstrating causal influence of prior model beliefs. Prompting did not restore reliable updating. Evidence-Validated Longitudinal Update (EVLU) identified fewer, more reliable revisions, revealing a reliability-coverage trade-off. These findings establish longitudinal belief updating as a distinct dimension of LLM reliability.

Min Zeng, Rui Zhang

Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, 55455, MN, USA.

Belief Updating in LLMs LLM Reliability Clinical Language Model Evaluation Longitudinal EHR Modeling

Ask PDF

Abstract

Cross-Fitting

Explore similar work

Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback

Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation

Large language models exhibit unreliable updating of clinical judgment as patient evidence evolves