When Can a Learned Design Surrogate Be Trusted in a Self-Driving Microfluidics Laboratory? An Offline Closed-Loop Framework Built and Tested on Six Public Tables

Zhiyuan Zhang

cs.LGOct 26, 2026

When Can a Learned Design Surrogate Be Trusted in a Self-Driving Microfluidics Laboratory? An Offline Closed-Loop Framework Built and Tested on Six Public Tables

Authors: Zhiyuan Zhang

Organizations: Independent research.

Abstract

Self-driving laboratories couple a learned design surrogate to an instrument, and on every new target domain it must answer four questions: when it can be trusted, how many target measurements it needs, which to make first, and when to stop adapting, switch models or escalate. We state a framework making each measurable (a competence ratio, a budget $k^*(\varepsilon)$ , a time-to-target and a priced decision) and instantiate it without wet experiments on six public droplet-microfluidic and liposome tables as offline laboratories: 107 leave-one-group tiers whose rows are hidden labels. A sealed, pre-registered programme (two sealed negative validations and a pre-registered test on three more tables) and seven frozen closed-loop protocols, each run twice, yield 23 findings. Shift cost is stratified by shift type; no label-free statistic certifies individual candidates; random-split model-family rankings predicted neither the sealed nor the loop rankings. Warm-start residual boosting returns a surrogate to $1.25\times$ its development reference within 20 target rows in 94% of chemistry tiers (27% under a strict noise-floor reading), 90% of laboratory and 40% of device tiers, and a budget planned from the shift type on other tables is not measurably worse on a new one. In the loop, one gain survives drift and shuffled-label controls on all tiers, carried by the sparsely sampled ones: proposing at once and warm-starting on every measurement reaches a 10% target in 2.80 measurements against 3.31 for the reference, and in 5.72 against 11.35 for random proposals on 301 hard units at a 3% tolerance. No label-free acquisition rule survives the shuffled-label control, no interval-aware proposal or other model family does better, and no threshold rule on the tested diagnostics beats always continuing once escalation is priced. No wet-lab validation is claimed.

Figures & tables

Panel of Figure 1 — Figure 1: Research design. Six public tables enter in four stages, drawn as four stage bands; within the first two bands the three columns are the three tables of that stage, and the fourth band replays all six tables as offline laboratories. Shading marks the evidence tier of each block, as listed in the key below the bands (white: retrospective; blue: pre-registered with sealed labels and a single reveal gate; teal: protocol frozen before the run and the run repeated with a byte-identical artifact manifest; orange: post hoc on revealed labels under a protocol frozen before the run; R7, frozen with two runs, shares the white box of R1–R4). Section 6 keeps the unit labels R1–R10 and presents them by question.

Panel of Figure 2 — Figure 2: The decision framework. A surrogate trained on a source table enters a loop on a new target domain; the four questions sit at the points where the loop must decide. Offline, the instrument is replaced by the rows of a held-out tier (Section 3.4 ); prospectively, by the instrument itself (Section 9 ).

Unit	Table(s)	Protocol status	Q	What it establishes
R1	D1	retrospective re-implementation; fidelity check (Appendix D )	Q1, P	the published configuration is reproduced up to one library default
R2	D1 (external)	retrospective	Q1	error grows across literature sources
R3	D1	retrospective	Q1	nominal intervals lose coverage under shift
R4	D1	retrospective, with negative controls	Q1	label-free pruning gains are source-level, not candidate-level
R5	D2	pre-registered, sealed, one reveal	Q1	an unseen chemistry inside the marginal ranges fails
R6	D2	frozen protocol, retrospective after the reveal	Q4	hard bounds remove invalid predictions but do not beat the XGBoost baseline

Table 1: Map of the evidence units. “Retrospective” means every analyst choice was made with the labels available; “sealed” means labels were split off before any analysis and revealed once through an exclusive gate; “frozen” means the protocol was hashed and committed before the run. Q1 competence, Q2 budget, Q3 acquisition, Q4 decision; P provenance; M method.

Metric	Published [ 2 ]	This re-implementation
Diameter MAPE	5.4%	$5.268\%\pm 0.421\%$
Diameter MAE	—	$4.502\pm 0.529$ µm
Diameter $R^{2}$	—	$0.960\pm 0.010$
Frequency MAPE (mass conservation)	16.6%	$16.697\%\pm 1.819\%$

Table 2: Baseline re-implementation on 15 random 80/20 splits of the 868-row DAFD 3.0 comprehensive table (mean

\pm

SD over splits).

Source	$n$	MAPE	MAE (µm)	$R^{2}$
Galogahi et al.	24	8.35%	10.95	0.812
Lashkaripour et al.	20	8.78%	3.77	0.724
Mazutis et al.	6	19.41%	8.53	$-10.18$
Roumpea et al.	14	16.29%	28.08	0.272
All external	64	11.26%	12.23	0.917

Table 3: External literature set after training on all 868 comprehensive rows.

R^{2}

for

n\leq 14

sources is unstable and not interpreted alone.

Figure 3: Nominal 90% CQR coverage. Left pair: internal random test rows versus the external literature set over 15 runs of the pilot analysis. Right pair: external coverage in the expanded suite under row-random and geometry-grouped calibration (15 seeds each; Section 6.1.6 ). Error bars are one standard deviation over runs. Geometry-grouped calibration recovers 84.8% coverage at a mean interval width of 89.8 µm versus 34.1 µm.

	Pilot (LOO q95)	Canonical (grouped q95)
Threshold (standardised distance)	0.9572	1.5647
Retained / pruned	26 / 38	32 / 32
Retention by source (G/L/M/R)	0/24, 20/20, 6/6, 0/14	6/24, 20/20, 6/6, 0/14
Full-pool success, 10 µm	59.4% (38/64)	59.4% (38/64)
Retained precision, 10 µm	92.3% (24/26)	93.8% (30/32)
Successful-candidate recall, 10 µm	63.2% (24/38)	78.9% (30/38)

Table 4: Label-free 5-NN pruning of the 64 external candidates: pilot rule (leave-one-out training threshold) and canonical rule (dependency-grouped calibration threshold). Precision is over retained candidates; recall is over all successful candidates in the full pool.

Figure 4: Pilot rule: standardised 5-NN distance to the training inputs versus absolute external error, with the 95th-percentile threshold.

Figure 5: Pilot rule: fraction of candidates within tolerance before and after pruning. Precision rises while recall falls; both must be read together.

Endpoint	Value
MAPE	19.8436%
Reproduction criterion ( $<11.4\%$ )	not met
Strong-replication criterion ( $\leq 6.0\%$ )	not met
MAE / RMSE	0.022615 / 0.028681 mm
$R^{2}$	$-0.2436$
Nominal 90% coverage (mean width)	0.4483 (0.03281 mm)

Table 5: Locked V1 endpoints on the sealed TX100 group (

n=87

Figure 6: Negative result. Left: frozen V1 predictions versus observed TX100 diameters (87 rows), coloured by absolute relative error. Right: signed residuals versus prediction; the dashed line is the mean bias (0.0213 mm).

Figure 7: Negative result, applicability view. Tie-aware selective risk on the TX100 rows for the four locked screens. No screen brings the retained risk below the full-pool risk except at a large loss of retention; the only pre-declared benefit (Isolation Forest) does not survive BH adjustment.

Figure 8: Focal XGBoost diameter MAPE under four split protocols on the 868-row table. Grouped protocols prevent double-emulsion pairs and device geometries from crossing the fit/test boundary.

Figure 9: Relative-risk versus retained-coverage curves for applicability scores and controls on the 64 external rows. The hydraulic-diameter-only control has the lowest AURC; no label-free applicability score beats it.

Figure 10: Retention rate by external source for binary rules at the 95% calibration quantile. For every rule retention is all-or-none within source–geometry contexts, which is why within-source-context permutation nulls explain the observed risk reduction.

			Point error (MAPE; ratio to reference)				Nominal 90% coverage
Cell	Shift	Repeats	XGB	Ratio	RF	MLP	Split	CQR	Mondrian	Ens.
D1 random	reference	15	6.71%	1.00	6.86%	14.52%	0.959	0.948	0.998	0.917
D1 geometry	device geometry	30	12.43%	1.85	11.78%	55.67%	0.860	0.866	0.970	0.854
D1 external	cross-literature	1	11.66%	1.74	11.80%	16.70%	0.453	0.484	0.734	0.547
D1 SE $\to$ DE	device type	1	22.12%	3.30	21.19%	266.34%	0.330	0.414	0.525	0.586
D1 DE $\to$ SE	device type	1	69.18%	10.31	73.67%	3592.49%	0.065	0.051	0.097	0.259

Table 6: Method-matrix benchmark. XGBoost MAPE and its ratio to the dataset’s random-split reference, the alternative point models, and empirical coverage of nominal 90% intervals (XGBoost; means over repeats). Reference cells use the fit partition and are not comparable with Table 2 .

Cell	Full pool	Random	Size-only	5-NN	Mahalanobis	Isol. Forest	Domain clf.	Density ratio
D1 random	0.0671	0.0662	0.0541	0.0466	0.0516	0.0535	0.0653	0.0543
D1 geometry	0.1243	0.1255	0.1141	0.1085	0.1142	0.1225	0.1153	0.1097
D1 external	0.1166	0.1107	0.0913	0.1135	0.0957	0.1178	0.1161	0.1039
D1 SE $\to$ DE	0.2212	0.2256	0.3128	0.1442	0.1467	0.2219	0.1394	0.1737
D1 DE $\to$ SE	0.6918	0.6734	0.6423	0.6017	0.5874	0.4182	0.5729	0.5263
D2 random	0.0391	0.0388	0.0398	0.0365	0.0379	0.0371	0.0368	0.0324

Table 7: Method-matrix benchmark. AURC of the relative-error risk–coverage curve (lower is better) for each applicability score and control, XGBoost, means over repeats; bold marks the best entry in each cell.

Cell	MdAPE	sMAPE	Log-ratio	Ratio MdAPE	Ratio sMAPE	Ratio log
D1 random	4.39%	6.80%	0.068	1.00	1.00	1.00
D1 geometry	9.71%	12.45%	0.125	2.21	1.83	1.84
D1 external	12.69%	11.66%	0.117	2.89	1.71	1.71
D1 SE $\to$ DE	21.60%	25.89%	0.263	4.92	3.81	3.85
D1 DE $\to$ SE	65.92%	47.47%	0.492	15.02	6.98	7.20
D2 random	2.88%	3.83%	0.038	1.00	1.00	1.00

Table 8: Post-hoc metric robustness of the method matrix (XGBoost; means over repeats; not part of the frozen protocol). Median APE (MdAPE) and symmetric MAPE (sMAPE,

100\cdot\mathrm{mean}\,|\hat{d}-d|/((|\hat{d}|+|d|)/2)

) in percent; log-ratio error is the mean absolute

\log(\hat{d}/d)

. Ratios are to the same dataset’s random-split reference cell under the same metric.

Cell	Model	MAPE	Ratio	Native	Split	Conformalised	AURC uncert.	AURC size-only
D1 random	QRF	7.65%	1.00	0.908	0.947	0.903	0.0540	0.0622
	GP	9.51%	1.00	0.860	0.974	0.953	0.0596	0.0679
	Deep ens.	14.46%	1.00	0.403	0.960	0.914	0.0784	0.1065
D1 geometry	QRF	12.00%	1.57	0.859	0.863	0.873	0.1047	0.1115
	GP	20.81%	2.19	0.709	0.862	0.885	0.1847	0.1907
	Deep ens.	39.81%	2.75	0.608	0.847	0.885	0.3328	0.3838

Table 9: Uncertainty-aware model families on the method-matrix cells (frozen W4 protocol; means over repeats; two runs byte-identical; QRF: quantile regression forest). Ratio is MAPE relative to the same model’s random-split reference cell. Coverage of nominal 90% intervals: native, split conformal around the point prediction, and the conformalised native interval. AURC of each model’s relative-error risk–coverage curve (lower is better) for its relative uncertainty score and for the size-only control.

	Unweighted		Domain-classifier odds					$k$ -NN density ratio
Cell	Cov.	Width	Cov.	Inf.	Fin. cov.	Width	ESS	Cov.	Inf.	Fin. cov.	Width	ESS
D1 random	0.959	33.0	0.955	0.000	0.955	32.4	0.841	0.863	0.002	0.862	24.1	0.137
D1 geometry	0.860	43.9	0.908	0.209	0.880	52.1	0.420	0.886	0.553	0.788	42.0	0.168
D1 external	0.453	26.3	1.000	1.000	—	—	0.076	1.000	1.000	—	—	0.281
D1 SE $\to$ DE	0.330	8.9	1.000	1.000	—	—	0.998	1.000	1.000	—	—	0.416
D1 DE $\to$ SE	0.065	21.2	1.000	1.000	—	—	0.836	1.000	1.000	—	—	0.018

Table 10: Covariate-shift-weighted split conformal for the reference XGBoost model (frozen W5 protocol; means over repeats; two runs byte-identical). “Cov.” counts infinite intervals as covering; “inf.” is the fraction of infinite intervals; “fin. cov.” and “width” are over finite intervals only; ESS is the effective sample size of the calibration weights divided by the calibration size: it describes how evenly the calibration weights are spread among themselves, not the test weight relative to them, so an interval can be infinite (test-point weight share above

1-0.90

) while ESS is near one, as in D1 SE

\to

DE. Dashes: every interval infinite.

Endpoint	Value
Development reference MAPE (laboratory A, grouped 5-fold CV, frozen pre-reveal)	39.41%
Reproduction ( $\leq 1.25\times$ ) / strong-replication ( $\leq 1.00\times$ ) thresholds	49.27% / 39.41%
Sealed laboratory-B MAPE	240.39%
Reproduction / strong-replication criteria	not met / not met
MdAPE (descriptive)	206.74%
MAE / RMSE	595.3 / 1335.0 nm

Table 11: Locked endpoints of the second sealed validation (Smart Microfluidics, laboratory A

\to

laboratory B,

n=57

Table	Tier ( $n$ )	dev-ref	XGBoost MAPE [95% CI]	$r$ [95% CI]	Cov.	log-/raw-MLP MAPE (invalid)	$\rho$
A1	S2 fluid family (40)	21.82	28.78 [25.0, 32.7]	1.32 [1.15, 1.50]	0.900	15.6 / 18.0 (0 / 0)	$-0.77$
A1	S3 device (200)	21.82	109.84 [95.1, 125.6]	5.03 [4.36, 5.76]	0.225	$1.66\times 10^{6}$ / 1134 (0 / 0)	$-0.14$
R2-A1	S2 fluid pair (164)	13.62	25.28 [22.4, 28.5]	1.86 [1.65, 2.10]	0.817	26.0 / 27.7 (0 / 0)	0.37
R2-A1	S3 device (109)	13.62	67.48 [62.3, 72.8]	4.95 [4.57, 5.34]	0.330	97.5 / 2286 (0 / 103)	$-0.71$
R2-A2	S2 ionizable lipid (32)	40.87	32.63 [23.9, 42.0]	0.80 [0.58, 1.03]	0.938	31.8 / 36.7 (0 / 0)	$-0.54$
R2-A2	S3 source paper (16)	40.87	35.26 [19.5, 58.3]	0.86 [0.48, 1.43]	0.938	29.6 / 36.0 (0 / 0)	0.20

Table 12: R10 sealed tiers. dev-ref is the grouped cross-validation MAPE of the locked XGBoost on the development rows (2 folds on R2-A1, 5 folds otherwise);

r

is the sealed MAPE over dev-ref with row-bootstrap 95% intervals; coverage is split-conformal coverage at nominal 0.90; the perceptron pair is the log-target (bounded) and raw-scale (unbounded) MLP with the number of non-positive predictions;

\rho

is the Spearman correlation between the six-model ranking on random splits and on the tier. A1 device-tier MLP values are the pre-registered run; R2-A1 bounded-MLP values are from the corrected re-run.

Hypothesis	A1	R2-A1	R2-A2	Protocol level
H-A band $[2.2,3.0]$	falsified (1.32 / 5.03)	falsified (1.86 / 4.95)	falsified (0.80 / 0.86; S4 0.62)	falsified on all three
H-B1 bounded vs unbounded MLP	falsified (S3)	supported	supported	falsified (A1 fails)
H-B2 anchored residual	no anchor	no anchor	no anchor	not evaluable
H-C1 / H-C2 dimensionless vs native	not supported ( $+0.28$ pp, $p=0.45$ ) / 0 pairs	not supported (2 groups) / 0 pairs	not evaluable	not determinable
H-D $\|A-W\|\leq 0.10$	$A=W=0$	$A=W=0$	$A=W=0$	supported (degenerate)
H-E1 rankings	$\rho\leq 0.5$	$\rho\leq 0.5$	$\rho\leq 0.5$	supported

Table 13: R10 pre-registered decisions (protocol section 5, computed by the frozen reveal and combination programs). “Supported” and “falsified” are the protocol’s terms for these data; “not determinable” means that the protocol’s own evaluability condition failed.

	$k=2$ (106 tiers)	$k=5$ (81)	$k=10$ (67)
Bias ratio of the $k$ log residuals	0.796*	0.717*	0.667*
Sign consistency of the $k$ residuals	0.509*	0.645*	0.626*
Residual magnitude / source CV error	0.387*	0.488*	0.548*
5-NN support overlap (label-free)	$-0.071$	$-0.096$	$-0.131$
Quantile-interval width ratio (label-free)	$-0.026$	$-0.117$	$-0.083$
Regret: never calibrate	11.30	15.89	15.71

Table 14: Meta-analysis of few-shot adaptation under a frozen protocol over 106 leave-one-group tiers in five tables. Top: Spearman correlation across tiers between each decision-time diagnostic and the warm-start gain (median over 100 draws of the MAPE reduction against the uncalibrated source model); an asterisk marks a table-block bootstrap interval excluding zero after Benjamini–Hochberg with the same sign within every table. Bottom: mean held-out regret (percentage points above the post-hoc best method) of the bias-ratio threshold rule fitted on four tables and applied to the fifth, against two baselines. A tier enters at a given

k

when at least ten of its rows remain unmeasured.

Stratum	tiers	log-affine: reached / $k^{*}_{90}$	warm-start: reached / $k^{*}_{90}$	scratch: reached / $k^{*}_{90}$
all tiers	107	68% / 21.0	76% / 21.0	71% / 21.0
chemistry	33	88% / 17.8	94% / 5.0	94% / 15.0
laboratory	29	83% / 21.0	90% / 10.6	86% / 21.0
composition	11	82% / 21.0	91% / 5.0	82% / 21.0
device	30	30% / 21.0	40% / 21.0	30% / 21.0
emulsion type	2	50% / 19.2	50% / 20.4	50% / 19.2

Table 15: E2 model task: fraction of tiers whose adapted surrogate returns to

\varepsilon=1.25\times

dev-ref within 20 target rows, and the 90th percentile over tiers of the budget

k^{*}(\varepsilon)

(rows), by stratum and adaptation method (100 random draws per

k

; median over draws). A 90th percentile of 21.0 means that at least a tenth of the tiers did not reach

\varepsilon

within the budget. The noise-floor rule of E0 is not applied (Section 6.2.2 ). Emulsion-type and preparation strata have two tiers each.

Figure 11: Share of tiers whose warm-start surrogate is usable (

\leq 1.25\times

dev-ref on the unmeasured rows) after

k

target rows: left, by shift type (number of tiers in brackets); right, all tiers by adaptation method. The values at

k=20

are those of Table 15 . Steps are read off the E2 budget curves (100 random draws per

k

, median over draws); the noise-floor rule of E0 is not applied (Section 6.2.2 ).

Rule	transferred	table-specific	transfer regret [95%]	transferred $-$ R0 [95%]	under-planned	verdict
R0 full budget	18.07	18.07	—	—	0%	no advantage
R1 shift type	4.86	2.78	$+2.09$ [ $-0.48$ , $+7.62$ ]	$-13.21$ [ $-15.71$ , $-6.50$ ]	14%	transfers
R2 shift type $\times$ density	4.77	2.31	$+2.46$ [ $+0.50$ , $+7.53$ ]	$-13.31$ [ $-15.71$ , $-7.19$ ]	14%	does not transfer
R3 shift type $\times$ support diagnostic	4.32	2.48	$+1.84$ [ $+0.08$ , $+6.05$ ]	$-13.75$ [ $-15.74$ , $-8.18$ ]	17%	does not transfer
R4 shift type $\times$ bias ratio	5.28	2.36	$+2.93$ [ $+0.44$ , $+9.17$ ]	$-12.79$ [ $-15.50$ , $-5.55$ ]	19%	does not transfer

Table 16: E5: planned budgets under five-fold table rotation. Cost = rows measured beyond the budget the tier needed, or the full budget of 20 when the plan fell short (mean over the 81 tiers that reach

\varepsilon

within 20 rows). “Transferred” rules are learned on four tables and applied to the fifth; “table-specific” rules are learned on the held-out table itself. Regret = transferred

-

table-specific; intervals are table-level block bootstraps.

Policy	$k_{0}$	acquisition	adaptation	proposal	mean $T^{*}$	median	success	$\Delta$ vs P1 [95%]	verdict
P0	0	—	none	greedy	3.79	2.67	98.8%	$+0.48$ [ $-0.38$ , $+1.47$ ]	n.d.
P1 (reference)	2	random	warm-start	greedy	3.31	2.87	99.5%	—	—
P2	0	—	warm-start	greedy	2.80	2.00	100.0%	$-0.50$ [ $-0.87$ , $-0.10$ ]	better
P3	5	random	warm-start	greedy	3.75	3.43	99.8%	$+0.44$ [ $-0.07$ , $+0.95$ ]	n.d.
P4	2	farthest-point	warm-start	greedy	3.43	3.00	99.7%	$+0.12$ [ $-0.01$ , $+0.29$ ]	n.d.
P5	2	$k$ -means	warm-start	greedy	3.23	2.67	100.0%	$-0.075$ [ $-0.137$ , $-0.027$ ]	better, fragile

Table 17: E1 point task on the 107 tiers: the 14 loop policies, their mean and median time-to-target

T^{*}

(measurements; three targets

\times

10 seeds per tier;

\tau=0.10

B=20

), the success rate within the budget, and the difference to the reference policy P1 with its table-level block-bootstrap interval. Verdicts after Benjamini–Hochberg control over the 13 non-reference policies; “fragile” marks a sign change under a

\pm 20\%

oracle drift. Adaptation methods as in Section 5.12 ; the preview rule (

\dagger

) is the H-J threshold of Table 14 : warm-start when the two-row bias ratio exceeds 0.63, otherwise no adaptation. CEIL is the ceiling policy (Section 5.12 ).

Model	Macro MAPE	Worst group	Row MAPE	Point invalid	Interval invalid
B0 anchor equation	18.38%	28.31%	13.99%	1.29%	—
B1 four-input XGBoost	10.27%	19.96%	9.12%	3.22%	—
B2 native-input XGBoost	9.13%	16.94%	8.86%	2.58%	—
B3 native+physics MLP (unbounded)	16.64%	32.50%	14.45%	13.73%	0
V2-H bounded MLP	10.59%	14.63%	10.72%	0	0
V2-M V2-P + monotonic penalty	12.05%	14.84%	11.30%	0	0

Table 18: Frozen V2 model matrix under nested leave-one-chemistry-out evaluation (466 rows, six groups, five seeds; TX100 is post-reveal retrospective).

Panel of Figure 12 — Figure 12: Negative/mixed result. Paired difference in held-out-group MAPE, V2-P minus B1, for the six chemistry groups (negative favours V2-P). The TX100 group is a post-reveal retrospective analysis, not a blind test. V2-P wins only two groups, failing locked condition 2.

$\lambda$	fold rules (A1 / D1 / D2 / R2-A1 / R2-A2 held out)	rule	escalate $+\lambda$	oracle	rule $-$ continue [95%]	rule $-$ none [95%]	holds
0	always escalate in all five folds	2.34	2.34	2.31	$-0.96$ [ $-1.37$ , $-0.48$ ]	$-1.57$ [ $-2.48$ , $-0.67$ ]	yes, fragile
5	two folds: high bias ratio $\to$ switch, else continue; three folds: always continue	3.32	7.34	2.77	$+0.01$ [ $+0.00$ , $+0.03$ ]	$-0.59$ [ $-1.09$ , $-0.24$ ]	no
10	as for $\lambda=5$	3.32	12.34	2.82	$+0.01$ [ $+0.00$ , $+0.03$ ]	$-0.59$ [ $-1.10$ , $-0.24$ ]	no

Table 19: E3 decision rule after two random measurements, learned by five-fold table rotation and priced by the escalation cost

\lambda

(measurement-equivalents). Costs are mean

T^{*}

over the 107 tiers (three targets

\times

10 seeds), with

\lambda

added to every escalation; the constant arms, which share the two phase-A rows, cost 3.31 (continue, warm-start), 3.91 (none) and 3.71 (switch) at every

\lambda

(P0 of Table 17 starts without phase-A rows). “Oracle” is the best action chosen after the fact for every unit. The rule holds at

\lambda

when it beats both “always continue” and “never adapt” with block intervals excluding zero. The rule’s difference to “always continue” under drifted oracles is given in the text. Each interval comes from its own bootstrap resample, so identical rules can differ in the last digit.

Family	P0	P1	P2	$\Delta$ P2 vs XGBoost [95%]	lowest or tied (P2)	zero-shot MAPE
XGBoost (reference)	3.79	3.31	2.80	—	42%	20.2%
random forest	3.30	3.26	2.83	$+0.02$ [ $-0.18$ , $+0.17$ ] n.d., fragile	31%	18.8%
perceptron	4.23	3.34	3.16	$+0.36$ [ $+0.01$ , $+0.67$ ] n.d., fragile	31%	23.5%
quantile forest	3.10	3.24	2.83	$+0.02$ [ $-0.07$ , $+0.11$ ] n.d.	36%	18.4%
Gaussian process	4.73	3.55	3.41	$+0.61$ [ $+0.32$ , $+1.01$ ] worse	22%	22.7%
deep ensemble	4.37	3.41	3.25	$+0.45$ [ $-0.02$ , $+1.38$ ] n.d.	24%	22.9%

Table 20: E4: six source-model families inside the loop (107 tiers; three targets

\times

10 seeds per tier;

\tau=0.10

B=20

). Mean

T^{*}

under P0 (no adaptation), P1 (two random rows, warm-start) and P2 (warm-start at once); difference to XGBoost under P2 with table-level block interval and Benjamini–Hochberg verdict; share of tiers where the family has the lowest or tied-lowest

T^{*}

under P2 (ties are common); median zero-shot MAPE on the tier rows.

Figure 13: Mean time-to-target of the six source-model families without adaptation (P0), after two random rows and warm-start (P1), and with warm-start from the first measurement (P2); 107 tiers,

\tau=0.10

B=20

. Adaptation removes most of the spread between families that the unadapted models show.

Decision	E1 $(0.10,20)$	$(0.05,20)$	$(0.10,10)$	$(0.05,10)$	grade
P1 mean $T^{*}$	3.31	4.75	3.17	4.31	—
CEIL mean $T^{*}$ : $\delta=0$ ; $-0.2$ / $+0.2$	1.23; 7.28 / 5.81	1.55; 9.63 / 7.94	1.21; 6.17 / 5.07	1.45; 7.53 / 6.48	robust to difficulty
F14: P2 ( $k_{0}=0$ ) vs P1	$-0.50$ [ $-0.87$ , $-0.10$ ] better	$-0.66$ [ $-1.07$ , $-0.40$ ] better	$-0.45$ [ $-0.76$ , $-0.04$ ] better	$-0.50$ [ $-0.87$ , $-0.20$ ] better	robust to difficulty
F15: P5 ( $k$ -means) vs P1	$-0.075$ [ $-0.137$ , $-0.027$ ] better, fragile	$-0.14$ [ $-0.40$ , $+0.07$ ] n.d.	$-0.04$ [ $-0.12$ , $+0.14$ ] n.d., fragile	$-0.07$ [ $-0.16$ , $+0.08$ ] n.d., fragile	robust to difficulty
F16: P7 (log-affine) vs P1	$+0.49$ [ $+0.33$ , $+0.86$ ] worse	$+0.42$ [ $+0.29$ , $+0.83$ ] worse	$+0.37$ [ $+0.26$ , $+0.59$ ] worse	$+0.35$ [ $+0.26$ , $+0.62$ ] worse	robust to difficulty
F16: P8 (scratch) vs P1	$+0.79$ [ $+0.13$ , $+1.62$ ] worse	$+1.11$ [ $+0.24$ , $+2.37$ ] worse	$+0.73$ [ $+0.16$ , $+1.38$ ] worse	$+0.94$ [ $+0.25$ , $+1.76$ ] worse	robust to difficulty

Table 21: E1b difficulty sensitivity: the E1 decisions recomputed at a halved tolerance and a halved budget,

(\tau,B)

cells

(0.05,20)

(0.10,10)

and

(0.05,10)

beside E1 itself. Entries are

\Delta

vs P1 with table-level block intervals and the Benjamini–Hochberg verdict; a finding is “robust to difficulty” when it holds in all three new cells. The last two rows recompute the two acquisition-and-

k_{0}

decisions on the 18 dense tiers alone (the original eligibility gate of the first frozen protocol, Section 5.12 ).

Gain	where it was found	status
P2 warm-start at once	E1 and the three E1b cells, all tiers	4 confirmed
P2 warm-start at once	E1b dense tiers, two cells	2 fragile
P5 $k$ -means seeding	E1 all and dense tiers; E1b dense tiers	5 artefact
E3 rule “always escalate” at $\lambda=0$	E3	1 fragile
CEIL ceiling (refitted on the tier labels)	E1, E1b	8 positive control
any source-model family vs XGBoost	E4	none “better”

Table 22: E6: every “better” verdict (Benjamini–Hochberg) of E1, E1b, E3 and E4, classified by the two negative controls. Confirmed = sign unchanged under a

\pm 20\%

oracle drift and not “better” under shuffled labels; fragile = sign changes under the drift; artefact = also “better” under shuffled labels; positive control = label-informed by construction.

Figure 14: Difference in mean

T^{*}

to the reference P1 for four policies across the closed-loop settings, with table-level block intervals. Marker and colour give the status after both negative controls: better and confirmed (blue circle), better also under shuffled labels (orange cross), not distinguishable (open grey), worse (dark square).

Figure 15: Mean time-to-target against task difficulty, measured by the expected number of measurements a uniformly random proposer needs (dashed diagonal). Units of E1 (targets at the 20th, 50th and 80th label percentiles,

\tau=0.10

) and E7 (targets at the 5th, 10th, 90th and 95th,

\tau=0.03

) are pooled and binned by that difficulty (bin edges at 2, 3, 4, 6, 9, 13 and 17 measurements; markers at the bin means). On easy units every policy is close to chance; the gap between the warm-started loop and chance grows with difficulty.

Policy	mean $T^{*}$	success	$\Delta$ vs P1 [95%]	fragile	shuffled
P0 no adaptation	6.58	90%	$+0.07$ [ $-0.60$ , $+0.88$ ] n.d.	no	—
P1 two rows, warm-start (reference)	6.52	94%	—	—	—
P2 warm-start at once	5.72	93%	$-0.79$ [ $-1.37$ , $-0.09$ ] better	no	—
P3 five rows	7.97	94%	$+1.46$ [ $+0.82$ , $+1.94$ ] worse	no	—
P4 farthest-point	6.64	93%	$+0.12$ [ $-0.26$ , $+0.41$ ] n.d.	yes	—
P5 $k$ -means	7.05	93%	$+0.53$ [ $+0.30$ , $+0.97$ ] worse	no	—

Table 23: E7 on the 301 hard units (tail targets,

\tau=0.03

B=20

; at most 10% of a tier’s rows within tolerance of the target; 92 tiers; chance level 11.35 measurements). Mean

T^{*}

, success within the budget, difference to P1 with table-level block interval and Benjamini–Hochberg verdict, fragility under a

\pm 20\%

oracle drift, and the shuffled-label check.

	Finding	Q	tier	units
F1	The cost of shift is stratified by its type (geometry and literature, then chemistry, then device type and laboratory), and no fixed multiplier of the development reference describes it.	Q1	S, P	R2, R7, R8, R10
F2	With no target label, no statistic of the inputs separates reliable from unreliable rows at the candidate or query level; the gains of label-free screening are source-level, and a size proxy matches them under literature and laboratory shift.	Q1	F, R	R4, R7, R8, R9, R10
F3	Under shift, interval coverage is recovered only by widening (grouped calibration) or by abstaining (weighted conformal); no interval family keeps nominal coverage where supports do not overlap.	Q1	F	R3, R7, R8, R10
F4	Random-split accuracy and model rankings do not predict accuracy or rankings under shift; grouped cross-validation predicts chemistry shift but understates device and laboratory shift.	Q1	P	R7, R8, R10
F5	Output bounds remove physically invalid predictions and narrow the worst-group error, but do not bring chemistry-macro error below the published XGBoost baseline; the constraint form must match the flow regime.	Q4	F	R6, R9, R10
F6	On the revealed sealed sets a few target labels recover chemistry shift but not the device tier or laboratory B; across the 107 tiers most chemistry and laboratory tiers recover within twenty rows and most device tiers do not; the gain is usually positive and its size depends on the domain.	Q2	H, F	R10 constructive, E2

Table 24: Framework-level findings. Tier: S sealed, P pre-registered, F frozen protocol with two runs, H post hoc under a frozen protocol, R retrospective. Q: question (Q1–Q4), P provenance, M method. Units as in Table 1 .

Stage	What the evidence supports	findings
Before any target measurement	Do not read the random-split error or the random-split ranking of model families as the deployment estimate; expect a shift-type-dependent factor, largest for a new device type or a new laboratory. No label-free screen certifies individual candidates; use it, with a size control, to choose sources and to find support gaps. Do not expect nominal interval coverage to transfer. Run a bounded model beside every unbounded one.	F1–F5, F9, F13
Choice of source model	Once the loop adapts, no tested family beats XGBoost; the forests, the perceptron and the deep ensemble are not distinguishable from it and the Gaussian process is worse. Do not choose the family by its random-split error, and avoid a poorly converged Gaussian process as the source.	F13, F21
First measurements	Start proposing at once and warm-start the source model on every measurement; do not spend a calibration budget first (under laboratory shift the reference loop did as well). Choose rows at random and propose greedily: no label-free acquisition rule or interval-aware proposal helps, even on hard targets.	F14, F15, F16, F22, F23
Budget	Plan the budget from the shift type: a handful of rows under chemistry, composition or laboratory shift, while under device shift most tiers are not usable within twenty rows; a plan learned on other tables was not measurably worse than one learned on the new table. Expect many domains to be usable at the first row and some not within the budget.	F6, F18, F20
After two or four rows	The bias ratio says how far the model will move, not whether to stop adapting, switch or escalate; keep adapting unless a full recalibration is nearly free.	F7, F19, F23
Throughout	Keep measuring and updating: a frozen prior, however good, loses to a constant instrument scale error that an adaptive loop absorbs.	F17

Table 25: What the findings imply at each stage of a loop built on a learned surrogate, on tables like these.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Object	Artifact file(s)
Table 2	artifacts/reproduction_metrics.json .
Figure 3	artifacts/reliability_metrics.json (pilot pair; the artifact data behind artifacts/figure_cqr_coverage_shift.png ) and expanded_artifacts_canonical_run2/interval_summary.csv (expanded pair); re-rendered for v0.3 and extended for v0.3.3 by tools/render_figures.py .
Table 3	MAPE, MAE and $R^{2}$ : artifacts/reproduction_metrics.json ; $n$ : artifacts/reliability_metrics.json .
Table 4	artifacts/reliability_metrics.json ; expanded_artifacts_canonical_run2/applicability_binary_summary.csv and applicability_source_binary_summary.csv .
Figure 4	artifacts/figure_distance_vs_error.png (re-rendered for v0.3 by tools/render_figures.py from the artifact data behind that file; legend wording updated for v0.3.3).
Figure 5	artifacts/figure_pruning_precision.png (re-rendered for v0.3.3 by tools/render_figures.py from the tolerance table in artifacts/reliability_metrics.json ; legend wording updated).

Appendix

Table 41

Unit	splits	M0 (ours)	M4 (upstream, 1.3.3)
Random 80/20	15	5.268	5.247
Exact-input group	15	5.742	5.699
Geometry group	30	11.689	11.661
Geometry + fluid group	30	10.176	10.287
External (64 rows)	1	11.258	10.814

Appendix

Table 26: Reproduction fidelity on identical splits: mean MAPE (%) of our D1 constructor (M0, xgboost 3.3.0, explicit parameters) and of the upstream training line executed verbatim in xgboost 1.3.3 with its input scaler (M4). Units as in Section 5.2 ; the external unit is one fit on all 868 rows scored on the 64 literature rows.

Autonomous Experimentation

Autonomous experimentation uses AI, active learning, and robotics to design, run, and interpret experiments with less human intervention.

See all 148 papers

Jul 5, 2026

Compressing the Validation Bottleneck: An Agentic Self-Driving Lab for Scientific Discovery

Agentic AI-for-Science can automate ideation, planning, and analysis, but final validation still depends on real experiments. A self-driving lab (SDL) can execute those experiments, yet the loop still has bottlenecks: the agent may spend too many rounds on low-value experiments, or each round may require a high-cost experiment. We target these two physical bottlenecks with one agent. First, a prior-aware agentic DOE loop uses domain knowledge and past results to propose feasible and informative next experiments, reducing trials-to-target. Second, a cost-aware surrogate agent predicts high-cost, high-resolution measurements from low-cost, low-resolution measurements. It chooses between a high- and a low-cost measurement based on the predicted uncertainty. We examine these directions in the biology and materials domains, respectively. Together, under a single agent, these components aim to accelerate the SDL loop by reducing both the number of loops and the cost per experiment.

Kyunghoon Hur, Chihun Lee

AX Research Division, Korea Electronics Technology Institute, Republic of Korea · Material Data Division, Korea Institute of Materials Science, Republic of Korea

AI for Science Multifidelity Modeling Laboratory Automation Autonomous Experimentation Active Experimental Design

Ask PDF

Aug 2, 2026

When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design

Design campaigns in chemistry, materials science, and machine learning share a bottleneck: determining how good a candidate truly is requires an expensive evaluation - an experiment, a first-principles simulation, or a full training run. Machine-learning surrogates that predict these outcomes are increasingly used not only to propose candidates but to grade them, and even to feed their own predictions back into the search as though they were measurements. Through mathematical analysis validated on three exhaustively ground-truthed design tasks, we establish when this practice is safe, what any certificate of safety must cost, and when the substitution provably pays. Predictive accuracy cannot anchor trust: near-perfect R^2 is compatible with worst-possible selections, and screening N candidates inflates the over-prediction at the selected candidate by a quantifiable "selection tax" with matching upper and lower bounds. Safety follows instead from an architectural rule - predictions may propose and train without restriction, but every certified conclusion must rest on true evaluations - which is sufficient with no assumptions on the surrogate, and necessary, since admitting predictions into certification with the standing of measurements opens a deterministic self-confirmation failure mode. We derive the minimal criterion under which a model may act as an oracle (rank preservation, not accuracy), show that trust must be purchased through selection-aware audits that are optimal in query complexity, and prove a dichotomy fixing when audited surrogates cut certified evaluation cost. Across 432 surrogate fits over six task-regime conditions, the audit statistic tracks deployed search performance at Spearman rank correlation 0.80-0.99, while the rank correlation of R^2 with deployed regret falls as low as 0.33; audited screening reduces certified oracle cost by a measured factor of 25.

Shuangxiu, Ma, Wenhe +1

Max · William G. Lowrie Department of Chemical and Biomolecular Engineering, The Ohio State University, Columbus, Ohio 43210, United States · Zachary +1

Surrogate Modeling Surrogate-Assisted Optimization Selection Bias Black-Box Model Auditing

Ask PDF

Sep 7, 2026

Online Surrogate Repair: Decoupling High-Fidelity Feedback from Search Length in Closed-Loop Discovery

Closed-loop AI scientists can generate candidate designs at low marginal computational cost, whereas reliable feedback may require wet-lab synthesis, characterization, or high-fidelity computation. Addressing this imbalance through custom laboratory automation remains infrastructure-intensive and costly, while replacing new experiments with a fixed surrogate leaves persistent model errors that can be amplified by optimization. We propose online surrogate repair (OSR), a closed-loop algorithm that uses sparse high-fidelity evaluations to update the surrogate throughout a longer agent search conducted primarily with inexpensive surrogate feedback. An acquisition rule selects which designs from the agent's accumulated proposals receive high-fidelity evaluation, and the resulting labels update the surrogate used in subsequent episodes. Across controlled synthetic environments, we demonstrate that improving global surrogate fit does not necessarily reduce maximum regret, whereas Q90-UCB and expected improvement (EI) substantially reduce regret by directing evaluations toward regions that determine the optimizer's decisions. On MADE, controls receiving high-fidelity feedback after every episode require

6.36

7.23\times

more oracle queries to match Online EI under two LLM orchestrators and

10.27\times

more under the non-LLM Chemeleon+MLIP workflow. Online surrogate repair introduces a novel third feedback regime between fixed-surrogate operation and high-fidelity feedback after every episode, separating the frequency of high-fidelity evaluation from the duration of the agent's search.

Xiaotang Feng, Philip Torr, Bruno Andreis

Department of Engineering Science, University of Oxford · Department of Physics, University of Oxford · Slater Labs

AI Agents for Scientific Discovery Surrogate-Assisted Optimization Adaptive Sampling

Ask PDF

Abstract

Autonomous Experimentation

Explore similar work

Compressing the Validation Bottleneck: An Agentic Self-Driving Lab for Scientific Discovery

When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design

Online Surrogate Repair: Decoupling High-Fidelity Feedback from Search Length in Closed-Loop Discovery