MECHVAR: Variance-Guided Mechanism Discrimination for Autonomous Machine Learning Experiment Selection

Yifan Guo

cs.LGOct 1, 2026

MECHVAR: Variance-Guided Mechanism Discrimination for Autonomous Machine Learning Experiment Selection

Authors: Yifan Guo

Organizations: Stony Brook University

Abstract

Benchmark gains are often mechanism-ambiguous: reproducing an improvement does not by itself identify why it occurs. We study finite-library mechanism discrimination, where posterior-weighted candidate mechanisms, executable probes, and a limited experimental budget define a sequential experiment-selection problem. MECHVAR selects the next probe by maximizing the posterior-weighted variance of its predicted responses. Under a shared-Gaussian predictive model, this score is exactly proportional to the classical Box--Hill posterior-weighted pairwise-KL criterion, yet it admits O(KE) vectorized rescoring and a transparent additive audit over mechanism pairs. A local expansion further links the score to expected information gain (EIG) when predicted response separations are small. In a 25-block stress audit, MECHVAR outperforms confirmation-first in several moderate misspecification regimes, while its primary comparisons with EIG remain statistically unresolved. In a held-out Digits loop, normalized mechanism-identification AUC is 0.8975 for MECHVAR, 0.7825 for a score-greedy policy, and 0.9092 for EIG. At K = 100, E = 200, median single-thread full-library scoring is 10.36 microseconds for MECHVAR versus 57.69 ms for six-node quadrature EIG in the recorded environment. MECHVAR therefore provides a lightweight, auditable acquisition rule for finite-library experiment selection when a shared predictive scale is a defensible approximation.

Figures & tables

Figure 1: Finite-library mechanism-discrimination loop. The posterior and response templates determine the MDV score; the pairwise decomposition records which mechanism pairs contribute most to the selected probe. Only the executed outcome is revealed before the posterior update.

Figure 2: Full misspecification sweeps. Left: prediction misspecification. Right: likelihood-noise mismatch. The clearest resolved differences from confirmation-first occur in the moderate misspecification regimes highlighted in the text, while the primary MECHVAR–EIG comparisons remain statistically unresolved.

Condition	Random	Confirm-first	EIG	MECHVAR	$\Delta_{M-C}$ (Holm $p$ )	$\Delta_{M-E}$ (Holm $p$ )
Well specified	95.92	99.68	100.00	100.00	$+0.32$ $(0.228)^{\dagger}$	$0.00$ (–)
Pred. $\sigma=.5$	75.28	91.12	94.08	94.32	$+3.20$ (0.00196)	$+0.24$ (1.000)
Pred. $\sigma=1$	50.32	63.84	64.64	65.36	$+1.52$ (0.497)	$+0.72$ (1.000)
Noise $2\times$	86.32	97.76	99.12	99.20	$+1.44$ (0.0276)	$+0.08$ (1.000)
Noise $3\times$	73.84	92.08	94.24	94.72	$+2.64$ (0.00800)	$+0.48$ (1.000)

Table 1: Selected stress-audit conditions. Accuracy columns report mechanism-identification percentages. Parentheses show family-wise Holm-adjusted Wilcoxon

p

-values for the indicated comparator.

Panel of Figure 3 — Figure 3: Left: mechanism-set scaling at fixed $B=4$ . Right: freshly rerun single-thread acquisition-score timing with cached six-node Gauss–Hermite nodes. Runtime is environment-specific; the widening gap reflects the different online scoring costs as $K$ grows.

Figure 4: Held-out Digits-Shortcut loop. Score-greedy optimizes expected benchmark accuracy yet identifies the hidden mechanism more slowly; MECHVAR and EIG both reach perfect mechanism identification by budget $B=2$ .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Mechanism pair	$c_{hh^{\prime}}(e)$	Share of MDV
$(h_{1},h_{2})$	0.05400	79.9%
$(h_{1},h_{3})$	0.00625	9.2%
$(h_{2},h_{3})$	0.00735	10.9%

Appendix

Table 2: Illustrative audit of one probe. The three pair contributions sum exactly to the MDV score.

Figure 5: Representative diagnostics. Left: shortcut exposure. Middle: structural controls. Right: semantic class permutation leaves MSP OOD discrimination nearly unchanged.

ID	Domain	Key diagnostic result	Interpretation
FB01	ColoredMNIST	OOD accuracy $0.2575\rightarrow 0.8612$ under color swap; $\Delta=+0.6028$ (95% CI $[0.5588,0.6469]$ ).	Shortcut dependence supported
FB02	CIFAR-100 SSL	Supervised-200 $0.0493$ vs. FixMatch $0.0225$ ; difference $-0.0268$ (95% CI $[-0.0677,0.0140]$ ).	No support under this protocol
FB03	IHDP causal inference	Targeted regularization improves DragonNet PEHE by $0.1346$ , while mean PEHE still favors TARNet ( $1.2581$ vs. $1.3852$ ).	Mixed component-level evidence
FB04	50-node causal discovery	SHD: NOTEARS $79.7$ ; correlation $111.7$ ; permuted $160.0$ ; random $162.7$ .	Structural sensitivity supported
FB05	Poisoned MNIST	Harmonic clean/robust gain: noise-mix $+0.0122$ (95% CI $[0.0027,0.0217]$ ); FGSM $+0.0475$ ; trigger gain approximately zero.	Partial robustness support
FB06	CIFAR-10 $\rightarrow$ SVHN OOD	Accuracy $0.9227\rightarrow 0.0385$ under class permutation while $\|\Delta\mathrm{MSP\text{-}AUROC}\|\leq 8.62\times 10^{-7}$ .	Semantic/OOD dissociation

Appendix

Table 3: Six diagnostic studies used to motivate the mechanism-identification objective. Planner performance is evaluated separately in the main experiments.

Figure 6: Mechanism-identification accuracy under five structured forecast-error models at scale $0.5$ , with $K=5$ , $E=10$ , and $B=4$ .

Error structure	Random	Confirm-first	EIG	MECHVAR
IID Gaussian	.769	.938	.944	.948
Probe-shared bias	.838	.951	.976	.978
Mechanism-shared bias	.775	.924	.934	.933
Low-rank correlated	.757	.901	.910	.918
Heavy-tail $t_{3}$	.820	.906	.938	.931

Appendix

Table 4: Mean mechanism-identification accuracy across 20 paired blocks under five structured forecast-misspecification models. This sweep is exploratory; no confirmatory pairwise inference is attached to the numerical ordering.

Nodes	Top-probe agreement	Median Spearman	Median max error	95th-pct. max error
6	0.997	1.000	$3.84\times 10^{-4}$	$7.99\times 10^{-4}$
20	1.000	1.000	$3.62\times 10^{-7}$	$3.16\times 10^{-6}$

Appendix

Table 5: Gauss–Hermite convergence over 1,000 synthetic planner states. Score errors are in nats and are measured against 50-node quadrature.

$\delta$	Top-probe agreement	Median Spearman	Median BH loss	95th-pct. BH loss
0.00	1.000	1.000	0.000	0.000
0.10	0.812	0.976	0.000	0.134
0.25	0.560	0.855	0.000	0.425
0.50	0.257	0.539	0.359	0.796

Appendix

Table 6: Synthetic sensitivity to mechanism–probe-dependent predictive scales. Relative BH loss is the fractional gap between the optimal heteroscedastic Box–Hill score and the score attained by the MECHVAR-selected probe.

$K$	$E$	MECHVAR ( $\mu$ s)	EIG ( $\mu$ s)	EIG/MECHVAR
5	10	2.90	129.41	44.6 $\times$
8	16	2.92	243.28	83.2 $\times$
12	24	3.03	463.51	152.8 $\times$
20	40	3.29	1134.53	345.4 $\times$
30	60	3.73	2674.41	718.0 $\times$
50	100	4.89	8958.05	1831.0 $\times$

Appendix

Table 7: Single-thread acquisition-score timing. EIG uses six-node Gauss–Hermite quadrature; values exclude model fitting and probe execution.

Policy	$B=0$	$B=1$	$B=2$	$B=3$	$B=4$
Random	.4067	.8290	.9486	.9817	.9946
Score-greedy	.4067	.6200	.9067	.9133	.9733
Confirm-first	.4067	.8067	.9933	1.0000	1.0000
EIG	.4067	.9333	1.0000	1.0000	1.0000
MECHVAR	.4067	.8867	1.0000	1.0000	1.0000

Appendix

Table 8: Held-out Digits-Shortcut mechanism-identification accuracy across budgets. Budget

0

denotes the posterior after the native run and before any diagnostic intervention is executed.

Figure 7: Normalized budget-AUC for leave-one-seed-out identification of empirical intervention-response profiles using context-derived priors.

Explore similar work

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer
Jul 19, 2026Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2Research Automation Auto Research
School of Computer Science, Carnegie Mellon University · 2DP Technology
Test-Time Scaling via Budgeted Multi-Attribute Verification
Sep 28, 2026Bo Xue, Ji Cheng, Shen-Huan Lyu +2Token Budget Allocation Large Language Model Responses
City University of Hong Kong · Hohai University · Zhejiang University
Experimental Experience Modeling for Autonomous Research
Sep 30, 2026Wenda Wei, Yingchen Zhang, Ruqing Zhang +3Bayesian Experimental Design Prioritized Experience Replay
State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences +1

Jul 19, 2026, cs.MAJ/K move · Enter open · S save

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Jingjie Ning, Xiaochuan Li, Shanshan Zhong, Ji Zeng+1

School of Computer Science, Carnegie Mellon University · 2DP Technology

Research Automation Auto Research Reusable Knowledge Feedback Materials

Open paper AskPDF arXiv

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Abstract

Explore similar work

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Test-Time Scaling via Budgeted Multi-Attribute Verification

Experimental Experience Modeling for Autonomous Research