A budget-dependent crossover between coverage- and response-based training-set selection for machine-learned interatomic potentials

Jia Bi; Alin-Marin Elena

cs.LGSep 5, 2026

A budget-dependent crossover between coverage- and response-based training-set selection for machine-learned interatomic potentials

Authors: Jia Bi, Alin-Marin Elena

Organizations: Science and Technology Facilities Council, Harwell Campus, Didcot, United Kingdom. · Science and Technology Facilities Council, Keckwick Lane, Daresbury, United Kingdom.

Abstract

Selecting compact training sets for machine-learned interatomic potentials requires deciding whether to preserve structural diversity or target configurations on which models disagree. The better choice can depend on how much data is retained, making a comparison at one training-set size insufficient. Here we link selection criteria to prediction accuracy through a budget-resolved comparison of retrained MACE models on GAP-20 Carbon and pooled revised MD17. Structural coverage is compared with a response-guided selector that targets disagreement between a coverage-trained model and a full-data reference. This retrospective response witness tests the value of model disagreement for compressing an already labelled pool. At 5%, coverage gives smaller absolute deviations from the full-data error than random sampling across four force endpoints in both datasets. The witness has larger deviations than coverage at 1% and 5%, but the ordering reverses at 20%. At 20%, witness-selected models also lower direct held-out force errors by 0.46--5.89% relative to coverage, with all eight paired training-seed intervals favouring the witness. Six errors fall below the full-data reference. Mean force-error reductions are 0.164--0.167~~meV~~ $\textÅ^{-1}$ , with larger gains for tail and masked endpoints. Complementary analyses show that learned similarity preserves the coverage ranking, while selecting by frozen-model error gives higher error than embedding coverage. These findings establish retained-data budget as a deciding variable in atomistic training-set selection and provide a direct test of when response-guided compression improves on structural coverage.

Figures & tables

Fig. 1: Matched-budget retraining separates structural coverage from response-guided data value. a , Representative schematics for the rMD17 molecular trajectories, GAP-20 and defect-Carbon pools, and the high-entropy-alloy analysis. b , Competing selections from the same labelled pool at a fixed budget

B

: coverage favours diversity in representation space, the retrospective response witness favours disagreement with the full-data reference, and Random provides an unstructured control. c , Each selected subset trains a separate MACE model and is evaluated on held-out DFT force endpoints alongside the full-data reference. d , A schematic summary of the observed ordering introduces the central question: whether selector ordering changes between 1%, 5% and 20% retained data. Quantitative estimates and paired-seed intervals are reported in Fig. 2 and Tables 1 and 2 .

Fig. 2: The response witness has larger absolute gaps at low budgets and smaller gaps at 20%. a,b , Absolute gaps to the full-data outcome for Carbon and pooled rMD17 at 1%, 5% and 20%, for the Mean, Validation-selected stratum, High-force tail and High-novelty mask endpoints. Points are five-seed means and lines are visual guides. These panels show point estimates; paired uncertainty for the 20% comparison is shown in c . c , Response-witness minus FPS/DPP absolute-gap differences at 20%. Markers show five-seed means and error bars show 95% percentile-bootstrap intervals from all paired resamples of the five seeds. Negative values indicate that the witness is closer to the full-data outcome. Table 2 reports the direct held-out force errors.

Dataset	Endpoint	Random $-$ FPS/DPP (pp)	95% interval (pp)
Carbon	Mean	$+0.303$	$[+0.111,+0.411]$
Carbon	Validation-selected stratum	$+0.251$	$[+0.125,+0.329]$
Carbon	High-force tail	$+0.931$	$[+0.321,+1.291]$
Carbon	High-novelty mask	$+0.470$	$[+0.182,+0.640]$
rMD17 full	Mean	$+0.244$	$[+0.084,+0.356]$
rMD17 full	Validation-selected stratum	$+0.197$	$[+0.085,+0.263]$

Table 1: FPS/DPP has smaller absolute gaps than Random at the 5% budget. Positive estimates mean that Random is farther from the full-data outcome. The 95% intervals enumerate all paired percentile-bootstrap resamples of five matched training seeds and describe variation across those seeds.

Dataset	Endpoint	Full data	FPS/DPP	Witness	Witness $r_{g}$ (95% CI), %	Difference (95% CI)
Carbon	Mean	36.162	36.380	36.214	$+0.144$ $[+0.127,+0.161]$	$-0.167$ $[-0.186,-0.146]$
Carbon	Validation-selected stratum	34.605	34.944	34.619	$+0.039$ $[+0.019,+0.059]$	$-0.325$ $[-0.347,-0.301]$
Carbon	High-force tail	37.015	38.555	36.672	$-0.927$ $[-1.132,-0.763]$	$-1.883$ $[-2.050,-1.745]$
Carbon	High-novelty mask	40.535	41.480	40.425	$-0.273$ $[-0.308,-0.240]$	$-1.056$ $[-1.099,-1.007]$
rMD17 full	Mean	22.519	22.659	22.495	$-0.106$ $[-0.139,-0.073]$	$-0.164$ $[-0.183,-0.146]$
rMD17 full	Validation-selected stratum	22.817	22.982	22.781	$-0.160$ $[-0.188,-0.133]$	$-0.201$ $[-0.217,-0.186]$

Table 2: Response-witness selection lowers direct held-out force error at the 20% budget. Errors are five-seed means in meV

\text{\AA}^{-1}

. The signed gap

r_{g}

is the percentage deviation of the witness error from the matched full-data error. Relative gaps are calculated within each seed before averaging and need not equal ratios of the displayed mean errors. Differences are response witness minus FPS/DPP. Brackets give 95% percentile-bootstrap intervals from all paired resamples of five matched seeds. Negative differences favour the witness over FPS/DPP; negative

r_{g}

indicates lower error than full data. Supplementary Table gives the seed values and paired-

t

sensitivity intervals.

Fig. 3: Response emphasis improves one local rMD17 endpoint but increases the other reported errors at 5%. a , Response-witness effects relative to FPS/DPP across four held-out force-error endpoints. b , Response-repair effects relative to FPS/DPP. Negative values indicate lower error; error bars are grouped-bootstrap 95% confidence intervals. Endpoint masks can overlap and are not independent replications. The horizontal scales differ between panels. Carbon repair replaces three of 50 structures, reduces candidate-pool OBNL atom mass from 1.00% to zero and retains a Jaccard overlap of 0.887. rMD17 repair returns the coverage subset because FPS/DPP already covers every eligible high-response cell.

Fig. 4: Frozen-model error tracks embedding novelty, and error-based selections give higher Carbon-defect MAE than coverage. a , Spearman correlations of raw and residualised force error with five-nearest-neighbour novelty in the mean-pooled MACE-MP-0 embedding. b , Out-of-fold variance explained by novelty alone and by novelty plus nuisance covariates. Points and error bars in a and b show estimates and grouped-bootstrap 95% confidence intervals. c , Held-out Carbon-defect force-vector MAE for five training seeds. Small points show individual seeds; large symbols and error bars show five-seed means and reported 95% confidence intervals.

Fig. 5: Learned chemical and physical similarity preserves the coverage ranking. a , Selected-set overlap with Gaussian QUESTS across retained-data budgets; annotations report full-ranking Spearman correlations. b , Learned contributions from geometry and coverage, species and composition, and the residual channel. c , Learned-minus-Gaussian leave-one-group-out

\Delta

AUROC with grouped-bootstrap 95% confidence intervals. All three intervals include zero.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. S1: Learned chemical/physical similarity remains aligned with Gaussian QUESTS coverage. a–c , Top-budget Jaccard overlap from 1% to 20% for Carbon, rMD17 full and HEA, respectively. d , Learned channel weights across datasets. e , Leave-one-group-out AUROC for Gaussian QUESTS, fixed-bank similarity and the learned metric. f , Held-out-group learned-minus-Gaussian

\Delta

AUROC values; black markers show grouped-bootstrap means and 95% confidence intervals.

Level	Cumulative representation	Normalised squared error (95% CI)
G0	Composition, cell and generation controls	$1.000$ $[1.000,\,1.000]$
G1	G0 + two-body radial spectrum	$0.720$ $[0.669,\,0.805]$
G2	G1 + three-body angular spectrum	$0.500$ $[0.466,\,0.560]$
G3	G2 + ACE invariants	$0.320$ $[0.290,\,0.371]$
G4	G3 + frozen MACE embedding	$0.240$ $[0.222,\,0.278]$

Appendix

Table S2: Normalised out-of-fold squared error for non-spin HEA energy prediction. The regression target and error normalisation are defined in Eqs. S9 and S10 , and the cumulative G0–G4 feature sets are specified in the descriptor-strength ladder. Values are out-of-fold residual sums of squares normalised to G0; intervals are 95% grouped-bootstrap intervals over the eight FCC/BCC–generation-class groups.

Fig. S2: Foundation-model error analysis. a , Out-of-fold variance explained by embedding novelty alone and by novelty plus nuisance covariates across rMD17, amorphous Carbon and Carbon defects. b , Raw and residualised error correlations with novelty. In a and b , points and error bars show the reported estimates and 95% confidence intervals. c , Carbon-defect force-MAE differences from embedding coverage: open points show all five paired training seeds and filled points with error bars show the reported aggregate differences and 95% confidence intervals. Force MAE is in meV

\text{\AA}^{-1}

Fig. S3: Response-cell and repair sensitivity. a , Candidate-pool OBNL atom mass as the response threshold varies for Carbon and rMD17. b , Five-seed mean Carbon effects relative to FPS/DPP across realised replacement fractions of 2%, 6% and 10%; curves are labelled by endpoint. c , Selected-cell composition and repair-versus-coverage Jaccard overlap at the primary 5% budget. Carbon retains 94% of the coverage subset and replaces 6% with OBNL-cell representatives; rMD17 retains 100%. Cells and thresholds are fitted on training calibration and projected to the disjoint candidate pool. Panels show point estimates from the sensitivity analysis.

Fig. S4: Outcome gaps for seeds 1–5 at the 5% budget. Carbon ( a–d ) and rMD17 ( e–h ) panels show seed-level values and five-seed means for Random, FPS/DPP, QUESTS-MSC, response witness and response repair across the Mean, Validation-selected stratum, High-force tail and High-novelty mask endpoints. Grey lines join Random and FPS/DPP values from the same training seed. Lower absolute gaps indicate closer agreement with the full-data training outcome.

Fig. S5: Random and FPS/DPP outcome gaps across budget. Carbon and rMD17 panels compare the two baselines at 1%, 5% and 20% for four absolute gap endpoints. Points are five-seed means and lines are visual guides; the pale band marks the primary 5% budget, not an uncertainty interval. Error bars in the FPS/DPP mean panels show the reported 95% intervals.

Dataset	Budget	Selector	Mean	Validation-selected stratum	High-force tail	High-novelty mask
Carbon	$1\%$	Random	$6.50\%$	$7.22\%$	$16.50\%$	$8.00\%$
Carbon	$5\%$	Random	$2.10\%$	$2.42\%$	$11.45\%$	$6.02\%$
Carbon	$20\%$	Random	$0.63\%$	$0.95\%$	$4.35\%$	$2.25\%$
Carbon	$1\%$	FPS/DPP	$2.05\%$	$2.72\%$	$14.80\%$	$7.30\%$
Carbon	$5\%$	FPS/DPP	$1.80\%$	$2.17\%$	$10.52\%$	$5.55\%$
Carbon	$20\%$	FPS/DPP	$0.60\%$	$0.98\%$	$4.16\%$	$2.33\%$

Appendix

Table S6: Direct outcome gaps across budget. Values are five-seed means relative to the full-data training outcome; lower absolute gaps indicate closer agreement. “Validation-selected stratum” denotes the coded stratum fixed on validation data.

Panel	Dataset/method	Endpoint	Change vs FPS/DPP	95% CI	CI half-width	Interpretation
A	Carbon	Mean gap	$+55.474$	$[+53.908,\,+56.795]$	$1.443$	Witness farther from full-data outcome
A	Carbon	Validation-selected stratum gap	$+58.565$	$[+56.943,\,+59.875]$	$1.466$	Witness farther from full-data outcome
A	Carbon	High-force tail gap	$+44.728$	$[+43.245,\,+46.189]$	$1.472$	Witness farther from full-data outcome
A	Carbon	High-novelty mask gap	$+48.654$	$[+47.124,\,+49.902]$	$1.389$	Witness farther from full-data outcome
A	rMD17	Mean gap	$+22.458$	$[+21.492,\,+23.219]$	$0.863$	Witness farther from full-data outcome
A	rMD17	Validation-selected stratum gap	$+21.156$	$[+20.172,\,+21.888]$	$0.858$	Witness farther from full-data outcome

Appendix

Table S7: Confidence intervals for outcome contrasts at the 5% budget. Panel A reports response-witness minus FPS/DPP absolute-gap differences; Panel C reports Random minus FPS/DPP absolute-gap differences. Both enumerate all

5^{5}

paired training-seed bootstrap resamples. Panel B reports relative response-risk effects with the grouped intervals from Supplementary Table S4 . Panels A and C are in percentage points; Panel B is in percent. The CI half-width is one half of the unrounded 95% interval width.

Fig. S6: The learned metric responds to an injected independent value channel. a , Evaluation AUROC for the coverage baseline and learned metric under null, positive and shuffled controls. b , Learned-minus-coverage

\Delta

AUROC with paired-bootstrap 95% confidence intervals. c , Spearman rank correlation with the coverage ordering. d , Top-5% Jaccard overlap with the coverage subset. e , Learned channel weights under the three control settings. f , Top-5% enrichment of the target group; the dashed line marks no enrichment. This methodological control uses generated features with a known injected signal and disjoint fitting and evaluation pools.

Contrast	Evidence at 5%	Evidence at 1% and 20%	Supported interpretation
Random $-$ FPS/DPP absolute gap	Five paired seeds for all four endpoints; fully enumerated $5^{5}$ paired-seed bootstrap	Five-seed means are reported at both budgets; no additional Random–FPS/DPP interval is shown	Direct paired interval at 5%; descriptive ordering at 1% and 20%
Witness $-$ FPS/DPP absolute gap	Five paired seeds for all four endpoints; $5^{5}$ paired-seed bootstrap	Five-seed means at 1%; paired-seed interval at 20%	Paired-seed contrasts at 5% and 20%
Witness or repair $-$ FPS/DPP risk	Grouped effect intervals for all four endpoints	Five-seed means are reported at both budgets; witness direct DFT-risk intervals are shown at 20%	Endpoint trade-offs at 5%; lower witness DFT risk across all eight 20% comparisons
Witness signed gap from full-data risk	Not used for the primary 5% contrast	Five matched seeds with fully enumerated paired bootstrap and paired- $t$ intervals at 20%	Witness risk below full-data risk in six of eight 20% comparisons
FPS/DPP gap to full-data outcome	Reported mean-gap intervals	Reported mean-gap intervals at 1% and 20%	Uncertainty in gap magnitude; direct 20% direction is assessed through paired DFT risks
Repair $-$ FPS/DPP absolute gap	Five paired-seed summaries; grouped risk effects	Five-seed means are reported at both budgets; no additional direct interval is shown	Selection identity is established only for rMD17 at 5%

Appendix

Table S10: Uncertainty estimates for the outcome comparisons. Every budget comprises five training seeds. Grouped intervals resample the stated scientific groups with paired seeds; paired-seed intervals resample the five matched training seeds. The primary 5% coverage contrast and the 20% response crossover have paired-seed intervals. The 1% comparison reports the ordering of five-seed means.

Artifact	Location	Scope
Main-text source tables	data/main/	CSV tables for main-text figures and tables, including the 20% paired response contrast
Supplementary source tables	data/si/	CSV tables including seed-level and budget-sensitivity analyses
Method metadata	method_metadata/	Endpoint-membership and calibration metadata, physical-group definitions and coverage-comparator choices
Release metadata	README.md , DATA_DICTIONARY.md , VALUE_PROVENANCE.md , FILE_INVENTORY.csv	Definitions, recorded and derived values, and file inventory
Analysis and tests	analysis/ ; reproduce.sh	Fully enumerated paired-bootstrap and paired- $t$ comparisons; no model training
Quantitative-figure code	plotting/	Portable renderers for Main Figs. 2–5 and the six Supplementary Figures

Appendix

Table S12: Source-data and analysis archive scope. Locations are relative to the archive root. The reproduction entry point checks paired-statistic arithmetic and agreement with the corresponding source tables.

Item	Archived source-data file(s)
Main Fig. 1	Author-supplied conceptual workflow; no numerical source table
Main Fig. 2	data/si/figS6_budget_sensitivity.csv ; data/main/table2_20pct_response_crossover.csv
Main Fig. 3	data/main/fig5_witness_effects.csv ; data/main/fig5_repair_effects.csv ; auxiliary OBNL quantities in data/main/fig5_obnl_overlap.csv
Main Fig. 4	data/main/fig4_foundation_summary.csv ; data/main/fig4_finetuning_outcomes.csv ; data/si/figS3_seed_outcomes.csv ; data/si/tableS3_foundation_error.csv
Main Fig. 5	data/main/fig2_rank_summary.csv ; data/si/figS1_budget_summary.csv ; data/main/fig2_weights.csv ; data/main/fig2_logo_auroc.csv ; data/main/fig2_logo_delta.csv
Main Table 1	data/si/figS5_seed_strata.csv ; regenerated in analysis/generated/table1_5pct_random_vs_coverage.csv

Appendix

Table S13: Result-figure and result-table source-data map. Paths are relative to the aggregate Zenodo release. Wildcards denote all CSV files with the stated prefix. Historical file basenames are retained for compatibility and need not match the current figure or table number; the data dictionary maps legacy fields to the displayed endpoints.

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 186 papers

Risk-Controlled DFT Selection for Machine-Learned Interatomic Potentials
Oct 14, 2026Kun Ryu, Yeo Jin Jung, Shyue Ping Ong +2Selective Prediction Computational Materials Science
Pritzker School of Molecular Engineering, The University of Chicago · Department of Statistics, The University of Chicago · Department of Material Science and Engineering, National University of Singapore
Data Curation for Machine Learning Interatomic Potentials by Determinantal Point Processes
Mar 23, 2026Joanna Zou, Youssef MarzoukTraining Data Curation Training Data Selection
Computational Science & Engineering Massachusetts Institute of Technology Cambridge, MA, 02139, USA
Flexible Cutoff Learning: Optimizing Machine Learning Potentials After Training
Mar 10, 2026Rick Oerder, Jan HamaekersMachine Learning Interatomic Potentials
Institute for Numerical Simulation, University of Bonn · Fraunhofer Institute for Algorithms and Scientific Computing SCAI · Institute for Numerical Simulation, University of Bonn, Friedrich-Hirzebruch-Allee 7, 53115 Bonn, Germany +1

Oct 14, 2026J/K move · Enter open · S save

Risk-Controlled DFT Selection for Machine-Learned Interatomic Potentials

Kun Ryu, Yeo Jin Jung, Shyue Ping Ong, Claire Donnat+1

Pritzker School of Molecular Engineering, The University of Chicago · Department of Statistics, The University of Chicago · Department of Material Science and Engineering, National University of Singapore

Selective Prediction Computational Materials Science Machine Learning Interatomic Potentials

Open paper AskPDF arXiv

Automated materials discovery requires evaluating large pools of candidate configurations, with density functional theory (DFT) as a major throughput bottleneck. Pretrained machine-learned interatomic potentials (MLIPs) approximate DFT cheaply, but their errors vary by orders of magnitude across configurations, making it difficult to decide when to trust the MLIP predictions and when to fall back to DFT. We frame this decision as a risk-controlled selection problem and solve it within a conformal risk control (CRC) framework. Each configuration is mapped from atomic numbers and Cartesian coordinates to a fixed geometric descriptor, which a lightweight predictor uses to estimate an MLIP-DFT disagreement score in energy, average force, and maximum force. CRC then calibrates this score into a selection threshold for DFT labeling such that the MLIP error on the unselected configurations, which keep their MLIP labels, stays below a user-chosen fraction

α

of the expected error. Under an identical guarantee across eight dataset-MLIP pairs, our predictor routes 7% and 14% fewer configurations to DFT at

α=0.3

than committee baselines. Because committee disagreement overlooks errors the MLIPs share, our gains are largest when the DFT labels and MLIP training data differ in DFT functional and basis set, and narrow when the two agree. Without model access, the structure-only predictor also ranks errors as well as the target MLIP's latent representation and outperforms widely used descriptors such as Smooth Overlap of Atomic Positions (SOAP) and Atom-Centered Symmetry Functions (ACSF).

Abstract

Training Data Selection

Explore similar work

Risk-Controlled DFT Selection for Machine-Learned Interatomic Potentials

Data Curation for Machine Learning Interatomic Potentials by Determinantal Point Processes

Flexible Cutoff Learning: Optimizing Machine Learning Potentials After Training