When should we trust the annotation? Selective prediction for molecular structure retrieval from mass spectra

Mira Jürgens; Gaetan De Waele; Morteza Rakhshaninejad; Willem Waegeman

cs.LGMar 11, 2026

When should we trust the annotation? Selective prediction for molecular structure retrieval from mass spectra

Authors: Mira Jürgens, Gaetan De Waele, Morteza Rakhshaninejad, Willem Waegeman

Organizations: Department of Data Analysis and Mathematical Modeling, Ghent University, Coupure Links 653, Ghent, 9000, Belgium. · Department of Computer Science, University of Antwerp, Middelheimlaan 1, Antwerp, 2020, Belgium.

Abstract

Machine learning methods for identifying molecular structures from tandem mass spectra (MS/MS) have advanced rapidly, yet current approaches still exhibit significant error rates. In high-stakes applications such as clinical metabolomics and environmental screening, incorrect annotations can have serious consequences, making it essential to determine when a prediction can be trusted. In this work, we introduce a selective prediction framework for molecular structure retrieval from MS/MS spectra, separating low-risk predictions automatically from lower-confidence predictions. We formulate the problem within the risk-coverage tradeoff framework and systematically evaluate uncertainty quantification strategies at three levels: input-level distance in the learned representation, fingerprint-level uncertainty over predicted molecular fingerprint bits, and retrieval-level uncertainty over candidate rankings. We compare scoring functions including first-order confidence measures, aleatoric and epistemic uncertainty estimates from second-order distributions, as well as distance-based measures in the latent space. All experiments are conducted on the MassSpecGym benchmark. Our analysis reveals that while fingerprint-level uncertainty scores are poor proxies for retrieval success, retrieval-level total uncertainty provides the overall strongest rejection criterion, and first-order confidence measures are computationally inexpensive, strong baselines. We demonstrate that by applying distribution-free risk control via generalisation bounds, practitioners can specify a tolerable error rate and obtain a subset of annotations satisfying that constraint with high probability.

Figures & tables

Figure 1: Overview of the selective prediction framework for molecular structure retrieval from tandem mass spectra. (a) Fingerprint-based molecular retrieval. A tandem mass spectrum $\boldsymbol{x}$ is mapped by a model $f$ to a vector of bitwise probabilities $\boldsymbol{\theta}\in[0,1]^{4096}$ . Each candidate fingerprint $\boldsymbol{c}_{j}$ in the instance-specific candidate set $\mathcal{C}_{i}$ is scored by cosine similarity $s_{j}=\mathrm{sim}(\boldsymbol{\theta},\mathbf{c}_{j})$ , and candidates are ranked by descending score. (b) Sources of uncertainty in the learned representation space. Aleatoric uncertainty is due to inherent noise and ambiguity in the data, and can arise for structurally similar molecules. Epistemic uncertainty reflects a general lack of knowledge, and arises in regions of the representation space that are far from training data. (c) Selective prediction. Left: test spectra are sorted by confidence in descending order. A threshold $\tau$ partitions spectra into accepted and rejected predictions. Right: sweeping the threshold $\tau$ over its full range traces the risk-coverage curve, where different points on the curve correspond to different risk-coverage trade-offs.

Score	Level	Order	Description
$\kappa_{\mathrm{conf}}$	Retrieval	1st	Max. candidate probability
$\kappa_{\mathrm{gap}}$	Retrieval	1st	Difference between top-two similarity scores
$\kappa_{\mathrm{bit}}^{\mathrm{tot}}$	Fingerprint	2nd	Total predictive entropy (fingerprint)
$\kappa_{\mathrm{bit}}^{\mathrm{al}}$	Fingerprint	2nd	Aleatoric uncertainty (fingerprint)
$\kappa_{\mathrm{bit}}^{\mathrm{ep}}$	Fingerprint	2nd	Epistemic uncertainty (fingerprint)
$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$	Retrieval	2nd	Total predictive entropy (candidate ranking)

Table 1: Overview of different scoring functions

\kappa

investigated in this work. All scores are oriented so that higher values indicate greater confidence. “Order” indicates whether the score requires a single prediction (1st) or multiple samples from a posterior distribution (2nd). Formal definitions are given in Appendix 5 .

Figure 2: Average

\mathrm{Hit}@K

rate for

K\in\{1,5,20\}

for samples from the second-order distribution (grey) and their aggregate (blue, green, purple). Results are shown for a Deep Ensemble of

S=5

members and the experimental setup as described in Section 2.6 , evaluated on the test set of MassSpecGym with candidates filtered by molecular formula. The aggregate is computed using the different aggregation strategies described in Section 2.6 .

MLP
		Aggregate			Avg. sample
Training loss	Method	Hit@1	Hit@5	Hit@20	Hit@1	Hit@5	Hit@20
Focal	Deep Ensemble ( $S\!=\!5$ )	11.43	23.80	42.36	10.71	22.67	40.70
	MC Dropout ( $S\!=\!50$ )	11.13	22.78	40.75	10.33	22.10	40.15
	Laplace ( $S\!=\!50$ )	11.22	22.62	41.39	11.01	22.65	41.38
Ranking	Deep Ensemble ( $S\!=\!5$ )	13.12	27.29	47.57	11.83	25.66	45.83

Table 2: Retrieval performance on the MassSpecGym test set for the two different model architectures discussed in Section 2.6 , under different second-order uncertainty estimation methods.

\mathrm{Hit@}K

(%) denotes the fraction of test spectra for which the correct molecule appears among the top-

K

retrieved candidates.

S

denotes the number of posterior samples (ensemble members or stochastic forward passes). Aggregate: prediction obtained by averaging the

S

predicted scores before ranking. Avg. sample: mean

\mathrm{Hit@}K

across the

S

individual predictions.

Figure 3: Risk-coverage analysis for different scoring functions

\kappa

based on different uncertainty estimates as described in Section 2.3 . The selective risk is calculated using

\ell_{K}=1-\mathrm{Hit@}K

for

K\in\{1,5,20\}

. Results are shown for a Deep Ensemble of

S=5

members and the MLP model trained with ranking loss, as described in Section 2.6 , evaluated on the test set of MassSpecGym with candidates filtered by molecular formula. Colors encode the uncertainty component: blue tones indicate total uncertainty, purple indicates aleatoric uncertainty, and red tones indicate epistemic uncertainty. Darker shades correspond to retrieval-level scores, lighter shades to fingerprint-level scores. (a) Risk-coverage curves. (b) AURC values.

Retrieval-level
	Deep Ensemble			MC Dropout			Laplace
$\kappa$	$\ell_{1}$	$\ell_{5}$	$\ell_{20}$	$\ell_{1}$	$\ell_{5}$	$\ell_{20}$	$\ell_{1}$	$\ell_{5}$	$\ell_{20}$
$\kappa_{\mathrm{conf}}$	0.366	0.406	0.498	0.397	0.440	0.527	0.419	0.521	0.614
$\kappa_{\mathrm{gap}}$	0.439	0.576	0.696	0.461	0.591	0.702	0.465	0.627	0.733
$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$	0.347	0.323	0.359	0.369	0.358	0.388	0.392	0.442	0.500
$\kappa_{\mathrm{ret}}^{\mathrm{al}}$	0.364	0.351	0.407	0.373	0.366	0.407	0.394	0.446	0.508

Table 3: Selective prediction quality for different scoring functions, evaluated by relAURC. Lower is better. Results are shown for different second-order uncertainty estimation methods and the MLP model trained with ranking loss, evaluated on the test set of MassSpecGym with candidates filtered by molecular formula. The selective risk is evaluated using hit rate,

\ell_{K}=1-\mathrm{Hit}@K

K=1,5,20

. relAURC

=0

corresponds to perfect (oracle) rejection, relAURC

=1

corresponds to random rejection.

	MLP			Transformer
$\kappa$	$\ell_{1}$	$\ell_{5}$	$\ell_{20}$	$\ell_{1}$	$\ell_{5}$	$\ell_{20}$
$\kappa_{\mathrm{conf}}$ confidence	0.366	0.406	0.498	0.356	0.367	0.420
$\kappa_{\mathrm{gap}}$ score gap	0.439	0.576	0.696	0.384	0.513	0.606
$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$ total	0.347	0.323	0.359	0.365	0.317	0.320
$\kappa_{\mathrm{ret}}^{\mathrm{al}}$ aleatoric	0.364	0.351	0.407	0.377	0.342	0.369
$\kappa_{\mathrm{rank}}$ $K\!=\!1$	0.473	0.448	0.469	0.446	0.404	0.401

Table 4: Robustness of the relAURC (Eqn. 10 , lower is better) ordering across architectures. Values are given for the strongest retrieval-level scoring functions for the MLP and transformer Deep Ensembles (

S=5

), on the MassSpecGym test set with candidates filtered by molecular formula. Full transformer results across all scoring functions are given in Appendix 8.3 .

Figure 4: Effect of the candidate set size on the retrieval performance and selective prediction quality. Results are shown for a Deep Ensemble of

S=5

members and the experimental setup as described in Section 2.6 , evaluated on the test set of MassSpecGym with candidates filtered by molecular formula. Left : Average hit rate on subsets with binned candidate set size

|\mathcal{C}|

. Middle , Right : AURC values for

\ell_{1}=1-\mathrm{Hit@}1

and different scoring functions, on subsets of data with

|\mathcal{C}|<256

vs.

|\mathcal{C}|=256

Figure 5: Risk-controlled annotation with the SGR algorithm with

\delta=0.001

and different target risks

r^{*}

. Results are shown for the MLP Deep Ensemble of

S=5

members and the experimental setup as described in Section 2.6 , with candidates filtered by molecular formula. The threshold

\tau^{*}

is calibrated on one half of the test set, coverage and empirical risk are evaluated on the held-out other half. Top : coverage attained at each target risk level

r^{*}

for

K\in\{1,5,20\}

. Note the independent vertical scales. Bottom : empirical risk versus target risk.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Pairwise scatter plots of precursor mass, average pairwise cosine similarity and number of candidates per spectrum in the candidate sets of the MassSpecGym test set. Spearman rank correlations are shown in each panel. Histograms on the top and right axes show the marginal distribution of each variable. Larger candidate sets arise for lower precursor masses and contain more structurally diverse candidates.

Fingerprint-level
	Continuous		Discrete
$\kappa$	Tan.	Cos.	Tan.	Cos.	Ham.
$\kappa_{\mathrm{bit}}^{\mathrm{tot}}$ total	0.367	0.712	0.818	0.974	1.335
$\kappa_{\mathrm{bit}}^{\mathrm{al}}$ aleatoric	0.534	0.713	0.853	0.971	1.171
$\kappa_{\mathrm{bit}}^{\mathrm{ep}}$ epistemic	1.142	1.150	1.099	1.061	0.777
Retrieval-level

Appendix

Table 5: Selective prediction quality for fingerprint-level similarity losses, reported as relAURC (Eqn. 10 ). Lower is better. Left: continuous evaluation on predicted probabilities

\overline{\boldsymbol{\theta}}

. Right: discrete evaluation on binarised fingerprints

\hat{\mathbf{y}}=\mathbb{I}[\overline{\boldsymbol{\theta}}>0.5]

. Hamming always uses discrete inputs. Results are shown for the Deep Ensemble (

S=5

) trained with focal loss, and the experimental setup as described in Section 2.6 .

Figure 7: Pairwise Spearman rank correlations among the main scoring functions for the MLP Deep Ensemble trained with ranking loss, evaluated on the official formula-filtered MassSpecGym test candidate sets at $T_{\mathrm{eval}}=0.003$ . All quantities are oriented so that higher values indicate greater confidence; the candidate-set-size criterion is therefore shown as $-\log|\mathcal{C}|$ . Horizontal and vertical lines separate retrieval-level scores, fingerprint-level scores, and the candidate-set-size criterion.

Figure 8: Average

\mathrm{Hit}@K

rate for

K\in\{1,5,20\}

for the individual transformer ensemble members (grey) and their aggregate (blue, green, purple), for a Deep Ensemble of

S=5

members on the MassSpecGym test set with candidates filtered by molecular formula.

Retrieval-level
$\kappa$	$\ell_{1}$	$\ell_{5}$	$\ell_{20}$
$\kappa_{\mathrm{conf}}$ confidence	0.356	0.367	0.420
$\kappa_{\mathrm{gap}}$ score gap	0.384	0.513	0.606
$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$ total	0.365	0.317	0.320
$\kappa_{\mathrm{ret}}^{\mathrm{al}}$ aleatoric	0.377	0.342	0.369
$\kappa_{\mathrm{ret}}^{\mathrm{ep}}$ epistemic	0.501	0.542	0.591

Appendix

Table 6: relAURC (lower is better) of all scoring functions for the transformer Deep Ensemble (

S=5

) trained with ranking loss, on the MassSpecGym test set with candidates filtered by molecular formula.

Figure 9: Risk–coverage analysis for the transformer spectrum encoder ensemble on the MassSpecGym test set, for

\ell_{K}=1-\mathrm{Hit@}K

K\in\{1,5,20\}

Figure 10: Risk-controlled annotation for the transformer Deep Ensemble with

\delta=0.001

and target risks

r^{*}

. Thresholds are calibrated on one half of the MassSpecGym test set and evaluated on the held-out half, using formula-filtered candidates. Top : coverage at each target risk for

K\in\{1,5,20\}

. Bottom : held-out empirical risk versus target risk.

	Hit@1, $\kappa_{\mathrm{conf}}$		Hit@5, $\kappa_{\mathrm{ret}}^{\mathrm{tot}}$		Hit@20, $\kappa_{\mathrm{ret}}^{\mathrm{tot}}$
$r^{*}$	Coverage	Risk	Coverage	Risk	Coverage	Risk
0.10	—	—	7.67	6.84	15.54	6.74
0.15	1.80	5.70	9.23	10.00	32.71	12.75
0.20	5.14	13.08	14.99	16.95	42.48	17.35
0.25	7.29	20.47	22.98	22.01	51.91	22.16
0.30	8.74	24.51	28.05	26.24	64.91	27.99

Appendix

Table 7: Numerical SGR results for the transformer Deep Ensemble. For each retrieval cutoff, the scoring function is selected on the official validation fold: confidence for Hit@1 and retrieval-level total uncertainty for Hit@5 and Hit@20. Coverage and empirical risk are percentages evaluated on the held-out half of the test set after threshold calibration on the other half, with

\delta=0.001

. A dash indicates that no non-empty feasible selection was found.

Architecture	$K$	Best individual score	Individual relAURC	relAURC of combination	$\Delta$
MLP	1	$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$	0.347	0.362	$-0.014$
MLP	5	$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$	0.323	0.306	0.017
MLP	20	$\kappa_{\mathrm{rank}}$ ( $K\!=\!5$ )	0.322	0.277	0.045
Transformer	1	$\kappa_{\mathrm{conf}}$	0.356	0.363	$-0.007$
Transformer	5	$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$	0.317	0.297	0.019
Transformer	20	$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$	0.320	0.275	0.044

Appendix

Table 8: Comparison of relAURC values for the supervised logistic score described in Section 8.4 with the strongest individual score. A separate model is trained for each architecture and retrieval cutoff. Lower relAURC is better.

\Delta

is the relAURC of the best individual score minus that of the score combination, so positive values favor supervised combination.

Figure 11: Sensitivity of selective prediction quality (relAURC, lower is better) to the evaluation temperature

T_{\mathrm{eval}}

of the candidate softmax, for the MLP model (top) and the transformer model (bottom) Deep Ensembles and

\ell_{K}=1-\mathrm{Hit@}K

K\in\{1,5,20\}

, on the MassSpecGym test set. Dashed curves are the temperature-dependent uncertainty scores, solid lines are the temperature-independent scores. The dashed vertical line marks the training temperature

T_{\mathrm{train}}=0.003

Figure 12: Distribution of candidate-set sizes for the matched query pools used in the candidate-setting evaluation. Left : formula-filtered candidate sets. Uncapped PubChem pools extend beyond the MassSpecGym limit of 256 candidates; the horizontal axis is logarithmic. Right : comparison of capped candidate sets based on molecular formula (blue) and precursor mass (turquoise).

Arch.	Train	Evaluation	H@1	H@5	H@20
MLP	Formula cap.	Formula paired cap.	13.19	27.64	47.68
MLP	Formula cap.	Formula uncapped	11.53	22.97	40.09
Transformer	Formula cap.	Formula paired cap.	17.82	35.93	56.88
Transformer	Formula cap.	Formula uncapped	14.74	28.77	47.65
MLP	Mass cap.	Mass cap.	14.06	27.65	45.22

Appendix

Table 9: Retrieval accuracy under alternative candidate constructions. The formula models are trained on the official capped formula candidates; the paired capped and uncapped evaluations are constructed from the same PubChem candidate pools and use the same query mask. The mass-based results use an MLP trained and evaluated on capped mass-filtered candidates. Hit@

K

values are percentages.

Arch.	Evaluation candidates	$K$	$\kappa_{\mathrm{conf}}$	$\kappa_{\mathrm{gap}}$	$\kappa_{\mathrm{ret}}^{\mathrm{tot}}$	$\kappa_{\mathrm{ret}}^{\mathrm{al}}$	$\kappa_{\mathrm{ret}}^{\mathrm{ep}}$	$\kappa_{\mathrm{rank}}^{(K)}$	$-\log\|\mathcal{C}\|$
MLP	Formula paired cap.	1	0.363	0.437	0.346	0.362	0.431	0.494	0.550
		5	0.394	0.579	0.315	0.346	0.442	0.363	0.461
		20	0.485	0.694	0.355	0.403	0.483	0.328	0.391
MLP	Formula uncapped	1	0.318	0.409	0.303	0.318	0.395	0.402	0.469
		5	0.362	0.559	0.282	0.301	0.425	0.319	0.334
		20	0.427	0.684	0.269	0.305	0.405	0.228	0.254

Appendix

Table 10: Selective-prediction quality under alternative candidate constructions, measured by relAURC (lower is better). Formula models are trained on the official capped formula candidates, whereas the mass-based MLP is trained on capped mass-filtered candidates. The paired capped and uncapped formula evaluations use matched query pools. Rank variance is matched to the evaluated cutoff

K

, and

-\log|\mathcal{C}|

denotes the confidence-oriented candidate-set-size criterion. All softmax-derived scores use

T_{\mathrm{eval}}=0.003

. The best value in each row is shown in bold.

Selective Prediction

Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.

See all 286 papers

GLACIER: Rethinking Mass Spectrum Prediction as an Object Detection Problem
Jun 28, 2026Rui-Xi Wang, Runzhong Wang, Connor W. ColeyMass Spectrometry
Massachusetts Institute of Technology Cambridge, MA 02139
MS-GPT: Rethinking MS/MS De Novo Structure Elucidation as Spectrum-Induced Posterior Querying of a Molecule-Language Model
Jul 26, 2026Xin Zhao, Yumin Liu, Zhuo Li +5Language Modeling Molecular Generation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Instrumental Analysis Center, Shanghai Jiao Tong University, Shanghai, China · ByteDance Inc., Beijing, China +1
FlexMS: A Unified Public Benchmark for Molecule Tandem Mass Spectrum Prediction
Feb 26, 2026Yunhua Zhong, Yixuan Tang, Yifan Li +5Benchmark Design Mass Spectrometry
The Hong Kong University of Science and Technology (Guangzhou). · The University of Hong Kong. · Yangzhou University. +3

Jun 28, 2026J/K move · Enter open · S save

GLACIER: Rethinking Mass Spectrum Prediction as an Object Detection Problem

Rui-Xi Wang, Runzhong Wang, Connor W. Coley

Massachusetts Institute of Technology Cambridge, MA 02139

Mass Spectrometry

Open paper AskPDF arXiv

Predicting tandem mass spectra (MS/MS) from molecular structures represents a central task in analytical chemistry with direct relevance to clinical metabolomics, systems biology, and adjacent disciplines. In this work, we revisit the problem through the lens of object detection on molecular graphs. Molecular fragmentation, a central step in MS/MS prediction, can be approximated as detecting a set of subgraphs (i.e., fragments) and their associated spectral contributions. Existing fragment-based models follow a two-stage paradigm -- first generating candidate fragments and then scoring them -- analogous to two-stage R-CNNs in computer vision. Towards higher accuracy and faster inference, we introduce GLACIER, a single-stage transformer-based fragment detection neural network for molecular graphs. This unified formulation eliminates the need for candidate enumeration, enabling scalable and globally consistent modeling of molecular fragmentation. GLACIER is faster and more accurate than existing state-of-the-art by a significant margin, achieving 70.0% and 69.7% Top-1 retrieval accuracy with and without contrastive finetuning on the MassSpecGym dataset (from the previous SOTA of 64.0%) and 52.5% and 38.5% respectively on the NIST'20 dataset (from 33.2%). Furthermore, GLACIER provides nearly 8-fold inference speedup over our prior two-stage model. Code is available at https://github.com/coleygroup/ms-pred

Abstract

Selective Prediction

Explore similar work

GLACIER: Rethinking Mass Spectrum Prediction as an Object Detection Problem

MS-GPT: Rethinking MS/MS De Novo Structure Elucidation as Spectrum-Induced Posterior Querying of a Molecule-Language Model

FlexMS: A Unified Public Benchmark for Molecule Tandem Mass Spectrum Prediction