Small molecule retrieval from tandem mass spectrometry: what are we optimizing for?

Gaetan De Waele; Marek Wydmuch; Krzysztof Dembczyński; Wojciech Kotłowski; Willem Waegeman

cs.LGFeb 18, 2026

Small molecule retrieval from tandem mass spectrometry: what are we optimizing for?

Authors: Gaetan De Waele, Marek Wydmuch, Krzysztof Dembczyński, Wojciech Kotłowski, Willem Waegeman

Organizations: Department of Data Analysis and Mathematical Modelling, Ghent University, Belgium · Snowflake AI Research · Poznan University of Technology

Abstract

One of the central challenges in the computational analysis of liquid chromatography-tandem mass spectrometry (LC-MS/MS) data is to identify the compounds underlying the output spectra. In recent years, this problem is increasingly tackled using deep learning methods. A common strategy involves predicting a molecular fingerprint vector from an input mass spectrum, which is then used to search for matches in a chemical compound database. While various loss functions are employed in training these predictive models, their impact on model performance remains poorly understood. In this study, we investigate commonly used loss functions, deriving novel regret bounds that characterize when Bayes-optimal decisions for these objectives must diverge. Our results reveal a fundamental trade-off between the two objectives of (1) fingerprint similarity and (2) molecular retrieval. Optimizing for more accurate fingerprint predictions typically worsens retrieval results, and vice versa. Our theoretical analysis shows this trade-off depends on the similarity structure of candidate sets, providing guidance for loss function and fingerprint selection.

Figures & tables

Figure 1: Empirically, different loss functions form a pareto front trading off fingerprint similarity (y-axis) and retrieval performance (x-axis). See Section 3 for experimental details. Remark that maximizing Tanimoto similarity is the same as minimizing IoU loss.

Figure 2: (A) General framework used in this study. In this figure, all superscript notations indicating the sample index

(i)

are omitted. A model

f

predicts a fingerprint

\hat{\boldsymbol{y}}^{(i)}\in[0,1]^{m}

from an input spectrum

\boldsymbol{x}^{(i)}

. The prediction is then compared to a set of fingerprints

\mathcal{C}^{(i)}

derived from candidate molecules in a database. The true fingerprint

\boldsymbol{y}^{(i)}

is assumed to be part of the candidate set. (B) Loss functions for training

f(\cdot)

generally fall into one of three categories: (1) bitwise losses (e.g., binary cross entropy or binary focal loss), (2) vectorwise losses (e.g., soft cosine loss or soft IoU loss), or (3) listwise losses (e.g., contrastive loss).

	Retrieval w. equal mass candidates			Retrieval w. equal formula candidates
	HR@1	HR@5	HR@20	HR@1	HR@5	HR@20	Tanimoto (IoU)
This study
BCE	$10.12\pm 0.36$	$19.44\pm 0.16$	$34.96\pm 0.73$	$10.08\pm 0.22$	$22.60\pm 0.35$	$40.79\pm 0.41$	$16.75\pm 0.11$
FL	$11.54\pm 0.14$	$20.84\pm 0.27$	$36.72\pm 0.44$	$11.15\pm 0.14$	$23.82\pm 0.23$	$42.76\pm 0.53$	$16.58\pm 0.16$
Cosine Loss	$09.48\pm 0.19$	$18.19\pm 0.17$	$32.90\pm 0.38$	$09.12\pm 0.41$	$21.33\pm 0.30$	$39.38\pm 0.31$	$\underline{17.22\pm 0.09}$
IoU Loss	$05.58\pm 0.32$	$13.01\pm 0.39$	$26.51\pm 0.50$	$06.48\pm 0.32$	$17.62\pm 0.51$	$35.05\pm 0.22$	$\mathbf{19.42\pm 0.13}$

Table 1: Molecular retrieval and fingerprint accuracy performance on MassSpecGym ( Bushuiev et al., 2024a ) . The best and second-best performing models for each metric are indicated with boldface and underlined, respectively. For the models trained in this study, numbers report averages and standard deviations over 5 model runs. The performances of a random model and the MIST model ( Goldman et al., 2023 ) are taken from Bushuiev et al. (2024a) .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Validation performance (in terms of Average Tanimoto, blue, and HR@20, red) over training time. Curves are shown for the BCE Loss, IoU Loss, and Contrastive FP-Cos, respectively. Note that the Contrastive FP-Cos curve is obtained using a temperature of 1/256. As such, the model is tuned for better hit rates (see Table Table 3 ). In addition, note that our experimental setup accounts for different optima in time w.r.t. different metrics, as final scores are reported using the checkpoint for which each metric was optimal.

Focal $\gamma$	Tanimoto (IoU)	HR@1	HR@5	HR@20
$1.0$	$\mathbf{16.27\pm 0.07}$	$08.94\pm 0.18$	$18.14\pm 0.37$	$32.45\pm 0.33$
$\mathbf{2.0}$	$16.20\pm 0.06$	$\mathbf{09.20\pm 0.08}$	$\mathbf{18.74\pm 0.20}$	$\mathbf{33.05\pm 0.16}$
$5.0$	$15.88\pm 0.10$	$08.45\pm 0.18$	$18.10\pm 0.25$	$31.53\pm 0.32$
$10.0$	$15.70\pm 0.05$	$06.31\pm 0.23$	$15.00\pm 0.33$	$27.72\pm 0.27$

Appendix

Table 2: Focal loss

\gamma

tuning results ( Bushuiev et al., 2024a ) . All performances are validation scores. Numbers report averages and standard deviations over 5 model runs. All 20 models used in this table were trained with a learning rate of

7\text{e-}5

. Retrieval scores are those using equal mass candidates. The best hyperparameter value (bold) was chosen based on retrieval scores.

Contrastive loss type	Temperature $\tau$	Tanimoto (IoU)	HR@1	HR@5	HR@20
FP-Cos	$\mathbf{1/256}$	$02.67\pm 0.03$	$\mathbf{11.31\pm 0.59}$	$\mathbf{25.51\pm 1.02}$	$\mathbf{45.60\pm 1.09}$
	$1/64$	$04.06\pm 0.26$	$11.17\pm 0.18$	$24.75\pm 0.62$	$44.54\pm 0.65$
	$1/16$	$03.63\pm 0.05$	$09.75\pm 0.29$	$21.42\pm 0.25$	$37.66\pm 0.29$
	$1/4$	$06.66\pm 0.05$	$08.23\pm 0.18$	$18.32\pm 0.26$	$34.13\pm 0.23$
	$1$	$07.19\pm 0.04$	$08.17\pm 0.12$	$18.06\pm 0.18$	$33.48\pm 0.15$
	$4$	$\mathbf{07.38\pm 0.04}$	$08.14\pm 0.10$	$18.20\pm 0.19$	$33.77\pm 0.16$

Appendix

Table 3: Contrastive loss temperature

\tau

tuning results ( Bushuiev et al., 2024a ) . All performances are validation scores. Numbers report averages and standard deviations over 5 model runs. All models used in this table were trained with a learning rate of

7\text{e-}5

. Tanimoto scores are only reported for the ”FP-Cos” contrastive model, as this is the only one that can be interpreted as predicting a true fingerprint. Retrieval scores are those using equal mass candidates. Best hyperparameter values (bold) were chosen based on retrieval scores.

Model trained using	Evaluate retrieval using	HR@1	HR@5	HR@20
Cosine Loss	Cosine	$7.49\pm 0.16$	$15.82\pm 0.21$	$29.39\pm 0.21$
	Tanimoto (IoU)	$7.21\pm 0.21$	$15.34\pm 0.14$	$28.93\pm 0.25$
IoU Loss	Cosine	$4.29\pm 0.21$	$11.00\pm 0.18$	$23.66\pm 0.25$
	Tanimoto (IoU)	$3.86\pm 0.21$	$10.45\pm 0.20$	$23.01\pm 0.28$

Appendix

Table 4: Validation retrieval scores of models using vectorwise loss functions. Only results are shown on the equal mass candidate retrieval setting. On average, performing retrieval using cosine similarity performs better than using Tanimoto similarity (i.e., IoU). Here, IoU retrieval is performed with continuous (i.e., non-binarized) predictions. This holds true even for the model trained using the IoU loss. Scores indicate the average and standard deviation over 5 model runs.

Figure 4: Experimental results when training models using a weighted combination of loss functions. In blue, results are repeated from Figure 1 . In orange, results are shown using a combination of contrastive loss and IoU loss, with 5 different weights

\lambda

. Scores indicate the mean over 5 model runs. Notably, Even small weights for on contrastive loss components result in a marked increase in retrieval scores over non-contrastive loss functions.

Figure 5: Empirical distribution of

\sigma_{\min}

\sigma_{\max}

, and

\sigma_{\max}-\sigma_{\min}

in all MassSpecGym equal-mass candidate sets using the Tanimoto similarity on Morgan Fingerprints (4096 bits, radius 2) to compare candidates to true molecules.

\sigma_{\min}

denotes the lowest similarity to the true molecule found in the candidate set.

\sigma_{\max}

denotes the highest similarity to the true molecule found in the candidate set. In some cases, there are candidates with zero similarity to the true molecule (no fingerprint bits in common). Conversely, in some cases, there exist candidates with an equal fingerprint vector representation (i.e.,

\sigma_{\max}=1

in some cases). This arises due to the problem of duplicate fingerprints. The median

\sigma_{\min}

equals

0.026

, while the median

\sigma_{\max}

equals

0.367

Figure 6: Empirical distribution of

\sigma_{\min}

\sigma_{\max}

, and

\sigma_{\max}-\sigma_{\min}

in all MassSpecGym equal-mass candidate sets using the Cosine similarity on Morgan Fingerprints (4096 bits, radius 2) to compare candidates to true molecules.

\sigma_{\min}

denotes the lowest similarity to the true molecule found in the candidate set.

\sigma_{\max}

\sigma_{\max}=1

in some cases). This arises due to the problem of duplicate fingerprints. The median

\sigma_{\min}

equals

0.053

, while the median

\sigma_{\max}

equals

0.538

		Retrieval w. equal mass candidates			Retrieval w. equal formula candidates
Candidate set origin	Max. (aver.) # cands	HR@1	HR@5	HR@20	HR@1	HR@5	HR@20
MassSpecGym equal mass	256 (252.8)	$\mathbf{12.29\pm 0.69}$	$\mathbf{26.09\pm 1.43}$	$\mathbf{45.05\pm 0.71}$	$\mathbf{14.17\pm 0.39}$	$\mathbf{28.36\pm 0.46}$	$\mathbf{48.86\pm 0.94}$
MassSpecGym equal formula	256 (212.1)	$10.43\pm 0.66$	$21.91\pm 1.07$	$39.08\pm 0.72$	$13.62\pm 1.12$	$26.33\pm 1.06$	$46.50\pm 0.94$
”Hard” equal mass	64 (63.9)	$07.74\pm 0.61$	$18.81\pm 0.95$	$36.4\pm 1.62$	$11.23\pm 0.48$	$24.23\pm 0.94$	$44.13\pm 1.26$
”Hard” equal mass	128 (127.7)	$08.13\pm 0.57$	$20.64\pm 1.20$	$39.81\pm 1.70$	$12.25\pm 0.91$	$25.96\pm 0.33$	$45.73\pm 0.62$
”Hard” equal mass	256 (253.8)	$08.34\pm 0.60$	$21.51\pm 1.64$	$40.67\pm 1.58$	$12.30\pm 0.69$	$25.86\pm 1.40$	$44.90\pm 1.24$

Appendix

Table 5: Molecular retrieval and fingerprint accuracy performance on MassSpecGym ( Bushuiev et al., 2024a ) . The best and second-best performing models for each metric are indicated with boldface and underlined, respectively. For the models trained in this study, numbers report averages and standard deviations over 5 model runs.

Fingerprint type	$\sigma_{\text{min}}$	$\sigma_{\text{max}}$	$\sigma_{\text{max}}-\sigma_{\text{min}}$
Morgan $r=2$	0.026	0.367	0.342
Morgan $r=4$	0.018	0.241	0.223
Morgan $r=6$	0.017	0.211	0.193
Morgan $r=8$	0.017	0.204	0.187
RDKit	0.039	0.512	0.471
MAP4	0.025	0.198	0.173

Appendix

Table 6: Empirical medians of

\sigma_{\text{min}}

\sigma_{\text{max}}

, and

\sigma_{\text{max}}-\sigma_{\text{min}}

distributions in MassSpecGym equal-mass candidate sets using Tanimoto similarity, depending on the type of fingerprint.

	Scores obtained with IoU loss				Scores obtained with Contr. Emb-Cos loss
Fingerprint type	Tanimoto 1	HR@1	HR@5	HR@20	HR@1	HR@5	HR@20
Morgan $r=2$	$19.42\pm 0.13$	$5.58\pm 0.32$	$13.01\pm 0.39$	$26.51\pm 0.50$	$\mathbf{12.29\pm 0.69}$	$\mathbf{26.09\pm 1.43}$	$\underline{45.05\pm 0.71}$
Morgan $r=4$	$13.61\pm 0.02$	$5.56\pm 0.48$	$12.83\pm 0.64$	$25.80\pm 0.66$	$10.55\pm 0.54$	$23.96\pm 1.03$	$41.41\pm 0.92$
Morgan $r=6$	$11.92\pm 0.22$	$5.65\pm 0.27$	$13.27\pm 0.26$	$25.54\pm 0.56$	$09.51\pm 0.67$	$21.18\pm 0.84$	$37.54\pm 0.75$
Morgan $r=8$	$11.25\pm 0.09$	$5.68\pm 0.37$	$12.88\pm 0.52$	$25.81\pm 0.51$	$08.72\pm 0.52$	$19.67\pm 0.81$	$36.63\pm 0.77$
RDKit	$\mathbf{30.32\pm 0.15}$	$\underline{5.78\pm 0.17}$	$15.00\pm 0.40$	$\mathbf{29.96\pm 0.62}$	$09.63\pm 0.61$	$22.49\pm 0.60$	$44.13\pm 1.54$

Appendix

Table 7: Fingerprint benchmark results. Retrieval scores are those using equal mass candidates. All models used in this table were trained with a learning rate of

7\text{e-}5

. Numbers report averages and standard deviations over 5 model runs. The best and second-best performing models for each metric are indicated with boldface and underlined, respectively.

Mass Spectrometry

Mass spectrometry measures ion mass-to-charge patterns to identify and quantify molecules, with AI linking spectra to structures, peptides, and biological signals.

See all 33 papers

When should we trust the annotation? Selective prediction for molecular structure retrieval from mass spectra
Mar 11, 2026Mira Jürgens, Gaetan De Waele, Morteza Rakhshaninejad +1Selective Prediction Mass Spectrometry
Department of Data Analysis and Mathematical Modeling, Ghent University, Coupure Links 653, Ghent, 9000, Belgium. · Department of Computer Science, University of Antwerp, Middelheimlaan 1, Antwerp, 2020, Belgium.
Nearest-neighbour baselines for fingerprint prediction from MS/MS spectra under different assumptions
Sep 30, 2026Ling Min Serena Khook-Nearest Neighbors Mass Spectrometry
MIT
FlexMS: A Unified Public Benchmark for Molecule Tandem Mass Spectrum Prediction
Feb 26, 2026Yunhua Zhong, Yixuan Tang, Yifan Li +5Benchmark Design Mass Spectrometry
The Hong Kong University of Science and Technology (Guangzhou). · The University of Hong Kong. · Yangzhou University. +3

Mar 11, 2026J/K move · Enter open · S save

When should we trust the annotation? Selective prediction for molecular structure retrieval from mass spectra

Mira Jürgens, Gaetan De Waele, Morteza Rakhshaninejad, Willem Waegeman

Department of Data Analysis and Mathematical Modeling, Ghent University, Coupure Links 653, Ghent, 9000, Belgium. · Department of Computer Science, University of Antwerp, Middelheimlaan 1, Antwerp, 2020, Belgium.

Selective Prediction Mass Spectrometry Uncertainty Quantification

Open paper AskPDF arXiv

Machine learning methods for identifying molecular structures from tandem mass spectra (MS/MS) have advanced rapidly, yet current approaches still exhibit significant error rates. In high-stakes applications such as clinical metabolomics and environmental screening, incorrect annotations can have serious consequences, making it essential to determine when a prediction can be trusted. In this work, we introduce a selective prediction framework for molecular structure retrieval from MS/MS spectra, separating low-risk predictions automatically from lower-confidence predictions. We formulate the problem within the risk-coverage tradeoff framework and systematically evaluate uncertainty quantification strategies at three levels: input-level distance in the learned representation, fingerprint-level uncertainty over predicted molecular fingerprint bits, and retrieval-level uncertainty over candidate rankings. We compare scoring functions including first-order confidence measures, aleatoric and epistemic uncertainty estimates from second-order distributions, as well as distance-based measures in the latent space. All experiments are conducted on the MassSpecGym benchmark. Our analysis reveals that while fingerprint-level uncertainty scores are poor proxies for retrieval success, retrieval-level total uncertainty provides the overall strongest rejection criterion, and first-order confidence measures are computationally inexpensive, strong baselines. We demonstrate that by applying distribution-free risk control via generalisation bounds, practitioners can specify a tolerable error rate and obtain a subset of annotations satisfying that constraint with high probability.

Abstract

Mass Spectrometry

Explore similar work

When should we trust the annotation? Selective prediction for molecular structure retrieval from mass spectra

Nearest-neighbour baselines for fingerprint prediction from MS/MS spectra under different assumptions

FlexMS: A Unified Public Benchmark for Molecule Tandem Mass Spectrum Prediction