LLMs Depend on Specialist Tools for Out-of-Distribution Generalization in MS/MS Structural Elucidation
Organizations: Matterworks, Inc. Somerville, MA 02143, USA
Abstract
A frontier language model was recently reported to solve 1D NMR structural elucidation without tools. We ask whether this capability transfers to tandem mass spectrometry (MS/MS) in the de novo setting, where the model must propose a structure directly from the spectrum without a candidate list or reference library. To separate reasoning from recall, we build a 100-analyte benchmark from MassSpecGym, split by each analyte's textual footprint into in-distribution (well documented in literature) and out-of-distribution (essentially undocumented) arms. We hold the model, harness, and output format fixed and vary only the LLM's access to domain-specific tools. Without tools, a frontier LLM reaches 52% Top-1 accuracy in-distribution but only 2% out-of-distribution. Only access to de novo structure generators improves out-of-distribution accuracy, raising it to as much as 38% Top-1, and the resulting agent outperforms the standalone de novo models both in and out of distribution. These results suggest that LLMs are not effective at de novo MS/MS structural elucidation on their own, and that further gains depend on the specialist tools available to the agent rather than on the LLM itself.
Figures & tables
| Top-1 | Top-5 | Tanimoto@1 | MCES@1 | ||||||||||
| Model | ID | OOD | All | ID | OOD | All | ID | OOD | All | ID | OOD | All | |
| Non-agentic reference (formula given) | |||||||||||||
| – | Public de novo (FRIGID) | 24 (*) | 6 | 15 | 30 (*) | 12 | 21 | 0.51 | 0.38 | 0.44 | 9.4 (*) | 11.6 | 10.5 |
| – | Public de novo (MS-GPT) | 48 | 18 * | 33 | 54 | 30 *** | 42 * | 0.69 | 0.50 *** | 0.60 ** | 6.2 | 9.8 *** | 8.0 |
| – | In-house de novo ( LSM-Gen ) | 60 | 32 *** | 46 ** | 64 | 36 *** | 50 ** | 0.73 | 0.58 *** | 0.66 *** | 4.9 | 8.0 *** | 6.5 ** |
| Agentic ladder (formula given) | |||||||||||||
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Removed | Remaining | |
| Spectra | ||
| MassSpecGym v1.5 test fold | 17,556 | |
| absent from the candidate pool | 107 | 17,449 |
| fewer than four fragment peaks | 1,900 | 15,549 |
| Analytes | ||
| Standardized test-fold analytes | 2,992 | |
| experiment | Token USD | GPU s | GPU USD | Total USD | Top-1 | USD / correct |
|---|---|---|---|---|---|---|
| Non-agentic reference | ||||||
| FRIGID | 0.00 | 37 | 0.043 | 0.04 | 15% | 0.28 |
| MS-GPT | 0.00 | 32 | 0.036 | 0.04 | 33% | 0.11 |
| LSM-Gen | 0.00 | 41 | 0.046 | 0.05 | 46% | 0.10 |
| Agentic ladder | ||||||
| L0 | 1.27 | 0 | 0.000 | 1.27 | 27% | 4.70 |