Organizations: University of Tennessee, Knoxville · University of Illinois at Urbana-Champaign · University of California, Berkeley · University of Georgia · Florida State University
Large language models (LLMs) are increasingly used in molecular discovery and structure elucidation, where decisions depend on physical observations. Existing evaluations primarily focus on end-to-end molecular reconstruction, where correct predictions do not necessarily indicate reliance on the supplied properties. We introduce MolP2SBench, a benchmark for molecular property-to-structure (P2S) reasoning in LLMs. Each task presents a masked molecular scaffold with given properties and asks the model to select the missing structure from plausible candidates. We shuffle or replace properties to test reliance on correct property-structure pairing and responsiveness to counterfactual property changes. A channel-wise leaderboard uses Property-Grounded Recovery (PGR), giving equal weight to source and counterfactual recovery. Evaluating seven models, we find that LLMs can use physical properties for structure prediction, but performance varies substantially across property types. Higher accuracy with a property does not necessarily imply greater responsiveness to it: a model may benefit more from 1H NMR than from 13C NMR, yet be less likely to change its prediction when the 1H data is replaced. LLM performance also need not reflect the relative difficulty of these properties for human chemists. Although UV-Vis spectra provide less direct structural information than NMR spectra, Astra achieves higher accuracy with UV-Vis data than with NMR peaks. MolP2SBench supports studying how LLMs extract and use physical-property information, revealing patterns that may differ from conventional chemical intuition and providing a foundation for more reliably property-grounded models for molecular and quantum-chemical inference. Code and dataset: https://github.com/DUAL-Xiao/MolP2SBench.
Figures & tables
Figure 1: MolP2SBench: a fixed molecular choice under changing property inputs. Top: a masked scaffold, candidate SMILES, and physical observations define a local-completion task; the model selects a candidate and provides a short justification. The illustrative 1 H input contains calculated shieldings and counts for retained scaffold nuclei. Molecular drawings visualize the text inputs, and the response illustrates the output format. Bottom: structure-only input, cross-task shuffling, and same-scaffold counterfactual (CF) property substitution test the benefit of property inputs, pairing dependence, and whether the choice follows replacement property information. The scaffold and candidates remain fixed throughout.
Benchmark
Local editing/ [-2pt]completion
Masked-part [-2pt]reconstruction
Structural [-2pt]distractors
Property-driven [-2pt]answer switch
Explanation [-2pt]checks
FGBench ( Liu et al., 2026 )
×
×
×
×
✓
ChemCoTBench ( Hao et al., 2026 )
✓
×
×
×
×
MolLangBench ( Cai et al., 2026 )
✓
×
×
×
×
ChemIQ ( Runcie et al., 2026 )
×
×
×
×
✓
SpectrumBench ( Yang et al., 2026 )
×
×
✓
×
×
MolecularIQ ( Bartmann et al., 2026 )
×
×
×
×
✓
Table 1: Related work comparison. We compare benchmark support for resolving structure from input properties. ✓ marks an evaluated feature; × means it was not reported or not part of the task.
Table 2: Property-Grounded Recovery (PGR, %) leaderboard. Scores give equal weight to source and counterfactual recovery within each family (Eq. 1 ). Each channel uses the same complete-coverage, input-validated families across all seven models. Gold and silver mark the highest (bold) and second-highest model scores in each column. The random baseline is the expected score under uniform candidate selection.
Table 3: Reliance and responsiveness across seven models. (a) RM (pp) is the accuracy difference between source and shuffled properties; (b) CFR (%) is the task-averaged rate of flips to the CF candidate among initially correct sources. Gold and silver mark the highest (bold) and second-highest scores in each column, including ties. Metric definitions and evaluated cohorts appear in Appendix G .
Figure 2: Success/failure transitions across four pairs of closed-weight models under NMR-bundle input: (a) Luna–Sol, (b) Luna–Astra, (c) Luna–Opus, and (d) Opus–Astra. Percentages refer to all tasks; +/− indicate gains/losses from left to right, and net reports the difference in source-property accuracy in pp.
Figure 6
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Distance
Minimum
QM9S dipole norm
Absolute magnitude difference
0.30 D
HOMO–LUMO gap
Absolute difference
0.0005 Hartree
1 H NMR
Shielding-distribution Wasserstein distance
0.02 ppm
13 C NMR
Shielding-distribution Wasserstein distance
0.10 ppm
IR
Peak-overlap distance
0.05
Raman
Peak-overlap distance
0.05
Appendix
Table 5: Property-specific criteria for final cohort selection. Each threshold applies to every verifiable within-family candidate pair.
Channel
%
Median
Q1–Q3
Dipole magnitude
95.4
1.184
0.806–1.566
HOMO–LUMO gap
88.2
0.912
0.403–1.510
UV–Vis
100.0
6.486
5.309–8.980
IR
100.0
0.713
0.618–0.796
Raman
99.2
0.435
0.312–0.567
1 H peaks
94.1
0.808
0.291–1.342
Appendix
Table 6: Channel-wise property separability. Percentages refer to source tasks separated from every available distractor under the channel-specific reporting cutoffs described above. Distance summaries use the nearest available distractor for each task; Q1–Q3 is the interquartile range. Dipole distances are in D, gap in eV, and NMR in ppm; UV–Vis distances are cutoff-normalized, while IR and Raman use 1−M .
Property
L
S
A
O
Q
G
C
No properties ( P0 )
39.0
42.6
41.2
33.1
33.8
38.2
39.0
Dipole norm
65.4
63.2
69.1
60.3
44.9
44.9
39.0
Dipole vector
62.5
69.1
71.3
61.8
37.5
40.4
38.2
Gap
41.9
71.3
69.1
53.7
36.8
41.2
36.0
1 H peaks
37.5
54.4
69.9
55.1
32.4
39.7
39.0
1 H +Int.
44.9
61.0
69.9
55.1
35.3
38.2
41.9
Appendix
Table 7: Source-property accuracy (%) across seven models. L: Luna; S: Sol; A: Astra; O: Opus; Q: Qwen; G: Gemma; C: ChemDFM. All denominators are 136, including invalid responses.
Model
Property
CFR
Persist
Other
Luna
IR
80.3
11.1
8.6
Raman
56.2
27.9
15.9
Vibrational
85.0
8.5
6.5
All
75.8
16.2
8.0
Sol
IR
90.1
6.2
3.6
Raman
63.8
25.0
11.2
Appendix
Table 8: Conditional source-macro transitions (%). The CFR column uses Eq. 20 ; source-persistence and other-choice rates use the same cohort and source weighting with the corresponding outcome indicators. Minor sum differences reflect rounding.
Electronic
Vibrational
Magnetic Resonance
1 H NMR
13 C NMR
Model
Dipole
Gap
UV–Vis
IR
Raman
Peaks
+Int.
Atomwise
Peaks
+Deg.
Atomwise
Closed-weight LLMs
GPT-5.6-Luna
54.2
45.7
60.0
80.4
61.7
33.7
37.6
32.2
38.4
31.0
32.2
GPT-5.6-Sol
60.0
65.9
70.0
90.0
66.7
50.0
46.9
48.4
56.2
53.9
54.3
GPT-6-Astra
66.7
70.2
77.9
90.4
70.8
64.7
66.3
57.8
65.9
67.4
65.1
Appendix
Table 9: Unconditional CF Accuracy. CF-candidate selection accuracy (%) under same-scaffold property substitution, without conditioning on source correctness. Eligible, input-validated CF pairs receive equal weight; terminal-invalid responses count as failures. Model identifiers and property definitions follow Table 3 . Green shading and boldface mark the highest value in each column.
RM
CFR
n
ρ
Exact
MAD
ρ
Exact
MAD
25
0.9845
59.9
3.16
0.9859
63.3
2.94
50
0.9938
82.1
2.01
0.9936
82.2
1.80
75
0.9981
94.5
1.40
0.9969
91.2
1.26
100
0.9998
99.3
0.91
0.9992
97.7
0.80
125
1.0000
100.0
0.45
1.0000
100.0
0.41
Appendix
Table 10: Model-ranking stability under family subsampling. For each score, ρ is the mean Spearman correlation with its full-cohort model ranking; Exact is the percentage of draws preserving the complete ranking; MAD is the mean absolute score deviation from the full-cohort value, averaged across models and draws, in percentage points.
Property
ρ
Exact
MAD
Dipole
0.9865
62.4
1.34
Gap
0.9793
52.3
1.04
UV–Vis
0.9989
96.2
1.14
IR
0.9991
97.0
0.91
Raman
0.9887
66.1
1.08
1 H peaks
0.9398
20.0
1.10
Appendix
Table 11: Channel-wise PGR stability over 1,000 subsamples of 100 families. ρ is mean Spearman rank correlation; Exact is complete-ranking agreement (%); MAD is mean absolute score deviation (pp).
Model
Both correct
Gain
Loss
Both wrong
Δ [95% interval]
Luna
98
9
16
13
−5.1 [ −12.5 , 2.2 ]
Sol
119
8
3
6
3.7 [ −0.7 , 8.1 ]
Astra
130
4
1
1
2.2 [ −0.7 , 5.9 ]
Opus
115
6
8
7
−1.5 [ −6.6 , 3.7 ]
Appendix
Table 12: Paired source correctness when moving from the vibrational bundle to all-properties input. Each row totals 136. Gain/loss refer to all-properties input relative to vibrational input; loss denotes correct-to-incorrect corruption. These are within-model input comparisons, not cross-model transitions. Intervals are descriptive paired-bootstrap 95% intervals in pp.
Large language models (LLMs) are widely applied across chemical tasks, such as molecular property prediction, which underpins drug discovery. Molecular LLMs represent a molecule through several modalities, notably a 1D SMILES sequence or a 2D molecular graph. Both encode molecular information implicitly, so the contribution of individual substructures remains opaque. Retrieval and augmentation methods add context, but from external sources. However, the cues chemists reason over are the internal substructures that drive a property up or down. We propose MR-MoL, a multi-granular rationale-guided molecular LLM that supplies this evidence directly. A fine-tuned GNN scores each substructure through masking, and the most influential ones are serialized as a ranked, direction-tagged rationale that the LLM reads alongside the SMILES sequence and molecular graph. The rationale spans three levels of granularity: Murcko scaffolds with their side chains, BRICS fragments, and functional groups. This is, to our knowledge, the first method to expose GNN-derived attributions to an LLM as evidence for property prediction. On eight MoleculeNet tasks, MR-MoL achieves the best overall results among generalist models and narrows the gap to specialist models tuned for each task. Five diagnostics further confirm that the model reads the rationale rather than merely benefiting from its presence. Its direction, rank, and substructure each shape the prediction, and its attributions reproduce known structure-property relationships.
Large language models (LLMs) have shown promise for molecular property prediction, but their ability to reason over chemical structures remains limited, as molecular representations such as SMILES differ substantially from the natural language on which LLMs are primarily trained. To bridge this semantic and chemical knowledge gap, we propose MolE-RAG, a training-free, molecule-centric retrieval-augmented generation framework for LLM-based molecular property prediction. MolE-RAG augments each prediction with three complementary sources of inference-time context: retrieved chemistry literature, molecule-specific information including compound synonyms, identifiers, functional group annotations, and physicochemical descriptors, and structurally similar molecules retrieved from the training set. We evaluate MolE-RAG across nine molecular property prediction tasks using proprietary, chemistry-specialized, and open-source LLMs. Across general-purpose LLMs, MolE-RAG improves ROC-AUC by up to 28 percentage points on classification tasks and reduces regression RMSE by up to 67% relative to a SMILES-only baseline. We further find that the utility of each context source varies across models and tasks, with different models benefiting most from textual retrieval, molecular context, or structural retrieval. These results suggest that molecule-centric retrieval can improve LLM-based molecular property prediction without model fine-tuning while providing a flexible framework for integrating heterogeneous chemical knowledge at inference time.
Joey Chan, Wonbin Kweon, Ashley Shin +4
University of Illinois Urbana-Champaign · University of California, San Diego
Large language models (LLMs) are increasingly employed to predict molecular properties. However, prediction error alone cannot distinguish prediction from retrieval of published values. We audit 22 frontier models on 12 molecular regression datasets in a zero-shot setting, assessed against a molecule-blind reference derived from each dataset's labels. Significant retrieval is concentrated on 5 datasets, with isolated flagged LLMs elsewhere. Increasing the reasoning setting raises the number of flagged model--dataset combinations from 47 to 89 of 264. An in-context blinding experiment reduces retrieval but leaves a quarter of the combinations flagged. Blinding changes model rankings and increases errors. Because blinding also removes chemically interpretable structure, the error increase can only be partially attributed to reduced retrieval.
Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3