MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?

Yuhang Xie; Zhiyi Shi; Zeyi Zhang; Zhisheng Qi; Yushun Dong; Yu Wang; Xiongye Xiao

physics.chem-phOct 8, 2026

MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?

Authors: Yuhang Xie, Zhiyi Shi, Zeyi Zhang, Zhisheng Qi, Yushun Dong, Yu Wang, Xiongye Xiao

Organizations: University of Tennessee, Knoxville · University of Illinois at Urbana-Champaign · University of California, Berkeley · University of Georgia ·

Abstract

Large language models (LLMs) are increasingly used in molecular discovery and structure elucidation, where decisions depend on physical observations. Existing evaluations primarily focus on end-to-end molecular reconstruction, where correct predictions do not necessarily indicate reliance on the supplied properties. We introduce MolP2SBench, a benchmark for molecular property-to-structure (P2S) reasoning in LLMs. Each task presents a masked molecular scaffold with given properties and asks the model to select the missing structure from plausible candidates. We shuffle or replace properties to test reliance on correct property-structure pairing and responsiveness to counterfactual property changes. A channel-wise leaderboard uses Property-Grounded Recovery (PGR), giving equal weight to source and counterfactual recovery. Evaluating seven models, we find that LLMs can use physical properties for structure prediction, but performance varies substantially across property types. Higher accuracy with a property does not necessarily imply greater responsiveness to it: a model may benefit more from $^1$ H NMR than from $^{13}$ C NMR, yet be less likely to change its prediction when the $^1$ H data is replaced. LLM performance also need not reflect the relative difficulty of these properties for human chemists. Although UV-Vis spectra provide less direct structural information than NMR spectra, Astra achieves higher accuracy with UV-Vis data than with NMR peaks. MolP2SBench supports studying how LLMs extract and use physical-property information, revealing patterns that may differ from conventional chemical intuition and providing a foundation for more reliably property-grounded models for molecular and quantum-chemical inference. Code and dataset: https://github.com/DUAL-Xiao/MolP2SBench.

Figures & tables

Figure 1: MolP2SBench: a fixed molecular choice under changing property inputs. Top: a masked scaffold, candidate SMILES, and physical observations define a local-completion task; the model selects a candidate and provides a short justification. The illustrative 1 H input contains calculated shieldings and counts for retained scaffold nuclei. Molecular drawings visualize the text inputs, and the response illustrates the output format. Bottom: structure-only input, cross-task shuffling, and same-scaffold counterfactual (CF) property substitution test the benefit of property inputs, pairing dependence, and whether the choice follows replacement property information. The scaffold and candidates remain fixed throughout.

Benchmark	Local editing/ [-2pt]completion	Masked-part [-2pt]reconstruction	Structural [-2pt]distractors	Property-driven [-2pt]answer switch	Explanation [-2pt]checks
FGBench ( Liu et al., 2026 )	$\times$	$\times$	$\times$	$\times$	$\checkmark$
ChemCoTBench ( Hao et al., 2026 )	$\checkmark$	$\times$	$\times$	$\times$	$\times$
MolLangBench ( Cai et al., 2026 )	$\checkmark$	$\times$	$\times$	$\times$	$\times$
ChemIQ ( Runcie et al., 2026 )	$\times$	$\times$	$\times$	$\times$	$\checkmark$
SpectrumBench ( Yang et al., 2026 )	$\times$	$\times$	$\checkmark$	$\times$	$\times$
MolecularIQ ( Bartmann et al., 2026 )	$\times$	$\times$	$\times$	$\times$	$\checkmark$

Table 1: Related work comparison. We compare benchmark support for resolving structure from input properties.

\checkmark

marks an evaluated feature;

\times

means it was not reported or not part of the task.

Panel of Table 2 — Table 2: Property-Grounded Recovery (PGR, %) leaderboard. Scores give equal weight to source and counterfactual recovery within each family (Eq. 1 ). Each channel uses the same complete-coverage, input-validated families across all seven models. Gold and silver mark the highest (bold) and second-highest model scores in each column. The random baseline is the expected score under uniform candidate selection.

Panel of Table 3 — Table 3: Reliance and responsiveness across seven models. (a) RM (pp) is the accuracy difference between source and shuffled properties; (b) CFR (%) is the task-averaged rate of flips to the CF candidate among initially correct sources. Gold and silver mark the highest (bold) and second-highest scores in each column, including ties. Metric definitions and evaluated cohorts appear in Appendix G .

Figure 2: Success/failure transitions across four pairs of closed-weight models under NMR-bundle input: (a) Luna–Sol, (b) Luna–Astra, (c) Luna–Opus, and (d) Opus–Astra. Percentages refer to all tasks;

+/-

indicate gains/losses from left to right, and net reports the difference in source-property accuracy in pp.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Distance	Minimum
QM9S dipole norm	Absolute magnitude difference	0.30 D
HOMO–LUMO gap	Absolute difference	0.0005 Hartree
1 H NMR	Shielding-distribution Wasserstein distance	0.02 ppm
13 C NMR	Shielding-distribution Wasserstein distance	0.10 ppm
IR	Peak-overlap distance	0.05
Raman	Peak-overlap distance	0.05

Appendix

Table 5: Property-specific criteria for final cohort selection. Each threshold applies to every verifiable within-family candidate pair.

Channel	%	Median	Q1–Q3
Dipole magnitude	95.4	1.184	0.806–1.566
HOMO–LUMO gap	88.2	0.912	0.403–1.510
UV–Vis	100.0	6.486	5.309–8.980
IR	100.0	0.713	0.618–0.796
Raman	99.2	0.435	0.312–0.567
1 H peaks	94.1	0.808	0.291–1.342

Appendix

Table 6: Channel-wise property separability. Percentages refer to source tasks separated from every available distractor under the channel-specific reporting cutoffs described above. Distance summaries use the nearest available distractor for each task; Q1–Q3 is the interquartile range. Dipole distances are in D, gap in eV, and NMR in ppm; UV–Vis distances are cutoff-normalized, while IR and Raman use

1-M

Property	L	S	A	O	Q	G	C
No properties ( $P_{0}$ )	39.0	42.6	41.2	33.1	33.8	38.2	39.0
Dipole norm	65.4	63.2	69.1	60.3	44.9	44.9	39.0
Dipole vector	62.5	69.1	71.3	61.8	37.5	40.4	38.2
Gap	41.9	71.3	69.1	53.7	36.8	41.2	36.0
1 H peaks	37.5	54.4	69.9	55.1	32.4	39.7	39.0
1 H +Int.	44.9	61.0	69.9	55.1	35.3	38.2	41.9

Appendix

Table 7: Source-property accuracy (%) across seven models. L: Luna; S: Sol; A: Astra; O: Opus; Q: Qwen; G: Gemma; C: ChemDFM. All denominators are 136, including invalid responses.

Model	Property	CFR	Persist	Other
Luna	IR	80.3	11.1	8.6
	Raman	56.2	27.9	15.9
	Vibrational	85.0	8.5	6.5
	All	75.8	16.2	8.0
Sol	IR	90.1	6.2	3.6
	Raman	63.8	25.0	11.2

Appendix

Table 8: Conditional source-macro transitions (%). The CFR column uses Eq. 20 ; source-persistence and other-choice rates use the same cohort and source weighting with the corresponding outcome indicators. Minor sum differences reflect rounding.

Closed-weight LLMs
	Electronic			Vibrational		Magnetic Resonance
						1 H NMR			13 C NMR
Model	Dipole	Gap	UV–Vis	IR	Raman	Peaks	+Int.	Atomwise	Peaks	+Deg.	Atomwise
GPT-5.6-Luna	$54.2$	$45.7$	$60.0$	$80.4$	$61.7$	$33.7$	$37.6$	$32.2$	$38.4$	$31.0$	$32.2$
GPT-5.6-Sol	$60.0$	$65.9$	$70.0$	$90.0$	$66.7$	$50.0$	$46.9$	$48.4$	$56.2$	$53.9$	$54.3$
GPT-6-Astra	66.7	70.2	77.9	90.4	70.8	64.7	66.3	57.8	65.9	67.4	65.1

Appendix

Table 9: Unconditional CF Accuracy. CF-candidate selection accuracy (%) under same-scaffold property substitution, without conditioning on source correctness. Eligible, input-validated CF pairs receive equal weight; terminal-invalid responses count as failures. Model identifiers and property definitions follow Table 3 . Green shading and boldface mark the highest value in each column.

	RM			CFR
$n$	$\rho$	Exact	MAD	$\rho$	Exact	MAD
25	0.9845	59.9	3.16	0.9859	63.3	2.94
50	0.9938	82.1	2.01	0.9936	82.2	1.80
75	0.9981	94.5	1.40	0.9969	91.2	1.26
100	0.9998	99.3	0.91	0.9992	97.7	0.80
125	1.0000	100.0	0.45	1.0000	100.0	0.41

Appendix

Table 10: Model-ranking stability under family subsampling. For each score,

\rho

is the mean Spearman correlation with its full-cohort model ranking; Exact is the percentage of draws preserving the complete ranking; MAD is the mean absolute score deviation from the full-cohort value, averaged across models and draws, in percentage points.

Property	$\rho$	Exact	MAD
Dipole	0.9865	62.4	1.34
Gap	0.9793	52.3	1.04
UV–Vis	0.9989	96.2	1.14
IR	0.9991	97.0	0.91
Raman	0.9887	66.1	1.08
1 H peaks	0.9398	20.0	1.10

Appendix

Table 11: Channel-wise PGR stability over 1,000 subsamples of 100 families.

\rho

is mean Spearman rank correlation; Exact is complete-ranking agreement (%); MAD is mean absolute score deviation (pp).

Model	Both correct	Gain	Loss	Both wrong	$\Delta$ [95% interval]
Luna	98	9	16	13	$-5.1$ [ $-12.5$ , $2.2$ ]
Sol	119	8	3	6	$3.7$ [ $-0.7$ , $8.1$ ]
Astra	130	4	1	1	$2.2$ [ $-0.7$ , $5.9$ ]
Opus	115	6	8	7	$-1.5$ [ $-6.6$ , $3.7$ ]

Appendix

Table 12: Paired source correctness when moving from the vibrational bundle to all-properties input. Each row totals 136. Gain/loss refer to all-properties input relative to vibrational input; loss denotes correct-to-incorrect corruption. These are within-model input comparisons, not cross-model transitions. Intervals are descriptive paired-bootstrap 95% intervals in pp.

Task and SMILES	Candidate list	Scaffold
Task 005A3B23AB24FFB0 Source: O=CCCCC=O Scaffold: *:1CC=O	0 ∗ : O=CCCCC=O 1: O=CCOC(=O)CC=O
Task 0293945FD5430B3D Source: CNC(=O)OC1CC1 Scaffold: *:1C(=O)OC1CC1	0: CC(=O)C(=O)OC1CC1 1 ∗ : CNC(=O)OC1CC1
Task 043694CC3A98D45D Source: CC[C@H]1OC1(C)C Scaffold: [*:1]C[C@H]1OC1(C)C	0: COC[C@H]1OC1(C)C 1: CC(=O)C[C@H]1OC1(C)C 2 ∗ : CC[C@H]1OC1(C)C
Task 0556D16AF07422B2 Source: OCCC1(O)CC1 Scaffold: [*:1]CCC1(O)CC1	0: NC(=O)CCC1(O)CC1 1: N#CCCC1(O)CC1 2 ∗ : OCCC1(O)CC1 3: CCCC1(O)CC1
Task 0D39B8E892B8D428 Source: CC(C)OC(=O)NC=O Scaffold: *:1C(C)C	0 ∗ : CC(C)OC(=O)NC=O 1: CC(C)CNC=O 2: CC(C)C(=O)NC=O
Task 0D5F32ECEAFB257A Source: CCOCC1(C)CC1 Scaffold: *:1CC1(C)CC1	0: CCC(=O)CC1(C)CC1 1 ∗ : CCOCC1(C)CC1

Appendix

Table 13: Complete task families: source molecules, masked scaffolds, and candidate SMILES.

Scientific Reasoning in Language Models

Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.

See all 127 papers

Multi-Granular Rationale-Guided Molecular LLM for Property Prediction
Aug 11, 2026Junwoo Park, Minyoung Shin, Cheol Soon Lee +1Molecular Property Prediction Molecular Representation Learning
Sungkyunkwan University · Korea University
MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry
Jun 4, 2026Joey Chan, Wonbin Kweon, Ashley Shin +4Retrieval-Augmented Generation LLM Inference
University of Illinois Urbana-Champaign · University of California, San Diego
Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models
Sep 4, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation LLM Reliability
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3

Aug 11, 2026J/K move · Enter open · S save

Multi-Granular Rationale-Guided Molecular LLM for Property Prediction

Junwoo Park, Minyoung Shin, Cheol Soon Lee, Sujee Lee

Sungkyunkwan University · Korea University

Molecular Property Prediction Molecular Representation Learning LLMs for Graph Learning

Open paper AskPDF arXiv

Large language models (LLMs) are widely applied across chemical tasks, such as molecular property prediction, which underpins drug discovery. Molecular LLMs represent a molecule through several modalities, notably a 1D SMILES sequence or a 2D molecular graph. Both encode molecular information implicitly, so the contribution of individual substructures remains opaque. Retrieval and augmentation methods add context, but from external sources. However, the cues chemists reason over are the internal substructures that drive a property up or down. We propose MR-MoL, a multi-granular rationale-guided molecular LLM that supplies this evidence directly. A fine-tuned GNN scores each substructure through masking, and the most influential ones are serialized as a ranked, direction-tagged rationale that the LLM reads alongside the SMILES sequence and molecular graph. The rationale spans three levels of granularity: Murcko scaffolds with their side chains, BRICS fragments, and functional groups. This is, to our knowledge, the first method to expose GNN-derived attributions to an LLM as evidence for property prediction. On eight MoleculeNet tasks, MR-MoL achieves the best overall results among generalist models and narrows the gap to specialist models tuned for each task. Five diagnostics further confirm that the model reads the rationale rather than merely benefiting from its presence. Its direction, rank, and substructure each shape the prediction, and its attributions reproduce known structure-property relationships.

Abstract

Scientific Reasoning in Language Models

Explore similar work

Multi-Granular Rationale-Guided Molecular LLM for Property Prediction

MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry

Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models