Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction
Authors: Yanyi Su, Yu Zeng, Hongshuai Wang, Chenyan Wang, Zhifeng Gao, Xiaohui Yang
Organizations: State Key Laboratory of Physical Chemistry of Solid Surface, College of Chemistry and Chemical Engineering, Xiamen University, Xiamen, China · DP Technology, Beijing, China
LLMs have achieved strong performance on chemical prediction benchmarks. However, existing evaluation methods typically provide target entity identity information directly, making it difficult to determine whether model predictions arise from transferable structure-property reasoning capabilities or rely on target-related contextual information or memory retrieval. We propose the Identity-Blinded Reasoning Diagnostic framework to evaluate chemical prediction models when target entity identity information is hidden. In this framework, the target is no longer directly defined by its entity identity, but instead by a known source entity and an explicit structural edit operation. This setting preserves the information required to determine the target while reducing the opportunity for models to access target identity cues. We evaluate this framework across six tasks spanning odor prediction, molecular property prediction, retrosynthesis, and reaction yield prediction. The results show that models generally achieve better prediction performance when source and target entity properties are consistent. Providing explicit source entity labels further strengthens this dependence. Moreover, while keeping the target entity, structural change, and ground-truth label unchanged, replacing only the source entity label causes some model predictions to systematically shift toward the substituted source labels. These results indicate that high benchmark accuracy in chemical prediction tasks does not necessarily imply that LLMs possess reliable structure-property reasoning capabilities. Our diagnostic framework provides a controlled evaluation approach for testing whether models can genuinely update predictions according to entity changes. The framework can be extended to other AI for Science tasks, particularly those where targets can be defined through transformations of known entities.
Figures & tables
Figure 1: Overview of the Identity-Blinded Reasoning Diagnostic. (a) Comparison between standard benchmark prediction (C0) and identity-blinded prediction (C1–C3), where the target identity is hidden and defined through a source entity and edit description. (b) Source label control conditions and group analysis. C1 provides no source label, C2 provides the true source label, and C3 replaces it with a false source label. CR denotes the performance ratio between label-consistent and label-inconsistent groups. (c) Contextual evidence augmentation, where independent predictions from task-specific predictors are added as external evidence.
Figure 2: Validation of target structure reconstruction under the identity-blinded setting. The LLM reconstructs the complete SMILES of the target molecule from the source molecule structure and the structural change description. The reaction yield task involves multicomponent formulations, for which single-molecule structure reconstruction is not applicable.
Condition
Information provided to the model
C0
Target structure representation (e.g., SMILES)
C1
Source structure and edit description
C2
Source structure, edit description, and true source label ys
C3
Source structure, edit description, and false source label ys
Table 1: Prediction conditions.
Task
Label
N
C/I
Criterion
Metric
Structural odor
Descriptor set
179
122/57
Label overlap
F1
Chiral odor
Descriptor set
56
24/32
Label overlap
F1
BACE
Binary label
136
68/68
Activity agreement
Accuracy
HIV
Binary label
242
121/121
Activity agreement
Accuracy
Retrosynthesis
Reactant SMILES
42
28/14
Fingerprint similarity ≥ 0.5
Exact match
Reaction yield
Continuous value
100
50/50
Yield gap ≤ 15%
MAE
Table 2: Tasks and sample sizes in the main diagnostic.
Task
Claude
DeepSeek
GPT
Gemini
Structural odor F1
0.326
0.365
0.368
0.400
Chiral odor F1
0.198
0.250
0.234
0.272
BACE accuracy
57.40%
51.50%
51.50%
51.50%
HIV accuracy
57.90%
31.40%
34.30%
61.20%
Retrosynthesis EM
41.90%
34.90%
48.80%
44.20%
Yield MAE
32.34
56.46
48.00
35.04
Table 3: Prediction results under the target-identity-known condition C0.
Figure 3: Source information dependence analysis. (a) C1 provides only source structures and edit descriptions, and compares label-consistent and label-inconsistent groups. (b) C2 additionally provides true source labels and evaluates the same group comparison. Numbers above bars indicate collapse ratios (CR).
Figure 4: Effects of source label intervention on target prediction. (a) Binary prediction changes after replacing only the source label while keeping the target and other context fixed. Toward indicates movement toward the replacement source label, Away indicates movement away from it, and Unchanged indicates no prediction change. (b) Normalized directional response for set-valued predictions. (c) Same-direction response rate for numerical predictions, measuring whether prediction changes follow the direction of the replacement source label.
Figure 5: Effects of external predictive evidence augmentation on LLM prediction. Direct prediction is compared with prediction after adding task-relevant tool evidence. Columns denote models and rows denote odor-related tasks. Green and red regions indicate tasks with target-relevant evidence and limited evidence coverage, respectively.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Representative examples of the six diagnostic tasks. Each example shows the source molecule or reaction recipe, the natural-language edit instruction (Edit), the prediction query (Query), and the target structure or label after the edit. (a) Structural change odor prediction, where a terminal amine is replaced by a hydroxyl group. (b) Chirality change odor prediction, where a stereocenter is inverted while connectivity is preserved; red @ and @@ indicate chirality changes. (c) BACE activity prediction, where a fluorine substituent is removed and replaced by hydrogen. (d) HIV activity prediction, where a fluorine atom is added to an aromatic carbon. (e) Retrosynthesis prediction, where a fluorine substituent is replaced by a hydroxyl group and the corresponding reactants are predicted. (f) Reaction yield prediction, where MTBD is replaced by BTMG while other reaction conditions remain fixed. Orange and blue highlights indicate edited positions, and arrows indicate edit operations. The figure shows true source labels under C2 and false source labels under C3, while C1 provides no source label. All conditions share the same source entity, edit description, and prediction target. Target structures, Target SMILES, and target labels are shown only for illustration and evaluation and are not provided as model inputs. For yield prediction, the replaced component is part of the input condition, and the hidden target is the reaction yield.
Task
C0
Structure reconstruction
C1
C2
C3
Structural change olfaction
162
179
179
179
179
Chirality change olfaction
56
56
56
56
56
BACE
136
136
136
136
136
HIV
242
242
242
242
242
Retrosynthesis
43
43
42
42
42
Reaction yield
100
Not included
100
100
98
Appendix
Table A3: Sample sizes for each condition and paired analysis.
Task
Claude
DeepSeek
GPT
Gemini
Structural change
164/179, 91.62%
158/179, 88.27%
167/179, 93.30%
156/179, 87.15%
Chirality change
56/56, 100%
56/56, 100%
56/56, 100%
56/56, 100%
BACE
121/136, 88.97%
122/136, 89.71%
128/136, 94.12%
131/136, 96.32%
HIV
229/242, 94.63%
214/242, 88.43%
241/242, 99.59%
238/242, 98.35%
Retrosynthesis
41/43, 95.35%
37/43, 86.05%
42/43, 97.67%
42/43, 97.67%
Appendix
Table D1: Accuracy on the independent structure reconstruction task.
Model
BACE
HIV
Structural change
Chirality change
Yield
Claude
91.2%
94.6%
0.135
0.799
82.7%
DeepSeek
36.0%
19.8%
0.133
0.307
61.2%
GPT
58.8%
71.5%
0.114
0.783
98.0%
Gemini
43.4%
44.6%
0.059
0.869
84.7%
Appendix
Table D2: Paired response metrics under source label intervention between the C2 and C3 conditions.
Model
C2
C3
Claude-Opus-4.8
20.52
27.65
DeepSeek-V4-Pro-Preview
37.60
36.91
GPT-5.5
19.43
33.15
Gemini-3.1-Pro
17.51
29.09
Appendix
Table D3: Yield MAE on the same 98 paired samples.
Task
Model
C1
C2
Structural change
Claude-Opus-4.8
0.405 / 0.219
0.289 / 0.088
Structural change
DeepSeek-V4-Pro-Preview
0.426 / 0.243
0.431 / 0.160
Structural change
GPT-5.5
0.455 / 0.271
0.479 / 0.254
Structural change
Gemini-3.1-Pro
0.453 / 0.248
0.478 / 0.270
Chirality change
Claude-Opus-4.8
0.229 / 0.147
0.559 / 0.000
Chirality change
DeepSeek-V4-Pro-Preview
0.215 / 0.084
0.405 / 0.111
Appendix
Table D4: Performance of the label-consistent/label-inconsistent groups under the C1 and C2 conditions.
Task
Data source
Questions
Metric
Odor-to-mol retrieval
IFF
173
Hit@20
Primary odor selection
IFRA 2019
1146
Three-choice accuracy
Molecule-to-receptor
Emter et al.
145
Five-choice accuracy
Receptor-to-molecule
Emter et al.
145
Five-choice accuracy
Structural change olfaction
Molecular edit pairs from the main diagnostic
57
F1
Chirality change olfaction
Enantiomer pairs from the main diagnostic
32
F1
Appendix
Table E1: Evaluation sets for the contextual evidence experiments.
Task
Tool evidence
Description
Primary odor and odor retrieval
Molecule–descriptor matching scores and candidate rankings
NOSE training does not distinguish primary odors or assign different weights to the descriptors of a molecule. Task data are excluded from NOSE training to avoid data leakage.
Bidirectional receptor matching
Matching scores between query molecules and candidate receptors
The test set is independent of the NOSE training data to avoid data leakage.
Structural changes
Structures edited by RDKit and odor predictions from NOSE
The corresponding structural change test set is excluded from NOSE training to avoid data leakage.
Chirality changes
Structures inverted by RDKit and general odor predictions
NOSE models distance matrices and cannot distinguish chiral isomers.
Mixtures
Single-component odor predictions and retrieval using molecular representations weighted by component amounts
NOSE does not include a mixture input module.
Concentration changes
Molecular odor predictions without concentration parameters
NOSE does not include concentration-conditioned inputs.
Appendix
Table E2: Information provided by the tools.
Task
Gemini
DeepSeek
Gemma-4-31B
Qwen3.5-27B
Qwen3.5-9B
Odor-to-mol Hit@20
0.382 / 0.618
0.000 / 0.699
0.087 / 0.480
0.006 / 0.607
0.000 / 0.526
Molecule-to-receptor accuracy
0.372 / 0.455
0.435 / 0.559
0.138 / 0.559
0.166 / 0.559
0.207 / 0.559
Receptor-to-molecule accuracy
0.372 / 0.497
0.297 / 0.510
0.166 / 0.510
0.186 / 0.510
0.152 / 0.510
Primary odor accuracy
0.464 / 0.578
0.253 / 0.611
0.434 / 0.612
0.383 / 0.613
0.360 / 0.613
Structural change F1
0.270 / 0.338
0.160 / 0.312
0.073 / 0.207
0.054 / 0.237
0.035 / 0.248
Chirality change F1
0.074 / 0.114
0.111 / 0.134
0.000 / 0.018
0.000 / 0.104
0.000 / 0.169
Appendix
Table E3: Prediction results for direct prediction and prediction augmented by tool evidence. Each cell reports the result of direct prediction without tools followed by the result after adding tool evidence. Task scope and sample sizes are given in Table E1 .
Model
nc/ni
Sc
Si
CR and 95% CI
q
Effect size
BACE
Gemini
68/68
57.4%
42.6%
1.34 [0.970, 1.950]
0.1546
0.295
GPT
68/68
60.3%
39.7%
1.52 [1.065, 2.222]
0.0361
0.415
Claude
68/68
72.1%
42.6%
1.69 [1.250, 2.400]
0.0017
0.605
DeepSeek
68/68
47.1%
52.9%
0.89 [0.622, 1.250]
0.6938
-0.118
HIV
Appendix
Table F1: Between-group difference tests under the C1 condition, with 12/24 significant comparisons. Subscripts c and i denote the label-consistent and label-inconsistent groups.
Model
nc/ni
Sc
Si
CR [95% CI]
q
Effect
BACE
Gemini
68/68
76.5%
30.9%
2.48 [1.759, 3.846]
4.91e-07
0.950
GPT
68/68
86.8%
23.5%
3.69 [2.500, 6.222]
6.26e-13
1.384
Claude
68/68
100.0%
0.0%
Undefined
8.07e-39
3.142
DeepSeek
68/68
70.6%
48.5%
1.45 [1.103, 1.966]
0.0226
0.454
HIV
Appendix
Table F2: Between-group difference tests under the C2 condition, with 22/24 significant comparisons. Subscripts c and i denote the label-consistent and label-inconsistent groups.
Task
Model
Metric
n
Value
95% CI
BACE
Gemini-3.1-Pro
D_binary
136
43.4%
[34.9%, 52.1%]
BACE
GPT-5.5
D_binary
136
58.8%
[50.1%, 67.2%]
BACE
Claude-4.8-Opus
D_binary
136
91.2%
[85.1%, 95.4%]
BACE
DeepSeek-V4-Pro
D_binary
136
36.0%
[28.0%, 44.7%]
HIV
Gemini-3.1-Pro
D_binary
242
44.6%
[38.3%, 51.1%]
HIV
GPT-5.5
D_binary
242
71.5%
[65.4%, 77.1%]
Appendix
Table F3: Prediction response metrics and confidence intervals under source label intervention between the C2 and C3 conditions.
Large language models (LLMs) have shown promise for molecular property prediction, but their ability to reason over chemical structures remains limited, as molecular representations such as SMILES differ substantially from the natural language on which LLMs are primarily trained. To bridge this semantic and chemical knowledge gap, we propose MolE-RAG, a training-free, molecule-centric retrieval-augmented generation framework for LLM-based molecular property prediction. MolE-RAG augments each prediction with three complementary sources of inference-time context: retrieved chemistry literature, molecule-specific information including compound synonyms, identifiers, functional group annotations, and physicochemical descriptors, and structurally similar molecules retrieved from the training set. We evaluate MolE-RAG across nine molecular property prediction tasks using proprietary, chemistry-specialized, and open-source LLMs. Across general-purpose LLMs, MolE-RAG improves ROC-AUC by up to 28 percentage points on classification tasks and reduces regression RMSE by up to 67% relative to a SMILES-only baseline. We further find that the utility of each context source varies across models and tasks, with different models benefiting most from textual retrieval, molecular context, or structural retrieval. These results suggest that molecule-centric retrieval can improve LLM-based molecular property prediction without model fine-tuning while providing a flexible framework for integrating heterogeneous chemical knowledge at inference time.
Joey Chan, Wonbin Kweon, Ashley Shin +4
University of Illinois Urbana-Champaign · University of California, San Diego
Large language models (LLMs) are widely applied across chemical tasks, such as molecular property prediction, which underpins drug discovery. Molecular LLMs represent a molecule through several modalities, notably a 1D SMILES sequence or a 2D molecular graph. Both encode molecular information implicitly, so the contribution of individual substructures remains opaque. Retrieval and augmentation methods add context, but from external sources. However, the cues chemists reason over are the internal substructures that drive a property up or down. We propose MR-MoL, a multi-granular rationale-guided molecular LLM that supplies this evidence directly. A fine-tuned GNN scores each substructure through masking, and the most influential ones are serialized as a ranked, direction-tagged rationale that the LLM reads alongside the SMILES sequence and molecular graph. The rationale spans three levels of granularity: Murcko scaffolds with their side chains, BRICS fragments, and functional groups. This is, to our knowledge, the first method to expose GNN-derived attributions to an LLM as evidence for property prediction. On eight MoleculeNet tasks, MR-MoL achieves the best overall results among generalist models and narrows the gap to specialist models tuned for each task. Five diagnostics further confirm that the model reads the rationale rather than merely benefiting from its presence. Its direction, rank, and substructure each shape the prediction, and its attributions reproduce known structure-property relationships.
Large language models (LLMs) are increasingly used in molecular discovery and structure elucidation, where decisions depend on physical observations. Existing evaluations primarily focus on end-to-end molecular reconstruction, where correct predictions do not necessarily indicate reliance on the supplied properties. We introduce MolP2SBench, a benchmark for molecular property-to-structure (P2S) reasoning in LLMs. Each task presents a masked molecular scaffold with given properties and asks the model to select the missing structure from plausible candidates. We shuffle or replace properties to test reliance on correct property-structure pairing and responsiveness to counterfactual property changes. A channel-wise leaderboard uses Property-Grounded Recovery (PGR), giving equal weight to source and counterfactual recovery. Evaluating seven models, we find that LLMs can use physical properties for structure prediction, but performance varies substantially across property types. Higher accuracy with a property does not necessarily imply greater responsiveness to it: a model may benefit more from 1H NMR than from 13C NMR, yet be less likely to change its prediction when the 1H data is replaced. LLM performance also need not reflect the relative difficulty of these properties for human chemists. Although UV-Vis spectra provide less direct structural information than NMR spectra, Astra achieves higher accuracy with UV-Vis data than with NMR peaks. MolP2SBench supports studying how LLMs extract and use physical-property information, revealing patterns that may differ from conventional chemical intuition and providing a foundation for more reliably property-grounded models for molecular and quantum-chemical inference. Code and dataset: https://github.com/DUAL-Xiao/MolP2SBench.
Yuhang Xie, Zhiyi Shi, Zeyi Zhang +4
University of Tennessee, Knoxville · University of Illinois at Urbana-Champaign · University of California, Berkeley +2