Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction

Yanyi Su; Yu Zeng; Hongshuai Wang; Chenyan Wang; Zhifeng Gao; Xiaohui Yang

cs.CLOct 14, 2026

Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction

Authors: Yanyi Su, Yu Zeng, Hongshuai Wang, Chenyan Wang, Zhifeng Gao, Xiaohui Yang

Organizations: State Key Laboratory of Physical Chemistry of Solid Surface, College of Chemistry and Chemical Engineering, Xiamen University, Xiamen, China · DP Technology, Beijing, China

Abstract

LLMs have achieved strong performance on chemical prediction benchmarks. However, existing evaluation methods typically provide target entity identity information directly, making it difficult to determine whether model predictions arise from transferable structure-property reasoning capabilities or rely on target-related contextual information or memory retrieval. We propose the Identity-Blinded Reasoning Diagnostic framework to evaluate chemical prediction models when target entity identity information is hidden. In this framework, the target is no longer directly defined by its entity identity, but instead by a known source entity and an explicit structural edit operation. This setting preserves the information required to determine the target while reducing the opportunity for models to access target identity cues. We evaluate this framework across six tasks spanning odor prediction, molecular property prediction, retrosynthesis, and reaction yield prediction. The results show that models generally achieve better prediction performance when source and target entity properties are consistent. Providing explicit source entity labels further strengthens this dependence. Moreover, while keeping the target entity, structural change, and ground-truth label unchanged, replacing only the source entity label causes some model predictions to systematically shift toward the substituted source labels. These results indicate that high benchmark accuracy in chemical prediction tasks does not necessarily imply that LLMs possess reliable structure-property reasoning capabilities. Our diagnostic framework provides a controlled evaluation approach for testing whether models can genuinely update predictions according to entity changes. The framework can be extended to other AI for Science tasks, particularly those where targets can be defined through transformations of known entities.

Figures & tables

Figure 1: Overview of the Identity-Blinded Reasoning Diagnostic. (a) Comparison between standard benchmark prediction (C0) and identity-blinded prediction (C1–C3), where the target identity is hidden and defined through a source entity and edit description. (b) Source label control conditions and group analysis. C1 provides no source label, C2 provides the true source label, and C3 replaces it with a false source label. CR denotes the performance ratio between label-consistent and label-inconsistent groups. (c) Contextual evidence augmentation, where independent predictions from task-specific predictors are added as external evidence.

Figure 2: Validation of target structure reconstruction under the identity-blinded setting. The LLM reconstructs the complete SMILES of the target molecule from the source molecule structure and the structural change description. The reaction yield task involves multicomponent formulations, for which single-molecule structure reconstruction is not applicable.

Condition	Information provided to the model
C0	Target structure representation (e.g., SMILES)
C1	Source structure and edit description
C2	Source structure, edit description, and true source label $y_{s}$
C3	Source structure, edit description, and false source label $\widetilde{y}_{s}$

Table 1: Prediction conditions.

Task	Label	N	C/I	Criterion	Metric
Structural odor	Descriptor set	179	122/57	Label overlap	F1
Chiral odor	Descriptor set	56	24/32	Label overlap	F1
BACE	Binary label	136	68/68	Activity agreement	Accuracy
HIV	Binary label	242	121/121	Activity agreement	Accuracy
Retrosynthesis	Reactant SMILES	42	28/14	Fingerprint similarity $\geq$ 0.5	Exact match
Reaction yield	Continuous value	100	50/50	Yield gap $\leq$ 15%	MAE

Table 2: Tasks and sample sizes in the main diagnostic.

Task	Claude	DeepSeek	GPT	Gemini
Structural odor F1	0.326	0.365	0.368	0.400
Chiral odor F1	0.198	0.250	0.234	0.272
BACE accuracy	57.40%	51.50%	51.50%	51.50%
HIV accuracy	57.90%	31.40%	34.30%	61.20%
Retrosynthesis EM	41.90%	34.90%	48.80%	44.20%
Yield MAE	32.34	56.46	48.00	35.04

Table 3: Prediction results under the target-identity-known condition C0.

Figure 3: Source information dependence analysis. (a) C1 provides only source structures and edit descriptions, and compares label-consistent and label-inconsistent groups. (b) C2 additionally provides true source labels and evaluates the same group comparison. Numbers above bars indicate collapse ratios (CR).

Figure 4: Effects of source label intervention on target prediction. (a) Binary prediction changes after replacing only the source label while keeping the target and other context fixed. Toward indicates movement toward the replacement source label, Away indicates movement away from it, and Unchanged indicates no prediction change. (b) Normalized directional response for set-valued predictions. (c) Same-direction response rate for numerical predictions, measuring whether prediction changes follow the direction of the replacement source label.

Figure 5: Effects of external predictive evidence augmentation on LLM prediction. Direct prediction is compared with prediction after adding task-relevant tool evidence. Columns denote models and rows denote odor-related tasks. Green and red regions indicate tasks with target-relevant evidence and limited evidence coverage, respectively.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Representative examples of the six diagnostic tasks. Each example shows the source molecule or reaction recipe, the natural-language edit instruction (Edit), the prediction query (Query), and the target structure or label after the edit. (a) Structural change odor prediction, where a terminal amine is replaced by a hydroxyl group. (b) Chirality change odor prediction, where a stereocenter is inverted while connectivity is preserved; red @ and @@ indicate chirality changes. (c) BACE activity prediction, where a fluorine substituent is removed and replaced by hydrogen. (d) HIV activity prediction, where a fluorine atom is added to an aromatic carbon. (e) Retrosynthesis prediction, where a fluorine substituent is replaced by a hydroxyl group and the corresponding reactants are predicted. (f) Reaction yield prediction, where MTBD is replaced by BTMG while other reaction conditions remain fixed. Orange and blue highlights indicate edited positions, and arrows indicate edit operations. The figure shows true source labels under C2 and false source labels under C3, while C1 provides no source label. All conditions share the same source entity, edit description, and prediction target. Target structures, Target SMILES, and target labels are shown only for illustration and evaluation and are not provided as model inputs. For yield prediction, the replaced component is part of the input condition, and the hidden target is the reaction yield.

Task	C0	Structure reconstruction	C1	C2	C3
Structural change olfaction	162	179	179	179	179
Chirality change olfaction	56	56	56	56	56
BACE	136	136	136	136	136
HIV	242	242	242	242	242
Retrosynthesis	43	43	42	42	42
Reaction yield	100	Not included	100	100	98

Appendix

Table A3: Sample sizes for each condition and paired analysis.

Task	Claude	DeepSeek	GPT	Gemini
Structural change	164/179, 91.62%	158/179, 88.27%	167/179, 93.30%	156/179, 87.15%
Chirality change	56/56, 100%	56/56, 100%	56/56, 100%	56/56, 100%
BACE	121/136, 88.97%	122/136, 89.71%	128/136, 94.12%	131/136, 96.32%
HIV	229/242, 94.63%	214/242, 88.43%	241/242, 99.59%	238/242, 98.35%
Retrosynthesis	41/43, 95.35%	37/43, 86.05%	42/43, 97.67%	42/43, 97.67%

Appendix

Table D1: Accuracy on the independent structure reconstruction task.

Model	BACE	HIV	Structural change	Chirality change	Yield
Claude	91.2%	94.6%	0.135	0.799	82.7%
DeepSeek	36.0%	19.8%	0.133	0.307	61.2%
GPT	58.8%	71.5%	0.114	0.783	98.0%
Gemini	43.4%	44.6%	0.059	0.869	84.7%

Appendix

Table D2: Paired response metrics under source label intervention between the C2 and C3 conditions.

Model	C2	C3
Claude-Opus-4.8	20.52	27.65
DeepSeek-V4-Pro-Preview	37.60	36.91
GPT-5.5	19.43	33.15
Gemini-3.1-Pro	17.51	29.09

Appendix

Table D3: Yield MAE on the same 98 paired samples.

Task	Model	C1	C2
Structural change	Claude-Opus-4.8	0.405 / 0.219	0.289 / 0.088
Structural change	DeepSeek-V4-Pro-Preview	0.426 / 0.243	0.431 / 0.160
Structural change	GPT-5.5	0.455 / 0.271	0.479 / 0.254
Structural change	Gemini-3.1-Pro	0.453 / 0.248	0.478 / 0.270
Chirality change	Claude-Opus-4.8	0.229 / 0.147	0.559 / 0.000
Chirality change	DeepSeek-V4-Pro-Preview	0.215 / 0.084	0.405 / 0.111

Appendix

Table D4: Performance of the label-consistent/label-inconsistent groups under the C1 and C2 conditions.

Task	Data source	Questions	Metric
Odor-to-mol retrieval	IFF	173	Hit@20
Primary odor selection	IFRA 2019	1146	Three-choice accuracy
Molecule-to-receptor	Emter et al.	145	Five-choice accuracy
Receptor-to-molecule	Emter et al.	145	Five-choice accuracy
Structural change olfaction	Molecular edit pairs from the main diagnostic	57	F1
Chirality change olfaction	Enantiomer pairs from the main diagnostic	32	F1

Appendix

Table E1: Evaluation sets for the contextual evidence experiments.

Task	Tool evidence	Description
Primary odor and odor retrieval	Molecule–descriptor matching scores and candidate rankings	NOSE training does not distinguish primary odors or assign different weights to the descriptors of a molecule. Task data are excluded from NOSE training to avoid data leakage.
Bidirectional receptor matching	Matching scores between query molecules and candidate receptors	The test set is independent of the NOSE training data to avoid data leakage.
Structural changes	Structures edited by RDKit and odor predictions from NOSE	The corresponding structural change test set is excluded from NOSE training to avoid data leakage.
Chirality changes	Structures inverted by RDKit and general odor predictions	NOSE models distance matrices and cannot distinguish chiral isomers.
Mixtures	Single-component odor predictions and retrieval using molecular representations weighted by component amounts	NOSE does not include a mixture input module.
Concentration changes	Molecular odor predictions without concentration parameters	NOSE does not include concentration-conditioned inputs.

Appendix

Table E2: Information provided by the tools.

Task	Gemini	DeepSeek	Gemma-4-31B	Qwen3.5-27B	Qwen3.5-9B
Odor-to-mol Hit@20	0.382 / 0.618	0.000 / 0.699	0.087 / 0.480	0.006 / 0.607	0.000 / 0.526
Molecule-to-receptor accuracy	0.372 / 0.455	0.435 / 0.559	0.138 / 0.559	0.166 / 0.559	0.207 / 0.559
Receptor-to-molecule accuracy	0.372 / 0.497	0.297 / 0.510	0.166 / 0.510	0.186 / 0.510	0.152 / 0.510
Primary odor accuracy	0.464 / 0.578	0.253 / 0.611	0.434 / 0.612	0.383 / 0.613	0.360 / 0.613
Structural change F1	0.270 / 0.338	0.160 / 0.312	0.073 / 0.207	0.054 / 0.237	0.035 / 0.248
Chirality change F1	0.074 / 0.114	0.111 / 0.134	0.000 / 0.018	0.000 / 0.104	0.000 / 0.169

Appendix

Table E3: Prediction results for direct prediction and prediction augmented by tool evidence. Each cell reports the result of direct prediction without tools followed by the result after adding tool evidence. Task scope and sample sizes are given in Table E1 .

Model	$n_{\mathrm{c}}/n_{\mathrm{i}}$	$S_{\mathrm{c}}$	$S_{\mathrm{i}}$	CR and 95% CI	$q$	Effect size
BACE
Gemini	68/68	57.4%	42.6%	1.34 [0.970, 1.950]	0.1546	0.295
GPT	68/68	60.3%	39.7%	1.52 [1.065, 2.222]	0.0361	0.415
Claude	68/68	72.1%	42.6%	1.69 [1.250, 2.400]	0.0017	0.605
DeepSeek	68/68	47.1%	52.9%	0.89 [0.622, 1.250]	0.6938	-0.118
HIV

Appendix

Table F1: Between-group difference tests under the C1 condition, with 12/24 significant comparisons. Subscripts c and i denote the label-consistent and label-inconsistent groups.

Model	$n_{\mathrm{c}}/n_{\mathrm{i}}$	$S_{\mathrm{c}}$	$S_{\mathrm{i}}$	CR [95% CI]	$q$	Effect
BACE
Gemini	68/68	76.5%	30.9%	2.48 [1.759, 3.846]	4.91e-07	0.950
GPT	68/68	86.8%	23.5%	3.69 [2.500, 6.222]	6.26e-13	1.384
Claude	68/68	100.0%	0.0%	Undefined	8.07e-39	3.142
DeepSeek	68/68	70.6%	48.5%	1.45 [1.103, 1.966]	0.0226	0.454
HIV

Appendix

Table F2: Between-group difference tests under the C2 condition, with 22/24 significant comparisons. Subscripts c and i denote the label-consistent and label-inconsistent groups.

Task	Model	Metric	n	Value	95% CI
BACE	Gemini-3.1-Pro	D_binary	136	43.4%	[34.9%, 52.1%]
BACE	GPT-5.5	D_binary	136	58.8%	[50.1%, 67.2%]
BACE	Claude-4.8-Opus	D_binary	136	91.2%	[85.1%, 95.4%]
BACE	DeepSeek-V4-Pro	D_binary	136	36.0%	[28.0%, 44.7%]
HIV	Gemini-3.1-Pro	D_binary	242	44.6%	[38.3%, 51.1%]
HIV	GPT-5.5	D_binary	242	71.5%	[65.4%, 77.1%]

Appendix

Table F3: Prediction response metrics and confidence intervals under source label intervention between the C2 and C3 conditions.

Chemical Reasoning in LLMs

See all 13 papers

MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry
Jun 4, 2026Joey Chan, Wonbin Kweon, Ashley Shin +4Retrieval-Augmented Generation LLM Inference
University of Illinois Urbana-Champaign · University of California, San Diego
Multi-Granular Rationale-Guided Molecular LLM for Property Prediction
Aug 11, 2026Junwoo Park, Minyoung Shin, Cheol Soon Lee +1Molecular Property Prediction Molecular Representation Learning
Sungkyunkwan University · Korea University
MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?
Oct 8, 2026Yuhang Xie, Zhiyi Shi, Zeyi Zhang +4Scientific Reasoning in Language Models LLM Evaluation
University of Tennessee, Knoxville · University of Illinois at Urbana-Champaign · University of California, Berkeley +2

Jun 4, 2026J/K move · Enter open · S save

MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry

Joey Chan, Wonbin Kweon, Ashley Shin, Niharika Bhattacharjee+3

University of Illinois Urbana-Champaign · University of California, San Diego

Retrieval-Augmented Generation LLM Inference Molecular Property Prediction

Open paper AskPDF arXiv

Large language models (LLMs) have shown promise for molecular property prediction, but their ability to reason over chemical structures remains limited, as molecular representations such as SMILES differ substantially from the natural language on which LLMs are primarily trained. To bridge this semantic and chemical knowledge gap, we propose MolE-RAG, a training-free, molecule-centric retrieval-augmented generation framework for LLM-based molecular property prediction. MolE-RAG augments each prediction with three complementary sources of inference-time context: retrieved chemistry literature, molecule-specific information including compound synonyms, identifiers, functional group annotations, and physicochemical descriptors, and structurally similar molecules retrieved from the training set. We evaluate MolE-RAG across nine molecular property prediction tasks using proprietary, chemistry-specialized, and open-source LLMs. Across general-purpose LLMs, MolE-RAG improves ROC-AUC by up to 28 percentage points on classification tasks and reduces regression RMSE by up to 67% relative to a SMILES-only baseline. We further find that the utility of each context source varies across models and tasks, with different models benefiting most from textual retrieval, molecular context, or structural retrieval. These results suggest that molecule-centric retrieval can improve LLM-based molecular property prediction without model fine-tuning while providing a flexible framework for integrating heterogeneous chemical knowledge at inference time.

Abstract

Chemical Reasoning in LLMs

Explore similar work

MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry

Multi-Granular Rationale-Guided Molecular LLM for Property Prediction

MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?