Chemical AI workflows routinely represent the same molecule through multiple co-referring identifiers, including trivial names, IUPAC names, and SMILES, implicitly assuming that these representations provide interchangeable access to an LLM's knowledge. To test this assumption, we first introduce a controlled benchmark and show that it does not hold. Across foundation models spanning 7B to 72B parameters, even chemistry-specialized models, co-referring molecular identifiers consistently expose different subsets of the model's knowledge. We then conduct a series of controlled behavioral, representational, and causal analyses to characterize this failure. We show that even when a model can identify a molecule from its SMILES and retrieve a fact through the corresponding name, the same fact can remain inaccessible directly from SMILES. We term this phenomenon cross-identifier binding failure. Further analysis of internal representations, conflicting identifiers, and parameter editing shows that neither improved chemical representation competence nor modification of individual facts is sufficient to establish reliable access across identifiers. Together, these results indicate that the core failure lies in the binding between an identifier and knowledge already present in the model. Motivated by this diagnosis, we introduce Binding Self-Distillation, a simple mitigation that uses lexical identity during training to transfer the model's existing knowledge access to the structural route. This enables factual retrieval directly from SMILES at inference time, without requiring a name lookup or training the underlying model from scratch.
Figures & tables
Figure 1: Equivalent molecular identifiers expose different knowledge. (A) A trivial name, IUPAC name, and SMILES string denote the same molecule (aspirin). (B–C) Identifier-conditioned accuracy over the 896 molecules in our benchmark, holding the molecule and queried property fixed while changing only the identifier. Lexical names provide substantially stronger access to entity-associated factual knowledge ( B , therapeutic indication), whereas structural identifiers provide stronger access to structure-computable knowledge ( C , benzene-ring presence). Representative results are shown for ChemDFM-v2.0-14B ( Zhao et al., 2025 ) .
Indication
Benzene
Nitrogens
Model
N
I
S
N
I
S
N
I
S
ChemDFM-v2.0-14B
38.8
2.2
2.1
76.7
93.2
94.6
41.6
70.3
59.6
ChemDFM-v1.5-8B
22.5
0.1
3.7
70.5
81.1
84.4
5.5
8.7
21.2
Qwen2.5-14B-Instruct
37.8
5.4
1.2
58.6
80.9
86.9
33.3
59.2
65.6
Qwen2.5-7B-Instruct
34.2
2.1
1.2
57.9
60.8
80.1
32.9
53.0
38.3
Qwen2-72B-Instruct
43.5
9.5
0.9
71.0
88.6
84.3
40.0
51.8
61.6
Table 1: Accuracy (%) under trivial name (N), IUPAC name (I), and SMILES (S). Indication denotes therapeutic-indication retrieval; Benzene denotes benzene-ring detection; and Nitrogens denotes nitrogen-atom counting. These three tasks provide the primary evidence for the cross-identifier dissociation; results for all seven benchmark tasks are reported in Appendix B.3 . Best performance within each model and task is shown in bold.
2AFC recognition
Free-form recognition
Model
n
SMILES
Name
Rest
n
SMILES
Name
Rest
ChemDFM-v2.0-14B
576
4.0
42.7
2.8
62
3.2
40.3
3.6
ChemDFM-v1.5-8B
487
3.7
22.0
4.9
72
8.3
19.4
3.9
Qwen2.5-14B-Instruct
332
1.2
44.9
1.4
1
0.0
100.0
1.3
Qwen2.5-7B-Instruct
240
0.0
40.4
1.7
1
0.0
100.0
1.2
Qwen2-72B-Instruct
376
1.3
42.6
0.6
7
0.0
42.9
0.9
Table 2: Therapeutic-indication accuracy (%) on molecules whose SMILES–name correspondence is recognized. “Rest” denotes SMILES accuracy on molecules outside the recognized subset. Higher accuracy indicates better performance; n denotes the size of the recognized subset.
Table 4: Conflicting identifiers control different outputs: Each conflicting prompt pairs the SMILES of molecule A with the incorrect trivial name of molecule B. For identity, we report how often the model returns the true molecule A or the supplied decoy B. For therapeutic indication, we report accuracy for B under the conflicting prompt and when B is queried by its name alone. For benzene-ring presence, we report accuracy for A under the conflicting prompt and when A is queried by its SMILES alone. All values are percentages. These quantities are diagnostic and do not have a single preferred direction.
Model
Peak depth
Recovery
Control
ChemDFM-v2.0-14B
0.71
105.4
-88.3
ChemDFM-v1.5-8B
0.50
76.4
-79.6
Qwen2.5-14B-Instruct
0.71
106.0
-197.7
Qwen2.5-7B-Instruct
0.75
104.9
-111.4
Meta-Llama-3-8B-Instruct
0.52
135.0
-179.9
Llama-3.1-8B-Instruct
0.50
112.4
-208.2
Table 5: Lexical activations causally restore retrieval from SMILES. Peak recovery (%) of the name–SMILES log-probability gap and the corresponding relative layer depth. Control patches use a different molecule with a different gold answer. Higher recovery indicates stronger restoration; peak depth is descriptive, while the control is expected not to yield positive recovery.
Figure 2: Cross-identifier representation alignment and chemistry specialization. Left: peak top-1 retrieval of the same molecule among 896 candidates across hidden-state depths; chance performance is 0.11% . Name–SMILES denotes trivial name → SMILES retrieval and IUPAC–SMILES denotes IUPAC name → SMILES retrieval. Right: comparison of the two general-purpose reference models with their chemistry-specialized counterparts. “Parametric facts known by name” denotes strict-scored trivial-name indication accuracy; “Name–SMILES binding” denotes the same cross-identifier retrieval metric; and “Indication via SMILES” denotes strict-scored SMILES-conditioned indication accuracy.
Model
SMILES
+ Lookup
Name
Gap rec.
ChemDFM-v2.0-14B
2.4
20.8
24.4
83.6
ChemDFM-v1.5-8B
5.4
14.2
14.5
96.7
Qwen2.5-14B-Instruct
1.2
25.3
29.2
86.1
Qwen2.5-7B-Instruct
0.6
22.6
23.2
97.3
Meta-Llama-3-8B-Instruct
1.2
26.8
27.1
98.8
Llama-3.1-8B-Instruct
0.6
26.8
28.9
92.6
Table 6: Aggregate accuracy (%) over available molecule–relation pairs. Higher values indicate better performance.
Model
Method
Effic.
SMILES
Leak
Specific [95% CI]
+Name
ChemDFM-v2.0-14B
ROME
52.2
22.1
19.8
+2.3 [-0.9,5.5]
45.5
FT
26.0
1.8
2.1
-0.3 [-1.6,1.2]
25.2
Qwen2.5-14B
ROME
17.9
4.4
4.9
-0.5 [-2.4,1.4]
12.3
FT
29.0
11.5
8.0
+3.5 [0.9,6.3]
26.7
Qwen2.5-7B
ROME
47.9
4.0
7.4
-3.4 [-5.7,-0.9]
11.4
FT
34.5
5.4
7.3
-1.9 [-4.4,0.8]
21.7
Table 7: All values are percentages. Specific transfer is computed as SMILES transfer minus leakage and is therefore reported in percentage points; brackets denote 95% bootstrap confidence intervals. Higher edit efficacy, specific transfer, and +Name recovery indicate better performance, while lower leakage is better. Raw SMILES transfer should be interpreted together with leakage.
Model
n
Facts/entity
ChemDFM-v2.0-14B
389
0.05→0.87
Qwen2.5-14B-Instruct
390
0.06→0.73
Llama-3.1-8B-Instruct
389
0.03→0.68
ChemDFM-v1.5-8B
47
0.51→0.85
Qwen2.5-7B-Instruct
351
0.04→0.85
Meta-Llama-3-8B-Instruct
378
0.02→0.78
Table 8: Facts/entity is the average number of available factual relations retrieved per molecule before and after the per-entity binding intervention. Higher Facts/entity indicates better performance; n is the model-specific number of molecules retained by the original BSD entity-selection pipeline.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Construction stage
# Molecules
ChEMBL molecules with qualifying phase-4 indications
3,025
Entity and structure filtering
1,553
Therapeutic-indication normalization and filtering
897
Successful PubChem IUPAC-name resolution
896
Appendix
Table 9: Benchmark construction. Successive filtering reduces the initial ChEMBL collection to 896 molecules with trivial-name, IUPAC-name, and SMILES identifier fields and scorable therapeutic-indication annotations.
Identifier
Prompt prefix
Trivial name
Consider the molecule commonly known as “{NAME}”.
IUPAC name
Consider the molecule named {IUPAC}.
SMILES
Consider the molecule with SMILES {SMILES}.
Appendix
Table 10: Identifier-specific prompt prefixes. Only the molecular identifier changes across the three conditions.
Task
Prompt
Identify
What is the common generic name of this molecule? Answer in English. Reply with the name only.
Indication
What medical condition is this substance primarily used to treat? Answer in English. Reply with the condition only. Do not explain.
Formula
What is the molecular formula of this molecule? Answer in English. Reply with the formula only, e.g. C9H8O4. Do not explain.
Benzene
Does this molecule contain a benzene ring? Answer in English. Reply with exactly one word: yes or no. Do not explain.
Nitrogens
How many nitrogen atoms does this molecule contain? Answer in English. Reply with a single integer only. Do not explain.
Ring Count
How many rings does this molecule contain? Answer in English. Reply with a single integer only. Do not explain.
Appendix
Table 11: Benchmark task prompts. Each task instruction is appended to one of the three identifier-specific prefixes in Table 10 .
Model
Identifier
Identify
Therapeutic indication
Benzene
Nitrogens
ChemDFM-v2.0-14B
Trivial name
93.3
38.8
76.7
41.6
IUPAC name
3.8
2.2
93.2
70.3
SMILES
6.9
2.1
94.6
59.6
ChemDFM-v1.5-8B
Trivial name
72.7
22.5
70.5
5.5
IUPAC name
2.7
0.1
81.1
8.7
SMILES
8.0
3.7
84.4
21.2
Appendix
Table 12: Primary cross-identifier results. Accuracy (%) when the underlying molecule and queried property are held fixed and only the supplied identifier changes. Identify denotes trivial-name recovery, Therapeutic indication denotes indication retrieval, Benzene denotes benzene-ring detection, and Nitrogens denotes nitrogen-atom counting. Best performance within each model and task is shown in bold.
Model
Identifier
Formula
Ring count
Heavy atoms
ChemDFM-v2.0-14B
Trivial name
7.7
24.4
6.8
IUPAC name
56.8
16.9
10.6
SMILES
43.5
7.7
9.8
ChemDFM-v1.5-8B
Trivial name
0.0
23.0
3.1
IUPAC name
0.0
26.0
2.3
SMILES
0.0
48.1
1.5
Appendix
Table 13: Additional structure-computable tasks. Accuracy (%) for molecular-formula prediction, total ring count, and heavy-atom count under each molecular identifier. Majority-class baselines for ring count and heavy-atom count over the 896 molecules are 23.9% and 5.5%, respectively. Best performance within each model and task is shown in bold.
Model
Identify: Name > SMILES
Indication: Name > SMILES
Benzene: SMILES > Name
ChemDFM-v2.0-14B
775 vs. 1 ( p=2×10−169 )
333 vs. 4 ( p=2×10−71 )
198 vs. 37 ( p=2×10−25 )
ChemDFM-v1.5-8B
598 vs. 19 ( p=9×10−120 )
183 vs. 14 ( p=5×10−33 )
212 vs. 88 ( p=1×10−12 )
Qwen2.5-14B-Instruct
765 vs. 1 ( p=3×10−167 )
331 vs. 3 ( p=1×10−71 )
322 vs. 68 ( p=1×10−37 )
Qwen2.5-7B-Instruct
370 vs. 1 ( p=2×10−81 )
299 vs. 4 ( p=5×10−64 )
292 vs. 93 ( p=6×10−24 )
Qwen2-72B-Instruct
553 vs. 4 ( p=2.9×10−119 )
384 vs. 2 ( p=8.9×10−84 )
215 vs. 96 ( p=2.2×10−11 )
Meta-Llama-3-8B-Instruct
281 vs. 5 ( p=2×10−59 )
338 vs. 0 ( p=5×10−75 )
248 vs. 51 ( p=9×10−30 )
Appendix
Table 14: Paired McNemar tests for cross-identifier differences. Each entry reports n10 versus n01 , where n10 favors the direction stated in the column heading, followed by the continuity-corrected McNemar p -value.
Property
Name
Name + SMILES
SMILES
Recovery
Benzene
71.1
85.7
84.3
111.0
Nitrogens
39.6
62.3
61.5
103.6
Appendix
Table 15: Reverse identifier bridge. Accuracy (%) for Qwen2-72B-Instruct. Reverse recovery is the fraction of the name-to-SMILES accuracy gap closed by adding SMILES to the name. Values above 100% indicate that the combined prompt slightly exceeds SMILES alone.
Model
Identity: A/B
Indication of B : conflict / name
Benzene of A : conflict / SMILES
ChemDFM-v2.0-14B
1.0 / 85.5
32.3 / 38.6
95.3 / 94.5
ChemDFM-v1.5-8B
0.6 / 73.4
17.4 / 23.0
80.5 / 85.0
Qwen2.5-14B-Instruct
0.0 / 81.1
35.9 / 37.8
84.9 / 86.9
Qwen2.5-7B-Instruct
0.0 / 60.4
32.1 / 33.9
75.9 / 80.2
Llama-3.1-8B-Instruct
0.0 / 86.8
37.3 / 39.0
89.3 / 90.3
Qwen2-72B-Instruct
0.0 / 77.8
43.2 / 44.1
83.9 / 84.3
Appendix
Table 16: Conflicting identifiers control different outputs: Each input pairs the SMILES of molecule A with the incorrect trivial name of molecule B . Identity reports retrieval of A versus B ; therapeutic indication compares retrieval of B ’s indication under conflict with retrieval from B ’s name alone; benzene compares accuracy under conflict with A ’s SMILES-only condition. All values are percentages.
Model
Correct
Near
Near
Gross
Gross
Near
accepted
reject
d′
reject
d′
2AFC
ChemDFM-v2.0-14B
93.6
36.9
1.19
84.3
2.53
73.9
ChemDFM-v1.5-8B
65.5
43.1
0.22
62.0
0.70
71.9
Qwen2.5-14B-Instruct
76.5
38.6
0.43
59.7
0.97
62.8
Qwen2.5-7B-Instruct
87.3
17.5
0.21
28.5
0.57
56.4
Qwen2-72B-Instruct
91.7
22.2
0.62
47.0
1.31
65.5
Appendix
Table 17: Direct detection of identifier conflicts. “Correct accepted” is the percentage of valid name–SMILES pairs that the model accepts as matching. “Near reject” and “Gross reject” are the percentages of near-analogue and gross mismatches correctly rejected, respectively. The corresponding d′ values measure discrimination while accounting for false alarms on correctly matched pairs. “Near 2AFC” is accuracy when choosing between the true trivial name and a structurally similar incorrect name. All rates are percentages except d′ .
Name → SMILES
IUPAC → SMILES
Model
Top-1
Peak idx.
Top-1
Peak idx.
States
ChemDFM-v2.0-14B
0.8
28
13.5
31
49
ChemDFM-v1.5-8B
2.8
24
15.0
32
33
Qwen2.5-14B-Instruct
0.7
36
2.6
30
49
Qwen2.5-7B-Instruct
0.7
3
1.8
25
29
Meta-Llama-3-8B-Instruct
0.8
13
3.9
18
33
Appendix
Table 18: Peak cross-identifier representation retrieval. Top-1 retrieval accuracy (%) among 896 molecule rows using mean-pooled whole-prompt hidden representations. Name–SMILES denotes trivial name → SMILES retrieval, and IUPAC–SMILES denotes IUPAC name → SMILES retrieval. “Peak idx.” is the hidden-state index attaining the maximum accuracy; index 0 is the embedding output. “States” is the total number of returned hidden states. Chance performance is 0.11% .
Relation
Molecules
Therapeutic indication
891
Molecular target
572
Mechanism of action
651
ATC therapeutic class
841
Total molecule–relation pairs
2,955
Appendix
Table 19: Coverage of the multi-relation evaluation set. Number of molecules with at least one accepted answer for each relation.
Model
Self-ID
Indication
Target
MoA
ATC
ChemDFM-v2.0-14B
7.1
4.9
11.4
4.5
1.4
ChemDFM-v1.5-8B
12.0
6.6
11.4
2.0
2.5
Qwen2.5-14B-Instruct
0.3
1.2
4.0
3.1
1.1
Qwen2.5-7B-Instruct
0.1
1.6
0.7
0.8
0.0
Meta-Llama-3-8B-Instruct
0.7
2.6
3.3
1.1
1.8
Llama-3.1-8B-Instruct
0.6
2.5
5.8
0.3
1.2
Appendix
Table 20: Self-resolution results. SMILES-to-name identification accuracy and downstream factual accuracy (%) when the generated name is inserted into the query. Relation columns are evaluated over molecules for which the corresponding relation is available.
Condition
Prompt
Efficacy
{Name.capitalize()} is primarily used to treat
IUPAC
The molecule named {iupac} is primarily used to treat
SMILES
The molecule with SMILES {smiles} is primarily used to treat
SMILES+Name
The molecule with SMILES {smiles}, which is known as "{name}", is primarily used to treat
Locality
{Unrelated_name.capitalize()} is primarily used to treat
Appendix
Table 21: Knowledge-editing evaluation prompts.
Model
Method
n
Efficacy
IUPAC
SMILES
Leak
Specific [95% CI]
SMILES+Name
Recovery
ChemDFM-v2.0-14B
ROME
389
52.2
24.2
22.1
19.8
+2.3 [ −0.9 ,5.5]
45.5
87.2
FT
389
26.0
2.8
1.8
2.1
-0.3 [ −1.6 ,1.2]
25.2
97.0
Qwen2.5-14B-Instruct
ROME
390
17.9
3.8
4.4
4.9
-0.5 [ −2.4 ,1.4]
12.3
68.6
FT
390
29.0
14.4
11.5
8.0
+3.5 [0.9,6.3]
26.7
92.0
Qwen2.5-7B-Instruct
ROME
351
47.9
4.3
4.0
7.4
-3.4 [ −5.7 , −0.9 ]
11.4
23.8
FT
351
34.5
8.5
5.4
7.3
-1.9 [ −4.4 ,0.8]
21.7
62.8
Appendix
Table 22: Full cross-identifier knowledge-editing results. Leak is the pooled locality rate and Specific is SMILES minus Leak. Brackets give 95% bootstrap confidence intervals. Recovery is SMILES+Name performance relative to name-conditioned efficacy.
Model
Method
nsucc
SMILES
Leak
Specific [95% CI]
SMILES+Name
ChemDFM-v2.0-14B
ROME
203
33.50
32.04
+1.46 [ −3.63 ,6.51]
67.98
FT
101
4.95
5.94
-0.99 [ −5.12 ,3.95]
64.36
Qwen2.5-14B-Instruct
ROME
70
18.57
20.99
-2.42 [ −10.98 ,6.83]
57.14
FT
113
30.97
20.14
+10.83 [3.39,18.63]
67.26
Qwen2.5-7B-Instruct
ROME
168
7.14
13.23
-6.09 [ −10.73 , −1.36 ]
22.62
FT
121
12.40
16.99
-4.59 [ −10.96 ,2.12]
46.28
Appendix
Table 23: Transfer conditioned on successful name edits. Specific is conditional SMILES transfer minus conditional leakage. Brackets give 95% bootstrap confidence intervals.
Model
Entities
ChemDFM-v2.0-14B
389
Qwen2.5-14B-Instruct
390
Llama-3.1-8B-Instruct
389
ChemDFM-v1.5-8B
47
Qwen2.5-7B-Instruct
351
Meta-Llama-3-8B-Instruct
378
Appendix
Table 24: Model-specific Binding Self-Distillation sets. The reported sets are the fixed entity sets selected by the original BSD entity-selection pipeline using therapeutic-indication retrieval through the trivial name.
Hyperparameter
Value
Editing mode
One adapter per entity
Target layer
24
LoRA layers
20–24
LoRA modules
gate/up/down projection
Rank r
8
Scaling α
16
Appendix
Table 25: Binding Self-Distillation configuration for the six verified local runs.
Indication
Target
MoA
ATC
Model
Before
After
Before
After
Before
After
Before
After
Facts/entity
ChemDFM-v2.0-14B
2.1
51.4
0.0
27.6
0.0
5.0
3.3
11.4
0.05 → 0.87
Qwen2.5-14B-Instruct
4.1
13.6
1.4
43.6
0.6
20.8
0.0
10.3
0.06 → 0.73
Llama-3.1-8B-Instruct
0.3
38.8
0.7
19.9
1.2
8.4
1.1
7.9
0.03 → 0.68
ChemDFM-v1.5-8B
25.5
42.6
5.1
23.1
17.1
12.2
6.8
13.6
0.51 → 0.85
Qwen2.5-7B-Instruct
1.1
36.2
0.4
32.3
0.0
19.9
2.7
9.1
0.04 → 0.85
Appendix
Table 26: Relation-wise Binding Self-Distillation results on the fixed entity sets selected by the original BSD entity-selection pipeline. Values are retrieval accuracy (%) through SMILES before and after the per-entity binding intervention.
Model
n
Facts/entity
Bridge
Recovery
ChemDFM-v2.0-14B
389
0.05 → 0.87
1.01
86%
Qwen2-72B-Instruct
435
0.01 → 0.85
1.34
63%
Appendix
Table 27: Binding Self-Distillation versus explicit lexical access on the fixed entity sets selected by the original BSD entity-selection pipeline. Bridge gives factual access when the correct name is explicitly supplied. Recovery measures the fraction of the SMILES-to-bridge gain recovered without the name at inference time.
Model
Entries
Symbol ID
Name ID
Accession ID
Loc. via Acc.
ID Bridge
Loc. Bridge
Decoy ID
ChemDFM-v2.0-14B
16,388
99.5
66.7
0.0
23.9
99.9
100.1
99.4
ChemDFM-v1.5-8B
1,000
5.5
18.8
0.0
0.0
20.0
5.6
1.0
Qwen2.5-14B-Instruct
1,000
98.0
60.8
0.0
22.0
100.7
101.0
98.6
Qwen2.5-7B-Instruct
1,000
86.4
51.8
0.0
21.9
98.3
101.4
85.3
Llama-3.1-8B-Instruct
1,000
90.8
60.3
0.0
9.3
98.9
97.6
90.9
Appendix
Table 28: Identifier-dependent access also appears for proteins. Identification columns report recovery of the official gene symbol. “Loc. via Acc.” is subcellular-location accuracy through the accession. ID and Loc. Bridge report the fraction of the respective accession–gene-symbol gap recovered when the correct gene symbol is supplied alongside the accession. Decoy ID measures whether identification follows an incorrect gene symbol in a conflicting-identifier prompt. All values except Entries are percentages.
Model
Items
Peak
Rel. depth
Recovery
Control
Rank layer
Rank top-1
ChemDFM-v2.0-14B
689
47
0.98
89.0
-104.2
40
11.6→50.7
ChemDFM-v1.5-8B
607
25
0.78
109.9
-113.7
28
12.7→62.6
Qwen2.5-14B-Instruct
677
39
0.81
94.6
-125.7
40
11.8→55.5
Qwen2.5-7B-Instruct
674
24
0.86
105.3
-137.3
24
11.9→37.8
Llama-3.1-8B-Instruct
680
31
0.97
88.0
-130.1
24
11.8→38.1
Appendix
Table 29: Activation patching in the protein domain. Peak is the layer maximizing mean normalized recovery after the complete coarse-to-fine sweep; relative depth is peak layer divided by model depth. Rank is evaluated at the coarse-sweep peak (see text). Rank top-1 reports whether the correct location has the highest teacher-forced score among the nine observed location classes.
Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark, MolRepBench, to evaluate LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI and canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a- judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes, and attention shows that representations are encoded differently inside the model; for example, structured representations require more attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.
Large language models (LLMs) are increasingly employed to predict molecular properties. However, prediction error alone cannot distinguish prediction from retrieval of published values. We audit 22 frontier models on 12 molecular regression datasets in a zero-shot setting, assessed against a molecule-blind reference derived from each dataset's labels. Significant retrieval is concentrated on 5 datasets, with isolated flagged LLMs elsewhere. Increasing the reasoning setting raises the number of flagged model--dataset combinations from 47 to 89 of 264. An in-context blinding experiment reduces retrieval but leaves a quarter of the combinations flagged. Blinding changes model rankings and increases errors. Because blinding also removes chemically interpretable structure, the error increase can only be partially attributed to reduced retrieval.
Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3
Large language models (LLMs) have shown promise for molecular property prediction, but their ability to reason over chemical structures remains limited, as molecular representations such as SMILES differ substantially from the natural language on which LLMs are primarily trained. To bridge this semantic and chemical knowledge gap, we propose MolE-RAG, a training-free, molecule-centric retrieval-augmented generation framework for LLM-based molecular property prediction. MolE-RAG augments each prediction with three complementary sources of inference-time context: retrieved chemistry literature, molecule-specific information including compound synonyms, identifiers, functional group annotations, and physicochemical descriptors, and structurally similar molecules retrieved from the training set. We evaluate MolE-RAG across nine molecular property prediction tasks using proprietary, chemistry-specialized, and open-source LLMs. Across general-purpose LLMs, MolE-RAG improves ROC-AUC by up to 28 percentage points on classification tasks and reduces regression RMSE by up to 67% relative to a SMILES-only baseline. We further find that the utility of each context source varies across models and tasks, with different models benefiting most from textual retrieval, molecular context, or structural retrieval. These results suggest that molecule-centric retrieval can improve LLM-based molecular property prediction without model fine-tuning while providing a flexible framework for integrating heterogeneous chemical knowledge at inference time.
Joey Chan, Wonbin Kweon, Ashley Shin +4
University of Illinois Urbana-Champaign · University of California, San Diego