Same molecule, Different Knowledge: Cross-Identifier Binding Failures in LLMs For chemistry

Khouloud Saadi; Di Wang

cs.LGOct 22, 2026

Same molecule, Different Knowledge: Cross-Identifier Binding Failures in LLMs For chemistry

Authors: Khouloud Saadi, Di Wang

Organizations: Department of Computer Science, King Abdullah University of Science and Technology, KAUST, Thuwal, Saudi Arabia

Abstract

Chemical AI workflows routinely represent the same molecule through multiple co-referring identifiers, including trivial names, IUPAC names, and SMILES, implicitly assuming that these representations provide interchangeable access to an LLM's knowledge. To test this assumption, we first introduce a controlled benchmark and show that it does not hold. Across foundation models spanning 7B to 72B parameters, even chemistry-specialized models, co-referring molecular identifiers consistently expose different subsets of the model's knowledge. We then conduct a series of controlled behavioral, representational, and causal analyses to characterize this failure. We show that even when a model can identify a molecule from its SMILES and retrieve a fact through the corresponding name, the same fact can remain inaccessible directly from SMILES. We term this phenomenon cross-identifier binding failure. Further analysis of internal representations, conflicting identifiers, and parameter editing shows that neither improved chemical representation competence nor modification of individual facts is sufficient to establish reliable access across identifiers. Together, these results indicate that the core failure lies in the binding between an identifier and knowledge already present in the model. Motivated by this diagnosis, we introduce Binding Self-Distillation, a simple mitigation that uses lexical identity during training to transfer the model's existing knowledge access to the structural route. This enables factual retrieval directly from SMILES at inference time, without requiring a name lookup or training the underlying model from scratch.

Figures & tables

Figure 1: Equivalent molecular identifiers expose different knowledge. (A) A trivial name, IUPAC name, and SMILES string denote the same molecule (aspirin). (B–C) Identifier-conditioned accuracy over the 896 molecules in our benchmark, holding the molecule and queried property fixed while changing only the identifier. Lexical names provide substantially stronger access to entity-associated factual knowledge ( B , therapeutic indication), whereas structural identifiers provide stronger access to structure-computable knowledge ( C , benzene-ring presence). Representative results are shown for ChemDFM-v2.0-14B ( Zhao et al., 2025 ) .

	Indication			Benzene			Nitrogens
Model	N	I	S	N	I	S	N	I	S
ChemDFM-v2.0-14B	38.8	2.2	2.1	76.7	93.2	94.6	41.6	70.3	59.6
ChemDFM-v1.5-8B	22.5	0.1	3.7	70.5	81.1	84.4	5.5	8.7	21.2
Qwen2.5-14B-Instruct	37.8	5.4	1.2	58.6	80.9	86.9	33.3	59.2	65.6
Qwen2.5-7B-Instruct	34.2	2.1	1.2	57.9	60.8	80.1	32.9	53.0	38.3
Qwen2-72B-Instruct	43.5	9.5	0.9	71.0	88.6	84.3	40.0	51.8	61.6

Table 1: Accuracy (%) under trivial name (N), IUPAC name (I), and SMILES (S). Indication denotes therapeutic-indication retrieval; Benzene denotes benzene-ring detection; and Nitrogens denotes nitrogen-atom counting. These three tasks provide the primary evidence for the cross-identifier dissociation; results for all seven benchmark tasks are reported in Appendix B.3 . Best performance within each model and task is shown in bold.

Model	$n$	SMILES	Name	Rest	$n$	SMILES	Name	Rest
	2AFC recognition				Free-form recognition
ChemDFM-v2.0-14B	576	4.0	42.7	2.8	62	3.2	40.3	3.6
ChemDFM-v1.5-8B	487	3.7	22.0	4.9	72	8.3	19.4	3.9
Qwen2.5-14B-Instruct	332	1.2	44.9	1.4	1	0.0	100.0	1.3
Qwen2.5-7B-Instruct	240	0.0	40.4	1.7	1	0.0	100.0	1.2
Qwen2-72B-Instruct	376	1.3	42.6	0.6	7	0.0	42.9	0.9

Table 2: Therapeutic-indication accuracy (%) on molecules whose SMILES–name correspondence is recognized. “Rest” denotes SMILES accuracy on molecules outside the recognized subset. Higher accuracy indicates better performance;

n

denotes the size of the recognized subset.

Model	SMILES	+ Name	Name	Recovery
ChemDFM-v2.0-14B	2.1	36.3	38.6	93.6
ChemDFM-v1.5-8B	3.9	21.4	23.0	91.8
Qwen2.5-14B-Instruct	1.3	37.6	37.8	99.4
Qwen2.5-7B-Instruct	1.2	32.8	33.9	96.6
Llama-3.1-8B-Instruct	1.2	38.6	39.0	99.1
Qwen2-72B-Instruct	0.8	42.5	43.5	97.7

Table 3: Supplying lexical identity restores factual access. Therapeutic-indication accuracy (%). Higher values indicate better performance.

	Identity under conflict		Indication of B		Benzene of A
Model	True A	Decoy B	Conflict	Name only	Conflict	SMILES only
ChemDFM-v2.0-14B	1.0	85.5	32.3	38.6	95.3	94.5
ChemDFM-v1.5-8B	0.6	73.4	17.4	23.0	80.5	85.0
Qwen2.5-14B-Instruct	0.0	81.1	35.9	37.8	84.9	86.9
Qwen2.5-7B-Instruct	0.0	60.4	32.1	33.9	75.9	80.2
Llama-3.1-8B-Instruct	0.0	86.8	37.3	39.0	89.3	90.3

Table 4: Conflicting identifiers control different outputs: Each conflicting prompt pairs the SMILES of molecule A with the incorrect trivial name of molecule B. For identity, we report how often the model returns the true molecule A or the supplied decoy B. For therapeutic indication, we report accuracy for B under the conflicting prompt and when B is queried by its name alone. For benzene-ring presence, we report accuracy for A under the conflicting prompt and when A is queried by its SMILES alone. All values are percentages. These quantities are diagnostic and do not have a single preferred direction.

Model	Peak depth	Recovery	Control
ChemDFM-v2.0-14B	0.71	105.4	-88.3
ChemDFM-v1.5-8B	0.50	76.4	-79.6
Qwen2.5-14B-Instruct	0.71	106.0	-197.7
Qwen2.5-7B-Instruct	0.75	104.9	-111.4
Meta-Llama-3-8B-Instruct	0.52	135.0	-179.9
Llama-3.1-8B-Instruct	0.50	112.4	-208.2

Table 5: Lexical activations causally restore retrieval from SMILES. Peak recovery (%) of the name–SMILES log-probability gap and the corresponding relative layer depth. Control patches use a different molecule with a different gold answer. Higher recovery indicates stronger restoration; peak depth is descriptive, while the control is expected not to yield positive recovery.

Figure 2: Cross-identifier representation alignment and chemistry specialization. Left: peak top-1 retrieval of the same molecule among 896 candidates across hidden-state depths; chance performance is $0.11\%$ . Name–SMILES denotes trivial name $\rightarrow$ SMILES retrieval and IUPAC–SMILES denotes IUPAC name $\rightarrow$ SMILES retrieval. Right: comparison of the two general-purpose reference models with their chemistry-specialized counterparts. “Parametric facts known by name” denotes strict-scored trivial-name indication accuracy; “Name–SMILES binding” denotes the same cross-identifier retrieval metric; and “Indication via SMILES” denotes strict-scored SMILES-conditioned indication accuracy.

Model	SMILES	+ Lookup	Name	Gap rec.
ChemDFM-v2.0-14B	2.4	20.8	24.4	83.6
ChemDFM-v1.5-8B	5.4	14.2	14.5	96.7
Qwen2.5-14B-Instruct	1.2	25.3	29.2	86.1
Qwen2.5-7B-Instruct	0.6	22.6	23.2	97.3
Meta-Llama-3-8B-Instruct	1.2	26.8	27.1	98.8
Llama-3.1-8B-Instruct	0.6	26.8	28.9	92.6

Table 6: Aggregate accuracy (%) over available molecule–relation pairs. Higher values indicate better performance.

Model	Method	Effic.	SMILES	Leak	Specific [95% CI]	+Name
ChemDFM-v2.0-14B	ROME	52.2	22.1	19.8	+2.3 [-0.9,5.5]	45.5
	FT	26.0	1.8	2.1	-0.3 [-1.6,1.2]	25.2
Qwen2.5-14B	ROME	17.9	4.4	4.9	-0.5 [-2.4,1.4]	12.3
	FT	29.0	11.5	8.0	+3.5 [0.9,6.3]	26.7
Qwen2.5-7B	ROME	47.9	4.0	7.4	-3.4 [-5.7,-0.9]	11.4
	FT	34.5	5.4	7.3	-1.9 [-4.4,0.8]	21.7

Table 7: All values are percentages. Specific transfer is computed as SMILES transfer minus leakage and is therefore reported in percentage points; brackets denote 95% bootstrap confidence intervals. Higher edit efficacy, specific transfer, and +Name recovery indicate better performance, while lower leakage is better. Raw SMILES transfer should be interpreted together with leakage.

Model	$n$	Facts/entity
ChemDFM-v2.0-14B	389	$0.05\rightarrow 0.87$
Qwen2.5-14B-Instruct	390	$0.06\rightarrow 0.73$
Llama-3.1-8B-Instruct	389	$0.03\rightarrow 0.68$
ChemDFM-v1.5-8B	47	$0.51\rightarrow 0.85$
Qwen2.5-7B-Instruct	351	$0.04\rightarrow 0.85$
Meta-Llama-3-8B-Instruct	378	$0.02\rightarrow 0.78$

Table 8: Facts/entity is the average number of available factual relations retrieved per molecule before and after the per-entity binding intervention. Higher Facts/entity indicates better performance;

n

is the model-specific number of molecules retained by the original BSD entity-selection pipeline.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Construction stage	# Molecules
ChEMBL molecules with qualifying phase-4 indications	3,025
Entity and structure filtering	1,553
Therapeutic-indication normalization and filtering	897
Successful PubChem IUPAC-name resolution	896

Appendix

Table 9: Benchmark construction. Successive filtering reduces the initial ChEMBL collection to 896 molecules with trivial-name, IUPAC-name, and SMILES identifier fields and scorable therapeutic-indication annotations.

Identifier	Prompt prefix
Trivial name	Consider the molecule commonly known as “{NAME}”.
IUPAC name	Consider the molecule named {IUPAC}.
SMILES	Consider the molecule with SMILES {SMILES}.

Appendix

Table 10: Identifier-specific prompt prefixes. Only the molecular identifier changes across the three conditions.

Task	Prompt
Identify	What is the common generic name of this molecule? Answer in English. Reply with the name only.
Indication	What medical condition is this substance primarily used to treat? Answer in English. Reply with the condition only. Do not explain.
Formula	What is the molecular formula of this molecule? Answer in English. Reply with the formula only, e.g. C9H8O4. Do not explain.
Benzene	Does this molecule contain a benzene ring? Answer in English. Reply with exactly one word: yes or no. Do not explain.
Nitrogens	How many nitrogen atoms does this molecule contain? Answer in English. Reply with a single integer only. Do not explain.
Ring Count	How many rings does this molecule contain? Answer in English. Reply with a single integer only. Do not explain.

Appendix

Table 11: Benchmark task prompts. Each task instruction is appended to one of the three identifier-specific prefixes in Table 10 .

Model	Identifier	Identify	Therapeutic indication	Benzene	Nitrogens
ChemDFM-v2.0-14B	Trivial name	93.3	38.8	76.7	41.6
	IUPAC name	3.8	2.2	93.2	70.3
	SMILES	6.9	2.1	94.6	59.6
ChemDFM-v1.5-8B	Trivial name	72.7	22.5	70.5	5.5
	IUPAC name	2.7	0.1	81.1	8.7
	SMILES	8.0	3.7	84.4	21.2

Appendix

Table 12: Primary cross-identifier results. Accuracy (%) when the underlying molecule and queried property are held fixed and only the supplied identifier changes. Identify denotes trivial-name recovery, Therapeutic indication denotes indication retrieval, Benzene denotes benzene-ring detection, and Nitrogens denotes nitrogen-atom counting. Best performance within each model and task is shown in bold.

Model	Identifier	Formula	Ring count	Heavy atoms
ChemDFM-v2.0-14B	Trivial name	7.7	24.4	6.8
	IUPAC name	56.8	16.9	10.6
	SMILES	43.5	7.7	9.8
ChemDFM-v1.5-8B	Trivial name	0.0	23.0	3.1
	IUPAC name	0.0	26.0	2.3
	SMILES	0.0	48.1	1.5

Appendix

Table 13: Additional structure-computable tasks. Accuracy (%) for molecular-formula prediction, total ring count, and heavy-atom count under each molecular identifier. Majority-class baselines for ring count and heavy-atom count over the 896 molecules are 23.9% and 5.5%, respectively. Best performance within each model and task is shown in bold.

Model	Identify: Name $>$ SMILES	Indication: Name $>$ SMILES	Benzene: SMILES $>$ Name
ChemDFM-v2.0-14B	775 vs. 1 ( $p=2\times 10^{-169}$ )	333 vs. 4 ( $p=2\times 10^{-71}$ )	198 vs. 37 ( $p=2\times 10^{-25}$ )
ChemDFM-v1.5-8B	598 vs. 19 ( $p=9\times 10^{-120}$ )	183 vs. 14 ( $p=5\times 10^{-33}$ )	212 vs. 88 ( $p=1\times 10^{-12}$ )
Qwen2.5-14B-Instruct	765 vs. 1 ( $p=3\times 10^{-167}$ )	331 vs. 3 ( $p=1\times 10^{-71}$ )	322 vs. 68 ( $p=1\times 10^{-37}$ )
Qwen2.5-7B-Instruct	370 vs. 1 ( $p=2\times 10^{-81}$ )	299 vs. 4 ( $p=5\times 10^{-64}$ )	292 vs. 93 ( $p=6\times 10^{-24}$ )
Qwen2-72B-Instruct	553 vs. 4 ( $p=2.9\times 10^{-119}$ )	384 vs. 2 ( $p=8.9\times 10^{-84}$ )	215 vs. 96 ( $p=2.2\times 10^{-11}$ )
Meta-Llama-3-8B-Instruct	281 vs. 5 ( $p=2\times 10^{-59}$ )	338 vs. 0 ( $p=5\times 10^{-75}$ )	248 vs. 51 ( $p=9\times 10^{-30}$ )

Appendix

Table 14: Paired McNemar tests for cross-identifier differences. Each entry reports

n_{10}

versus

n_{01}

, where

n_{10}

favors the direction stated in the column heading, followed by the continuity-corrected McNemar

p

-value.

Property	Name	Name + SMILES	SMILES	Recovery
Benzene	71.1	85.7	84.3	111.0
Nitrogens	39.6	62.3	61.5	103.6

Appendix

Table 15: Reverse identifier bridge. Accuracy (%) for Qwen2-72B-Instruct. Reverse recovery is the fraction of the name-to-SMILES accuracy gap closed by adding SMILES to the name. Values above

100\%

indicate that the combined prompt slightly exceeds SMILES alone.

Model	Identity: $A/B$	Indication of $B$ : conflict / name	Benzene of $A$ : conflict / SMILES
ChemDFM-v2.0-14B	1.0 / 85.5	32.3 / 38.6	95.3 / 94.5
ChemDFM-v1.5-8B	0.6 / 73.4	17.4 / 23.0	80.5 / 85.0
Qwen2.5-14B-Instruct	0.0 / 81.1	35.9 / 37.8	84.9 / 86.9
Qwen2.5-7B-Instruct	0.0 / 60.4	32.1 / 33.9	75.9 / 80.2
Llama-3.1-8B-Instruct	0.0 / 86.8	37.3 / 39.0	89.3 / 90.3
Qwen2-72B-Instruct	0.0 / 77.8	43.2 / 44.1	83.9 / 84.3

Appendix

Table 16: Conflicting identifiers control different outputs: Each input pairs the SMILES of molecule

A

with the incorrect trivial name of molecule

B

. Identity reports retrieval of

A

versus

B

; therapeutic indication compares retrieval of

B

’s indication under conflict with retrieval from

B

’s name alone; benzene compares accuracy under conflict with

A

’s SMILES-only condition. All values are percentages.

Model	Correct	Near	Near	Gross	Gross	Near
	accepted	reject	$d^{\prime}$	reject	$d^{\prime}$	2AFC
ChemDFM-v2.0-14B	93.6	36.9	1.19	84.3	2.53	73.9
ChemDFM-v1.5-8B	65.5	43.1	0.22	62.0	0.70	71.9
Qwen2.5-14B-Instruct	76.5	38.6	0.43	59.7	0.97	62.8
Qwen2.5-7B-Instruct	87.3	17.5	0.21	28.5	0.57	56.4
Qwen2-72B-Instruct	91.7	22.2	0.62	47.0	1.31	65.5

Appendix

Table 17: Direct detection of identifier conflicts. “Correct accepted” is the percentage of valid name–SMILES pairs that the model accepts as matching. “Near reject” and “Gross reject” are the percentages of near-analogue and gross mismatches correctly rejected, respectively. The corresponding

d^{\prime}

values measure discrimination while accounting for false alarms on correctly matched pairs. “Near 2AFC” is accuracy when choosing between the true trivial name and a structurally similar incorrect name. All rates are percentages except

d^{\prime}

Model	Top-1	Peak idx.	Top-1	Peak idx.	States
	Name $\rightarrow$ SMILES		IUPAC $\rightarrow$ SMILES
ChemDFM-v2.0-14B	0.8	28	13.5	31	49
ChemDFM-v1.5-8B	2.8	24	15.0	32	33
Qwen2.5-14B-Instruct	0.7	36	2.6	30	49
Qwen2.5-7B-Instruct	0.7	3	1.8	25	29
Meta-Llama-3-8B-Instruct	0.8	13	3.9	18	33

Appendix

Table 18: Peak cross-identifier representation retrieval. Top-1 retrieval accuracy (%) among 896 molecule rows using mean-pooled whole-prompt hidden representations. Name–SMILES denotes trivial name

\rightarrow

SMILES retrieval, and IUPAC–SMILES denotes IUPAC name

\rightarrow

SMILES retrieval. “Peak idx.” is the hidden-state index attaining the maximum accuracy; index

0

is the embedding output. “States” is the total number of returned hidden states. Chance performance is

0.11\%

Relation	Molecules
Therapeutic indication	891
Molecular target	572
Mechanism of action	651
ATC therapeutic class	841
Total molecule–relation pairs	2,955

Appendix

Table 19: Coverage of the multi-relation evaluation set. Number of molecules with at least one accepted answer for each relation.

Model	Self-ID	Indication	Target	MoA	ATC
ChemDFM-v2.0-14B	7.1	4.9	11.4	4.5	1.4
ChemDFM-v1.5-8B	12.0	6.6	11.4	2.0	2.5
Qwen2.5-14B-Instruct	0.3	1.2	4.0	3.1	1.1
Qwen2.5-7B-Instruct	0.1	1.6	0.7	0.8	0.0
Meta-Llama-3-8B-Instruct	0.7	2.6	3.3	1.1	1.8
Llama-3.1-8B-Instruct	0.6	2.5	5.8	0.3	1.2

Appendix

Table 20: Self-resolution results. SMILES-to-name identification accuracy and downstream factual accuracy (%) when the generated name is inserted into the query. Relation columns are evaluated over molecules for which the corresponding relation is available.

Condition	Prompt
Efficacy	{Name.capitalize()} is primarily used to treat
IUPAC	The molecule named {iupac} is primarily used to treat
SMILES	The molecule with SMILES {smiles} is primarily used to treat
SMILES+Name	The molecule with SMILES {smiles}, which is known as "{name}", is primarily used to treat
Locality	{Unrelated_name.capitalize()} is primarily used to treat

Appendix

Table 21: Knowledge-editing evaluation prompts.

Model	Method	$n$	Efficacy	IUPAC	SMILES	Leak	Specific [95% CI]	SMILES+Name	Recovery
ChemDFM-v2.0-14B	ROME	389	52.2	24.2	22.1	19.8	+2.3 [ $-0.9$ ,5.5]	45.5	87.2
	FT	389	26.0	2.8	1.8	2.1	-0.3 [ $-1.6$ ,1.2]	25.2	97.0
Qwen2.5-14B-Instruct	ROME	390	17.9	3.8	4.4	4.9	-0.5 [ $-2.4$ ,1.4]	12.3	68.6
	FT	390	29.0	14.4	11.5	8.0	+3.5 [0.9,6.3]	26.7	92.0
Qwen2.5-7B-Instruct	ROME	351	47.9	4.3	4.0	7.4	-3.4 [ $-5.7$ , $-0.9$ ]	11.4	23.8
	FT	351	34.5	8.5	5.4	7.3	-1.9 [ $-4.4$ ,0.8]	21.7	62.8

Appendix

Table 22: Full cross-identifier knowledge-editing results. Leak is the pooled locality rate and Specific is SMILES minus Leak. Brackets give 95% bootstrap confidence intervals. Recovery is SMILES+Name performance relative to name-conditioned efficacy.

Model	Method	$n_{\mathrm{succ}}$	SMILES	Leak	Specific [95% CI]	SMILES+Name
ChemDFM-v2.0-14B	ROME	203	33.50	32.04	+1.46 [ $-3.63$ ,6.51]	67.98
	FT	101	4.95	5.94	-0.99 [ $-5.12$ ,3.95]	64.36
Qwen2.5-14B-Instruct	ROME	70	18.57	20.99	-2.42 [ $-10.98$ ,6.83]	57.14
	FT	113	30.97	20.14	+10.83 [3.39,18.63]	67.26
Qwen2.5-7B-Instruct	ROME	168	7.14	13.23	-6.09 [ $-10.73$ , $-1.36$ ]	22.62
	FT	121	12.40	16.99	-4.59 [ $-10.96$ ,2.12]	46.28

Appendix

Table 23: Transfer conditioned on successful name edits. Specific is conditional SMILES transfer minus conditional leakage. Brackets give 95% bootstrap confidence intervals.

Model	Entities
ChemDFM-v2.0-14B	389
Qwen2.5-14B-Instruct	390
Llama-3.1-8B-Instruct	389
ChemDFM-v1.5-8B	47
Qwen2.5-7B-Instruct	351
Meta-Llama-3-8B-Instruct	378

Appendix

Table 24: Model-specific Binding Self-Distillation sets. The reported sets are the fixed entity sets selected by the original BSD entity-selection pipeline using therapeutic-indication retrieval through the trivial name.

Hyperparameter	Value
Editing mode	One adapter per entity
Target layer	24
LoRA layers	20–24
LoRA modules	gate/up/down projection
Rank $r$	8
Scaling $\alpha$	16

Appendix

Table 25: Binding Self-Distillation configuration for the six verified local runs.

Model	Before	After	Before	After	Before	After	Before	After	Facts/entity
	Indication		Target		MoA		ATC
ChemDFM-v2.0-14B	2.1	51.4	0.0	27.6	0.0	5.0	3.3	11.4	0.05 $\rightarrow$ 0.87
Qwen2.5-14B-Instruct	4.1	13.6	1.4	43.6	0.6	20.8	0.0	10.3	0.06 $\rightarrow$ 0.73
Llama-3.1-8B-Instruct	0.3	38.8	0.7	19.9	1.2	8.4	1.1	7.9	0.03 $\rightarrow$ 0.68
ChemDFM-v1.5-8B	25.5	42.6	5.1	23.1	17.1	12.2	6.8	13.6	0.51 $\rightarrow$ 0.85
Qwen2.5-7B-Instruct	1.1	36.2	0.4	32.3	0.0	19.9	2.7	9.1	0.04 $\rightarrow$ 0.85

Appendix

Table 26: Relation-wise Binding Self-Distillation results on the fixed entity sets selected by the original BSD entity-selection pipeline. Values are retrieval accuracy (%) through SMILES before and after the per-entity binding intervention.

Model	$n$	Facts/entity	Bridge	Recovery
ChemDFM-v2.0-14B	389	0.05 $\rightarrow$ 0.87	1.01	86%
Qwen2-72B-Instruct	435	0.01 $\rightarrow$ 0.85	1.34	63%

Appendix

Table 27: Binding Self-Distillation versus explicit lexical access on the fixed entity sets selected by the original BSD entity-selection pipeline. Bridge gives factual access when the correct name is explicitly supplied. Recovery measures the fraction of the SMILES-to-bridge gain recovered without the name at inference time.

Model	Entries	Symbol ID	Name ID	Loc. via Acc.	ID Bridge	Loc. Bridge	Decoy ID
ChemDFM-v2.0-14B	16,388	99.5	66.7	23.9	99.9	100.1	99.4
ChemDFM-v1.5-8B	1,000	5.5	18.8	0.0	20.0	5.6	1.0
Qwen2.5-14B-Instruct	1,000	98.0	60.8	22.0	100.7	101.0	98.6
Qwen2.5-7B-Instruct	1,000	86.4	51.8	21.9	98.3	101.4	85.3
Llama-3.1-8B-Instruct	1,000	90.8	60.3	9.3	98.9	97.6	90.9

Appendix

Table 28: Identifier-dependent access also appears for proteins. Identification columns report recovery of the official gene symbol. “Loc. via Acc.” is subcellular-location accuracy through the accession. ID and Loc. Bridge report the fraction of the respective accession–gene-symbol gap recovered when the correct gene symbol is supplied alongside the accession. Decoy ID measures whether identification follows an incorrect gene symbol in a conflicting-identifier prompt. All values except Entries are percentages.

Model	Items	Peak	Rel. depth	Recovery	Control	Rank layer	Rank top-1
ChemDFM-v2.0-14B	689	47	0.98	89.0	-104.2	40	$11.6\rightarrow 50.7$
ChemDFM-v1.5-8B	607	25	0.78	109.9	-113.7	28	$12.7\rightarrow 62.6$
Qwen2.5-14B-Instruct	677	39	0.81	94.6	-125.7	40	$11.8\rightarrow 55.5$
Qwen2.5-7B-Instruct	674	24	0.86	105.3	-137.3	24	$11.9\rightarrow 37.8$
Llama-3.1-8B-Instruct	680	31	0.97	88.0	-130.1	24	$11.8\rightarrow 38.1$

Appendix

Table 29: Activation patching in the protein domain. Peak is the layer maximizing mean normalized recovery after the complete coarse-to-fine sweep; relative depth is peak layer divided by model depth. Rank is evaluated at the coarse-sweep peak (see text). Rank top-1 reports whether the correct location has the highest teacher-forced score among the nine observed location classes.

Chemical Language Models

See all 10 papers

Rethinking Molecular Text Representations for LLMs: An Empirical Study
Jun 2, 2026Arun Raja, Garrett M. Morris, Kian Ming A. ChaiLLM Evaluation Molecular Generation
University of Oxford · DSO National Laboratories
Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models
Sep 4, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation LLM Reliability
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3
MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry
Jun 4, 2026Joey Chan, Wonbin Kweon, Ashley Shin +4Retrieval-Augmented Generation LLM Inference
University of Illinois Urbana-Champaign · University of California, San Diego

Jun 2, 2026J/K move · Enter open · S save

Rethinking Molecular Text Representations for LLMs: An Empirical Study

Arun Raja, Garrett M. Morris, Kian Ming A. Chai

University of Oxford · DSO National Laboratories

LLM Evaluation Molecular Generation Molecular Representation Learning

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark, MolRepBench, to evaluate LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI and canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a- judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes, and attention shows that representations are encoded differently inside the model; for example, structured representations require more attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.

Abstract

Chemical Language Models

Explore similar work

Rethinking Molecular Text Representations for LLMs: An Empirical Study

Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models

MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry