Do LLMs Know When They Are Wrong About Molecules? In-Context Property Prediction with Self-Reported Uncertainty

Matthias Busch; Marius Tacke; Tamadur Albaraghtheh; Sviatlana V. Lamaka; Mikhail L. Zheludkevich; Roland C. Aydin; Christian Feiler; Christian J. Cyron

cs.LGOct 30, 2026

Do LLMs Know When They Are Wrong About Molecules? In-Context Property Prediction with Self-Reported Uncertainty

Authors: Matthias Busch, Marius Tacke, Tamadur Albaraghtheh, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Roland C. Aydin, Christian Feiler, Christian J. Cyron

Organizations: Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Metallic Biomaterials, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany ·

Abstract

Large language models (LLMs) are used in many domains, but molecular property prediction with them is still little explored. In this work we evaluate the capability of 7 frontier LLMs to predict molecular properties together with a self-reported uncertainty from in-context examples, and compare them with classical baselines trained on the same molecules. We use 4 datasets and screen each for contamination of the training data of the LLMs: three are clean, and the fourth, ESOL, is memorised by most of the models and serves as a comparison. On the clean datasets every LLM improves with the context, at 30 examples the most accurate one beats every baseline. The requested reasoning effort changes accuracy little. The self-reported uncertainty ranks the prediction errors weakly, with Spearman rank correlations of up to 0.46, about the level reached by the uncertainties of the baselines. Combining uncertainty and prediction, on lipophilicity the most certain half of the best LLM is the most accurate at small context sizes; on BACE and antiviral, and on lipophilicity at large context sizes, the best LLMs are on par with the best baseline. On the contaminated ESOL dataset the LLMs show no improvement with the context size, as expected for a memorised dataset, and their stated uncertainty is smaller on the molecules whose value they recall.

Figures & tables

(a) Datasets
dataset	property	test	pool	largest $N$	in corpus	flagged
Lipophilicity [ 59 ]	logD at pH 7.4	150	4,042	1,000	0.0 %	0 of 7
BACE [ 53 ]	BACE-1 pIC50	150	1,355	1,000	0.0 %	0 of 7
Antiviral [ 3 ]	SARS-CoV-2 Mpro pIC50	150	844	844	0.0 %	0 of 7
ESOL [ 10 ]	logS (log 10 mol/L)	150	976	976	32.4 %	6 of 7

Table 1: Datasets and model panel. (a) The four datasets: test-set size, deduplicated training pool, largest context size

N

, share of the test molecules found in any of the seven searched pre-training corpora as a SMILES string (in corpus; maximum over the corpora, Fig. 5 b), and the number of panel models the zero-shot screen flags (flagged). (b) The seven language models: whether the endpoint accepts the reasoning effort none (every endpoint accepts minimal to high), the median emitted reasoning tokens per call at the effort low (range over the 24 main-run cells) and how they change along the reasoning ladder (Fig. 2 ), and on the ESOL test set the share of the molecules whose published value the model reproduced to three significant figures (hit 3 ; answers given in mol/L converted to logS, Section 2.1 ) with the outcome of the zero-shot screen and its regime. Endpoints, service tiers and prices: Table A1 . Endpoints as of 2026-10-08, zero-shot outcomes as of 2026-10-09 (GPT-6 Astra 2026-10-13).

Figure 1: Test error against the number of in-context molecules. Root-mean-square error on the 150 test molecules of each dataset for the seven language models at the effort low and four of the eight classical baselines trained on the same molecules. Lines follow context draw 1; at

N\leq 120

bars span the range over the three context draws, baselines averaged over their seeds within a draw. ESOL is the dataset the zero-shot screen flags; its axis is cut, and baselines above the cut are marked by arrows.

Figure 2: Accuracy along the reasoning ladder. Root-mean-square error against the median number of reasoning tokens a model emitted per call at each requested effort level, for the six ladder models at 30 in-context molecules (top) and at the largest context size (bottom); models that emit no reasoning tokens are drawn left of the axis break. At the effort high, Gemini 3.8 Flash reached the completion cap in up to 35 % of its calls on the clean datasets, so its points there rest on the remaining molecules.

Figure 3: Ranking of the errors by the uncertainty, the measure for measurement selection. Spearman rank correlation between the stated standard deviation and the absolute error on the 150 test molecules against the number of in-context molecules, for the seven language models at the effort low and four classical baselines with their own

\sigma

;

\sigma

rounded to 10 decimals before ranking. Lines and bars as in Fig. 1 .

Figure 4: Error of the most certain half, the measure for selective prediction. Root-mean-square error of the 75 test molecules with the smallest stated standard deviation against the number of in-context molecules, with molecules tied at the cut selected at random (square root of the expected mean squared error, Eq. 1 ), for the seven language models at the effort low and four classical baselines. Lines and bars as in Fig. 1 ; the ESOL axis is cut, baselines above it are marked by arrows.

Figure 5: Contamination screen of the four test sets. (a) Spearman rank correlation between the labels and the zero-shot predictions of each language model, asked with the dataset named for the published value of each test molecule (ESOL: positive answers to molecules with a negative label read as mol/L and converted to logS, Section 2.1 ). (b) Share of the test molecules found in the seven searched pre-training corpora as a SMILES string (verbatim or RDKit canonical) and as InChIKey; bars give the maximum over the corpora, dots the single corpora (ESOL: the 136 test molecules with a searchable string). (c) Retention

R_{12}

, the share of the molecules whose zero-shot value matches the label to one significant figure that also match it to two, against the floor expected from the labels alone (black line). The screen tests

R_{12}

and

R_{23}

, the retention from two to three matching figures, against their floors with Benjamini–Hochberg correction; hit 3 , the share of the molecules whose published value is reproduced to three significant figures, is given for ESOL in Table 1 b. Filled markers: cell flagged by the screen; open: clean. Scored are the labels with at least three significant figures, 81 (Lipo., lipophilicity), 143 (BACE), 136 (Antivir., antiviral) and 113 (ESOL);

R_{23}

, hit 3 , the floors and the

q

-values of every cell are in the repository.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

				USD per million tokens
model	vendor	provider endpoint	tier	input	cached	output
Haiku 5.5	Anthropic	anthropic	standard	0.1	0.01	0.5
Sonnet 5.5	Anthropic	anthropic	standard	2	0.1	10
GPT-6.1 Sol	OpenAI	openai/flex	flex	1	0.05	5
GPT-6 Luna	OpenAI	openai/flex	flex	0.05	0.005	0.25
GPT-6 Astra	OpenAI	openai/flex	flex	5	0.5	25

Appendix

Table A1: Endpoints and prices. The provider endpoint on OpenRouter to which the requests of each language model are pinned, with fallback to other hosts disabled, its service tier (flex: the provider’s flex tier, billed at half the standard price), and the price per million input, cached-input and output tokens at that endpoint. The reasoning efforts an endpoint accepts and the emitted reasoning tokens are in Table 1 b. Prices fetched from OpenRouter on 2026-10-08 ( src/registry/models_v2.json , which also gives the OpenRouter ids).

dataset	molecules	corpus filter	test	dedup.	pool	draw overlap (exp.)
Lipophilicity	4,200	20 of 170	150	8	4,042	5 (3.6)
BACE	1,513	0 of 150	150	8	1,355	10 (10.6)
Antiviral	1,006	0 of 150	150	12	844	20 (17.1)
ESOL	1,128	0 of 150	150	2	976	19 (14.8)

Appendix

Table A2: Datasets and splits. Number of molecules; test candidates rejected by the corpus filter (present in a searched corpus) among the positions checked; test-set size; pool molecules removed because their InChIKey connectivity block equals that of a test molecule (dedup.); training pool after removal; and the largest number of molecules shared by the first 120 molecules of two context draws, with the expectation for independent permutations in parentheses. Largest context size, corpus presence and screen outcome: Table 1 a and Fig. 5 ( data/splits/_v2.json , results/meta/context_draw_overlap.csv ).

method	features	settings	$\sigma$	seeds
TabPFN v3.5	RDKit 2D descriptors (159–201 after the filter)	v3.5, tabpfn 9.1.0, default estimators, CPU	$(q_{84}-q_{16})/2$ of the predictive distribution	0,1,2 / 0
GP (Tanimoto)	Morgan r = 2, 2,048 bits	$k=a\,T+s^{2}\delta$ , ML-II from 14 starts	posterior predictive SD incl. noise	0,1,2 / 0,1,2
RF (descriptors)	RDKit 2D descriptors (as TabPFN)	200 trees, scikit-learn defaults	SD over the trees	0,1,2 / 0
RF (Morgan)	Morgan r = 2, 2,048 bits	200 trees, scikit-learn defaults	SD over the trees	0,1,2 / 0
Chemprop ensemble	molecular graph (chemprop 2 featurizer)	5 members, chemprop 2.0.0 defaults, 50 epochs	SD over the members	0 / 0
MLP ensemble	Morgan r = 2, 2,048 bits	5 members, 2 $\times$ 256 ReLU, dropout 0.1, Adam 0.001	SD over the members	0 / 0

Appendix

Table A3: Classical baselines. Input features, fixed settings, the definition of the uncertainty

\sigma

(SD: standard deviation) and the seeds run in context draw 1 / in draws 2 and 3 (values are averaged over the seeds of a draw; the paired bootstrap uses seed 0). Every method is trained on exactly the in-context molecules of a cell and predicts the same 150 test molecules ( results/baselines/*.meta.json , summary.csv ).

Figure A1: Error and error in units of the stated

\sigma

on recalled and never-recalled ESOL molecules. Root-mean-square error and root mean square of

e/\sigma

for the 63 recalled (filled) and the 50 never-recalled (open) test molecules, at 30 (faint) and 976 (solid) in-context molecules; ticks mark all 150 test molecules, and the vertical line marks a root mean square of

e/\sigma

of 1. TabPFN, which has no access to published values, is shown for reference, averaged over its seeds.

		RMSE				RMS $e/\sigma$
model	hit 3 %	all	rec.	never	ratio	all	rec.	never
Haiku 5.5	10.6	0.44	0.43	0.54	1.2	0.95	0.87	1.11
Sonnet 5.5	37.2	0.29	0.20	0.40	2.0	0.87	0.42	1.27
GPT-6.1 Sol	19.5	0.40	0.29	0.58	2.0	0.76	0.52	1.06
GPT-6 Luna	8.8	0.56	0.56	0.71	1.3	1.02	1.04	1.22
GPT-6 Astra	18.6	0.37	0.28	0.53	1.9	0.72	0.52	1.00

Appendix

Table A4: ESOL by recall at 976 in-context molecules. Test molecules whose value at least one panel model reproduced to three significant figures in the zero-shot screen (rec., 63 molecules) against those no model reproduced (never, 50); the remaining 37 labels have fewer than three significant figures and are in neither subset. Root-mean-square error with the ratio never / recalled, and root mean square of the error in units of the stated

\sigma

(\frac{1}{n}\sum_{i}e_{i}^{2}/\sigma_{i}^{2})^{1/2}

with

\sigma

floored at 0.01, on all 150 molecules and on the two subsets; language models at effort low, TabPFN v3.5 seed 0, context draw 1 ( results/icl/calibration_cells.csv , columns RMSE and s ).

Figure A2: Similarity of the test molecules to their context against the error and the stated uncertainty. Spearman rank correlation between the largest Tanimoto similarity of a test molecule (Morgan fingerprint, radius 2, 2,048 bits) to any of the

N

in-context molecules and its absolute error (top) and its stated standard deviation (bottom), against the number of in-context molecules, for the seven language models at the effort low and four classical baselines; a negative value means that molecules with a close context neighbour are predicted more accurately (top) or with a smaller

\sigma

(bottom). Lines and bars as in Fig. 1 .

model	Lipophilicity	BACE	Antiviral
$\Delta$ RMSE
Haiku 5.5	$-$ 0.42 [ $-$ 0.53, $-$ 0.29] †	$-$ 0.01 [ $-$ 0.15, $+$ 0.15]	$-$ 0.07 [ $-$ 0.16, $+$ 0.01]
Sonnet 5.5	$-$ 0.50 [ $-$ 0.60, $-$ 0.41] †	$-$ 0.23 [ $-$ 0.34, $-$ 0.13] †	$-$ 0.10 [ $-$ 0.18, $-$ 0.03] †
GPT-6.1 Sol	$-$ 0.36 [ $-$ 0.47, $-$ 0.25] †	$-$ 0.11 [ $-$ 0.19, $-$ 0.03] †	$-$ 0.07 [ $-$ 0.16, $+$ 0.01]
GPT-6 Luna	$-$ 0.11 [ $-$ 0.23, $+$ 0.03]	$+$ 0.11 [ $-$ 0.00, $+$ 0.25]	$-$ 0.02 [ $-$ 0.13, $+$ 0.06]
GPT-6 Astra	$-$ 0.37 [ $-$ 0.47, $-$ 0.26] †	$-$ 0.14 [ $-$ 0.21, $-$ 0.07] †	$-$ 0.07 [ $-$ 0.16, $+$ 0.01]

Appendix

Table A5: Paired bootstrap at 30 in-context molecules, language model minus TabPFN v3.5. Difference with the percentile 95 % interval from 1,000 resamples of the 150 test molecules, computed in each of the three context draws on the same resampled molecules and averaged (TabPFN seed 0).

\dagger

\ddagger

: the interval excludes 0 in favour of the language model / of TabPFN. The intervals at

N

= 60, 120 and the largest

N

, and against the GP, are in the repository ( results/icl/bootstrap_cells.csv ).

Figure A3: Error of the most certain fifth and of the most certain four fifths. Root-mean-square error of the 20 % (top) and the 80 % (bottom) of the test molecules with the smallest stated standard deviation against the number of in-context molecules, with the tie rule of Eq. 1 , for the seven language models at the effort low and four classical baselines. Lines and bars as in Fig. 1 ; the ESOL axes are cut, baselines above them are marked by arrows.

Figure A4: Error of the most certain molecules as a function of how many are kept. For each dataset at

N=30

(context draw 1), the root-mean-square error of the fraction

q

of test molecules with the smallest stated

\sigma

(tie rule of Eq. 1 ), for Sonnet 5.5, Haiku 5.5, GPT-6 Astra, the pointwise median of the seven language models (thick grey), TabPFN and the GP (mean over seeds). Dotted references for Sonnet 5.5 and TabPFN in their colours: the oracle that keeps the molecules with the smallest absolute error, and random selection. The ESOL axis is cut at 1.12; the GP lies at 1.74 to 1.86 for every

q

(arrows).

Figure A5: Accuracy and selection at 30 in-context molecules. Root-mean-square error of all test molecules against the relative error reduction obtained by keeping the half with the smallest stated standard deviation, for the language models (filled) and the classical baselines (open; further baselines in light grey); markers show context draw 1, bars the range over the three context draws. Dashed lines connect points of equal error on the selected half.

Figure A6: Learning curves of the three context draws. Test RMSE at

N

= 30, 60 and 120 for the seven language models and TabPFN (averaged over its seeds within a draw), one line per context draw: draw 1, the nested chain of all other figures, bold, and draws 2 and 3 thin; models slightly offset in

N

Figure A7: Cost and error of each model at the smallest and the largest context. Top: OpenRouter cost in USD of one cell (150 test molecules,

R=3

calls, effort low, context draw 1; log scale) against the test RMSE, open marker

N=30

, filled marker largest

N

. Bottom: four classical baselines, CPU wall time per cell in seconds (mean over seeds; log scale) against the RMSE; seconds and dollars are not on one axis.

Molecular Property Prediction

Molecular property prediction estimates chemical, physical, and biological properties from molecular structure and context.

See all 207 papers

In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts
Mar 26, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation Knowledge Conflicts in Language Models
Institute for Continuum and Material Mechanics, Technical University of Hamburg, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +2
Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models
Sep 4, 2026Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4LLM Evaluation LLM Reliability
Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3
MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry
Jun 4, 2026Joey Chan, Wonbin Kweon, Ashley Shin +4Retrieval-Augmented Generation LLM Inference
University of Illinois Urbana-Champaign · University of California, San Diego

Mar 26, 2026J/K move · Enter open · S save

In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts

Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich+3

Institute for Continuum and Material Mechanics, Technical University of Hamburg, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany+2

LLM Evaluation Knowledge Conflicts in Language Models In-Context Learning Molecular Property Prediction

Open paper AskPDF arXiv

The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or instead rely on verbatim retrieval of memorized target values. Furthermore, we analyze the interplay between pre-trained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet datasets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information, complemented by 0-, 60-, and 1000-shot in-context sample sizes as an additional control for information access. To validate the memorization analysis and the blinding experiments, we add a positive and a negative control for the memorization experiments and structural reference baselines for the multi-shot experiments as well as bootstrap confidence intervals for all results. We find no evidence of verbatim retrieval on the legacy benchmarks and show that blinding exposes conflicts between pre-trained knowledge and in-context information. This work provides a principled framework for evaluating molecular property prediction under controlled information access.

Abstract

Molecular Property Prediction

Explore similar work

In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts

Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models

MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry