cs.LGOct 30, 2026

Do LLMs Know When They Are Wrong About Molecules? In-Context Property Prediction with Self-Reported Uncertainty

Authors: Matthias Busch, Marius Tacke, Tamadur Albaraghtheh, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Roland C. Aydin, Christian Feiler, Christian J. Cyron

Organizations: Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Metallic Biomaterials, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) are used in many domains, but molecular property prediction with them is still little explored. In this work we evaluate the capability of 7 frontier LLMs to predict molecular properties together with a self-reported uncertainty from in-context examples, and compare them with classical baselines trained on the same molecules. We use 4 datasets and screen each for contamination of the training data of the LLMs: three are clean, and the fourth, ESOL, is memorised by most of the models and serves as a comparison. On the clean datasets every LLM improves with the context, at 30 examples the most accurate one beats every baseline. The requested reasoning effort changes accuracy little. The self-reported uncertainty ranks the prediction errors weakly, with Spearman rank correlations of up to 0.46, about the level reached by the uncertainties of the baselines. Combining uncertainty and prediction, on lipophilicity the most certain half of the best LLM is the most accurate at small context sizes; on BACE and antiviral, and on lipophilicity at large context sizes, the best LLMs are on par with the best baseline. On the contaminated ESOL dataset the LLMs show no improvement with the context size, as expected for a memorised dataset, and their stated uncertainty is smaller on the molecules whose value they recall.

Figures & tables

(a) Datasets
datasetpropertytestpoollargest NNin corpusflagged
Lipophilicity [ 59 ]logD at pH 7.41504,0421,0000.0 %0 of 7
BACE [ 53 ]BACE-1 pIC501501,3551,0000.0 %0 of 7
Antiviral [ 3 ]SARS-CoV-2 Mpro pIC501508448440.0 %0 of 7
ESOL [ 10 ]logS (log 10 mol/L)15097697632.4 %6 of 7
Table 1: Datasets and model panel. (a) The four datasets: test-set size, deduplicated training pool, largest context size NN , share of the test molecules found in any of the seven searched pre-training corpora as a SMILES string (in corpus; maximum over the corpora, Fig. 5 b), and the number of panel models the zero-shot screen flags (flagged). (b) The seven language models: whether the endpoint accepts the reasoning effort none (every endpoint accepts minimal to high), the median emitted reasoning tokens per call at the effort low (range over the 24 main-run cells) and how they change along the reasoning ladder (Fig. 2 ), and on the ESOL test set the share of the molecules whose published value the model reproduced to three significant figures (hit 3 ; answers given in mol/L converted to logS, Section 2.1 ) with the outcome of the zero-shot screen and its regime. Endpoints, service tiers and prices: Table A1 . Endpoints as of 2026-10-08, zero-shot outcomes as of 2026-10-09 (GPT-6 Astra 2026-10-13).
Figure 1: Test error against the number of in-context molecules. Root-mean-square error on the 150 test molecules of each dataset for the seven language models at the effort low and four of the eight classical baselines trained on the same molecules. Lines follow context draw 1; at N≤120N\leq 120 bars span the range over the three context draws, baselines averaged over their seeds within a draw. ESOL is the dataset the zero-shot screen flags; its axis is cut, and baselines above the cut are marked by arrows.
Figure 2: Accuracy along the reasoning ladder. Root-mean-square error against the median number of reasoning tokens a model emitted per call at each requested effort level, for the six ladder models at 30 in-context molecules (top) and at the largest context size (bottom); models that emit no reasoning tokens are drawn left of the axis break. At the effort high, Gemini 3.8 Flash reached the completion cap in up to 35 % of its calls on the clean datasets, so its points there rest on the remaining molecules.
Figure 3: Ranking of the errors by the uncertainty, the measure for measurement selection. Spearman rank correlation between the stated standard deviation and the absolute error on the 150 test molecules against the number of in-context molecules, for the seven language models at the effort low and four classical baselines with their own σ\sigma ; σ\sigma rounded to 10 decimals before ranking. Lines and bars as in Fig. 1 .
Figure 4: Error of the most certain half, the measure for selective prediction. Root-mean-square error of the 75 test molecules with the smallest stated standard deviation against the number of in-context molecules, with molecules tied at the cut selected at random (square root of the expected mean squared error, Eq. 1 ), for the seven language models at the effort low and four classical baselines. Lines and bars as in Fig. 1 ; the ESOL axis is cut, baselines above it are marked by arrows.
Figure 5: Contamination screen of the four test sets. (a) Spearman rank correlation between the labels and the zero-shot predictions of each language model, asked with the dataset named for the published value of each test molecule (ESOL: positive answers to molecules with a negative label read as mol/L and converted to logS, Section 2.1 ). (b) Share of the test molecules found in the seven searched pre-training corpora as a SMILES string (verbatim or RDKit canonical) and as InChIKey; bars give the maximum over the corpora, dots the single corpora (ESOL: the 136 test molecules with a searchable string). (c) Retention R12R_{12} , the share of the molecules whose zero-shot value matches the label to one significant figure that also match it to two, against the floor expected from the labels alone (black line). The screen tests R12R_{12} and R23R_{23} , the retention from two to three matching figures, against their floors with Benjamini–Hochberg correction; hit 3 , the share of the molecules whose published value is reproduced to three significant figures, is given for ESOL in Table 1 b. Filled markers: cell flagged by the screen; open: clean. Scored are the labels with at least three significant figures, 81 (Lipo., lipophilicity), 143 (BACE), 136 (Antivir., antiviral) and 113 (ESOL); R23R_{23} , hit 3 , the floors and the qq -values of every cell are in the repository.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

    USD per million tokens
modelvendorprovider endpointtierinputcachedoutput
Haiku 5.5Anthropicanthropicstandard0.10.010.5
Sonnet 5.5Anthropicanthropicstandard20.110
GPT-6.1 SolOpenAIopenai/flexflex10.055
GPT-6 LunaOpenAIopenai/flexflex0.050.0050.25
GPT-6 AstraOpenAIopenai/flexflex50.525
Appendix
Table A1: Endpoints and prices. The provider endpoint on OpenRouter to which the requests of each language model are pinned, with fallback to other hosts disabled, its service tier (flex: the provider’s flex tier, billed at half the standard price), and the price per million input, cached-input and output tokens at that endpoint. The reasoning efforts an endpoint accepts and the emitted reasoning tokens are in Table 1 b. Prices fetched from OpenRouter on 2026-10-08 ( src/registry/models_v2.json , which also gives the OpenRouter ids).
datasetmoleculescorpus filtertestdedup.pooldraw overlap (exp.)
Lipophilicity4,20020 of 17015084,0425 (3.6)
BACE1,5130 of 15015081,35510 (10.6)
Antiviral1,0060 of 1501501284420 (17.1)
ESOL1,1280 of 150150297619 (14.8)
Appendix
Table A2: Datasets and splits. Number of molecules; test candidates rejected by the corpus filter (present in a searched corpus) among the positions checked; test-set size; pool molecules removed because their InChIKey connectivity block equals that of a test molecule (dedup.); training pool after removal; and the largest number of molecules shared by the first 120 molecules of two context draws, with the expectation for independent permutations in parentheses. Largest context size, corpus presence and screen outcome: Table 1 a and Fig. 5 ( data/splits/_v2.json , results/meta/context_draw_overlap.csv ).
methodfeaturessettingsσ\sigmaseeds
TabPFN v3.5RDKit 2D descriptors (159–201 after the filter)v3.5, tabpfn 9.1.0, default estimators, CPU(q84−q16)/2(q_{84}-q_{16})/2 of the predictive distribution0,1,2 / 0
GP (Tanimoto)Morgan r = 2, 2,048 bitsk=a T+s2δk=a\,T+s^{2}\delta , ML-II from 14 startsposterior predictive SD incl. noise0,1,2 / 0,1,2
RF (descriptors)RDKit 2D descriptors (as TabPFN)200 trees, scikit-learn defaultsSD over the trees0,1,2 / 0
RF (Morgan)Morgan r = 2, 2,048 bits200 trees, scikit-learn defaultsSD over the trees0,1,2 / 0
Chemprop ensemblemolecular graph (chemprop 2 featurizer)5 members, chemprop 2.0.0 defaults, 50 epochsSD over the members0 / 0
MLP ensembleMorgan r = 2, 2,048 bits5 members, 2 ×\times 256 ReLU, dropout 0.1, Adam 0.001SD over the members0 / 0
Appendix
Table A3: Classical baselines. Input features, fixed settings, the definition of the uncertainty σ\sigma (SD: standard deviation) and the seeds run in context draw 1 / in draws 2 and 3 (values are averaged over the seeds of a draw; the paired bootstrap uses seed 0). Every method is trained on exactly the in-context molecules of a cell and predicts the same 150 test molecules ( results/baselines/*.meta.json , summary.csv ).
Figure A1: Error and error in units of the stated σ\sigma on recalled and never-recalled ESOL molecules. Root-mean-square error and root mean square of e/σe/\sigma for the 63 recalled (filled) and the 50 never-recalled (open) test molecules, at 30 (faint) and 976 (solid) in-context molecules; ticks mark all 150 test molecules, and the vertical line marks a root mean square of e/σe/\sigma of 1. TabPFN, which has no access to published values, is shown for reference, averaged over its seeds.
  RMSERMS e/σe/\sigma
modelhit 3 %allrec.neverratioallrec.never
Haiku 5.510.60.440.430.541.20.950.871.11
Sonnet 5.537.20.290.200.402.00.870.421.27
GPT-6.1 Sol19.50.400.290.582.00.760.521.06
GPT-6 Luna8.80.560.560.711.31.021.041.22
GPT-6 Astra18.60.370.280.531.90.720.521.00
Appendix
Table A4: ESOL by recall at 976 in-context molecules. Test molecules whose value at least one panel model reproduced to three significant figures in the zero-shot screen (rec., 63 molecules) against those no model reproduced (never, 50); the remaining 37 labels have fewer than three significant figures and are in neither subset. Root-mean-square error with the ratio never / recalled, and root mean square of the error in units of the stated σ\sigma , (1n∑iei2/σi2)1/2(\frac{1}{n}\sum_{i}e_{i}^{2}/\sigma_{i}^{2})^{1/2} with σ\sigma floored at 0.01, on all 150 molecules and on the two subsets; language models at effort low, TabPFN v3.5 seed 0, context draw 1 ( results/icl/calibration_cells.csv , columns RMSE and s ).
Figure A2: Similarity of the test molecules to their context against the error and the stated uncertainty. Spearman rank correlation between the largest Tanimoto similarity of a test molecule (Morgan fingerprint, radius 2, 2,048 bits) to any of the NN in-context molecules and its absolute error (top) and its stated standard deviation (bottom), against the number of in-context molecules, for the seven language models at the effort low and four classical baselines; a negative value means that molecules with a close context neighbour are predicted more accurately (top) or with a smaller σ\sigma (bottom). Lines and bars as in Fig. 1 .
modelLipophilicityBACEAntiviral
Δ\Delta RMSE
Haiku 5.5−- 0.42 [ −- 0.53, −- 0.29] †−- 0.01 [ −- 0.15, ++ 0.15]−- 0.07 [ −- 0.16, ++ 0.01]
Sonnet 5.5−- 0.50 [ −- 0.60, −- 0.41] †−- 0.23 [ −- 0.34, −- 0.13] †−- 0.10 [ −- 0.18, −- 0.03] †
GPT-6.1 Sol−- 0.36 [ −- 0.47, −- 0.25] †−- 0.11 [ −- 0.19, −- 0.03] †−- 0.07 [ −- 0.16, ++ 0.01]
GPT-6 Luna−- 0.11 [ −- 0.23, ++ 0.03]++ 0.11 [ −- 0.00, ++ 0.25]−- 0.02 [ −- 0.13, ++ 0.06]
GPT-6 Astra−- 0.37 [ −- 0.47, −- 0.26] †−- 0.14 [ −- 0.21, −- 0.07] †−- 0.07 [ −- 0.16, ++ 0.01]
Appendix
Table A5: Paired bootstrap at 30 in-context molecules, language model minus TabPFN v3.5. Difference with the percentile 95 % interval from 1,000 resamples of the 150 test molecules, computed in each of the three context draws on the same resampled molecules and averaged (TabPFN seed 0). †\dagger / ‡\ddagger : the interval excludes 0 in favour of the language model / of TabPFN. The intervals at NN = 60, 120 and the largest NN , and against the GP, are in the repository ( results/icl/bootstrap_cells.csv ).
Figure A3: Error of the most certain fifth and of the most certain four fifths. Root-mean-square error of the 20 % (top) and the 80 % (bottom) of the test molecules with the smallest stated standard deviation against the number of in-context molecules, with the tie rule of Eq. 1 , for the seven language models at the effort low and four classical baselines. Lines and bars as in Fig. 1 ; the ESOL axes are cut, baselines above them are marked by arrows.
Figure A4: Error of the most certain molecules as a function of how many are kept. For each dataset at N=30N=30 (context draw 1), the root-mean-square error of the fraction qq of test molecules with the smallest stated σ\sigma (tie rule of Eq. 1 ), for Sonnet 5.5, Haiku 5.5, GPT-6 Astra, the pointwise median of the seven language models (thick grey), TabPFN and the GP (mean over seeds). Dotted references for Sonnet 5.5 and TabPFN in their colours: the oracle that keeps the molecules with the smallest absolute error, and random selection. The ESOL axis is cut at 1.12; the GP lies at 1.74 to 1.86 for every qq (arrows).
Figure A5: Accuracy and selection at 30 in-context molecules. Root-mean-square error of all test molecules against the relative error reduction obtained by keeping the half with the smallest stated standard deviation, for the language models (filled) and the classical baselines (open; further baselines in light grey); markers show context draw 1, bars the range over the three context draws. Dashed lines connect points of equal error on the selected half.
Figure A6: Learning curves of the three context draws. Test RMSE at NN = 30, 60 and 120 for the seven language models and TabPFN (averaged over its seeds within a draw), one line per context draw: draw 1, the nested chain of all other figures, bold, and draws 2 and 3 thin; models slightly offset in NN .
Figure A7: Cost and error of each model at the smallest and the largest context. Top: OpenRouter cost in USD of one cell (150 test molecules, R=3R=3 calls, effort low, context draw 1; log scale) against the test RMSE, open marker N=30N=30 , filled marker largest NN . Bottom: four classical baselines, CPU wall time per cell in seconds (mean over seeds; log scale) against the RMSE; seconds and dollars are not on one axis.
More in this topic

Molecular Property Prediction

Molecular property prediction estimates chemical, physical, and biological properties from molecular structure and context.

See all 207 papers

Explore similar work

CardsList
Mar 26, 2026

In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts

The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or instead rely on verbatim retrieval of memorized target values. Furthermore, we analyze the interplay between pre-trained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet datasets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information, complemented by 0-, 60-, and 1000-shot in-context sample sizes as an additional control for information access. To validate the memorization analysis and the blinding experiments, we add a positive and a negative control for the memorization experiments and structural reference baselines for the multi-shot experiments as well as bootstrap confidence intervals for all results. We find no evidence of verbatim retrieval on the legacy benchmarks and show that blinding exposes conflicts between pre-trained knowledge and in-context information. This work provides a principled framework for evaluating molecular property prediction under controlled information access.

Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4

Institute for Continuum and Material Mechanics, Technical University of Hamburg, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +2

LLM EvaluationKnowledge Conflicts in Language ModelsIn-Context LearningMolecular Property Prediction
AskPDF
Sep 4, 2026

Molecular Déjà Vu: Digit-Level Retrieval of Molecular Properties in Frontier Language Models

Large language models (LLMs) are increasingly employed to predict molecular properties. However, prediction error alone cannot distinguish prediction from retrieval of published values. We audit 22 frontier models on 12 molecular regression datasets in a zero-shot setting, assessed against a molecule-blind reference derived from each dataset's labels. Significant retrieval is concentrated on 5 datasets, with isolated flagged LLMs elsewhere. Increasing the reasoning setting raises the number of flagged model--dataset combinations from 47 to 89 of 264. An in-context blinding experiment reduces retrieval but leaves a quarter of the combinations flagged. Blinding changes model rankings and increases errors. Because blinding also removes chemically interpretable structure, the error increase can only be partially attributed to reduced retrieval.

Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4

Institute for Artificial Intelligence and Simulation in Mechanics, Hamburg University of Technology, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, Germany +3

LLM EvaluationLLM ReliabilityData Contamination in Language ModelsMolecular Property Prediction
AskPDF
Jun 4, 2026

MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry

Large language models (LLMs) have shown promise for molecular property prediction, but their ability to reason over chemical structures remains limited, as molecular representations such as SMILES differ substantially from the natural language on which LLMs are primarily trained. To bridge this semantic and chemical knowledge gap, we propose MolE-RAG, a training-free, molecule-centric retrieval-augmented generation framework for LLM-based molecular property prediction. MolE-RAG augments each prediction with three complementary sources of inference-time context: retrieved chemistry literature, molecule-specific information including compound synonyms, identifiers, functional group annotations, and physicochemical descriptors, and structurally similar molecules retrieved from the training set. We evaluate MolE-RAG across nine molecular property prediction tasks using proprietary, chemistry-specialized, and open-source LLMs. Across general-purpose LLMs, MolE-RAG improves ROC-AUC by up to 28 percentage points on classification tasks and reduces regression RMSE by up to 67% relative to a SMILES-only baseline. We further find that the utility of each context source varies across models and tasks, with different models benefiting most from textual retrieval, molecular context, or structural retrieval. These results suggest that molecule-centric retrieval can improve LLM-based molecular property prediction without model fine-tuning while providing a flexible framework for integrating heterogeneous chemical knowledge at inference time.

Joey Chan, Wonbin Kweon, Ashley Shin +4

University of Illinois Urbana-Champaign · University of California, San Diego

Retrieval-Augmented GenerationLLM InferenceMolecular Property Prediction
AskPDF