Mar 26, 2026 · cs.LGJ/K move · Enter open · S save
Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich+3
Institute for Continuum and Material Mechanics, Technical University of Hamburg, Eißendorfer Straße, 21073 Hamburg, Germany · Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 21502 Geesthacht, GermanyJun · Institute of Surface Science, Helmholtz-Zentrum Hereon, Max-Planck-Straße, 215022026 Geesthacht, Germany · 4German Center for Artificial Intelligence (DFKI), Saarland, Germany · 5Saarland University, Saarbrücken, Germany22
The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or rely primarily on memorized values. Furthermore, we analyze the interplay between pre-trained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet datasets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information. Complementing this, we utilize varying in-context sample sizes (0-, 60-, and 1000-shot) as an additional control for information access. This work provides a principled framework for evaluating molecular property prediction under controlled information access, addressing concerns regarding memorization and exposing conflicts between pre-trained knowledge and in-context information.