In-context molecular property prediction with LLMs: a blinding study on memorization and knowledge conflicts
Citation Link: https://doi.org/10.15480/882.19313
Publikationstyp
Journal Article
Date Issued
2026-09-17
Sprache
English
Author(s)
TORE-DOI
Volume
66
Issue
19
Start Page
12305
End Page
12315
Citation
Journal of Chemical Information and Modeling 66 (19): 12305-12315 (2026)
Publisher DOI
Scopus ID
Publisher
American Chemical Society (ACS)
The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or instead rely on verbatim retrieval of memorized target values. Furthermore, we analyze the interplay between pretrained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet data sets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information, complemented by 0-, 60-, and 1000-shot in-context sample sizes as an additional control for information access. To validate the memorization analysis and the blinding experiments, we add a positive and a negative control for the memorization experiments and structural reference baselines for the multishot experiments, as well as bootstrap confidence intervals for all results. We find no evidence of verbatim retrieval on the legacy benchmarks and show that blinding exposes conflicts between pretrained knowledge and in-context information. This work provides a principled framework for evaluating molecular property prediction under controlled information access.
DDC Class
006.31: Machine Learning
540: Chemistry
519: Applied Mathematics, Probabilities
Publication version
publishedVersion