physics.chem-phFeb 18, 2026

Clever Materials: When Models Identify Good Materials for the Wrong Reasons

Authors: Kevin Maik Jablonka

Organizations: Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldtstrasse 10, 07743 Jena, Germany · Helmholtz Institute for Polymers in Energy Applications Jena (HIPOLE Jena), Lessingstrasse 12-14, 07743 Jena, Germany · Center for Energy and Environmental Chemistry Jena (CEEC Jena), Friedrich Schiller University Jena, Philosophenweg 7a, 07743 Jena, Germany · Jena Center for Soft Matter (JCSM), Friedrich Schiller University Jena, Philosophenweg 7, 07743 Jena, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Machine learning can accelerate materials discovery. Models perform impressively on many benchmarks. However, strong benchmark performance does not imply that a model learned chemistry. I test a concrete alternative hypothesis: that property prediction can be driven by bibliographic confounding. Across five tasks spanning MOFs (thermal and solvent stability), perovskite solar cells (efficiency), batteries (capacity), and TADF emitters (emission wavelength), models trained on standard chemical descriptors predict author, journal, and publication year well above chance. When these predicted metadata ("bibliographic fingerprints") are used as the sole input to a second model, performance is sometimes competitive with conventional descriptor-based predictors. These results show that many datasets do not rule out non-chemical explanations of success. Progress requires routine falsification tests (e.g., group/time splits and metadata ablations), datasets designed to resist spurious correlations, and explicit separation of two goals: predictive utility versus evidence of chemical understanding.

Figures & tables

Figure 1
Figure 2: For the classification task of membership in the top-10% of thermally stable MOFs, one can be fooled (by Clever Hans effects). a The model predicts the authors of the associated paper with high accuracy, much better than a random baseline. b This also holds for predicting in which journal the entry was published or the year in which the paper was published c . Using the predicted bibliographic information, a model can predict with high accuracy if the MOF belongs to the top-10% thermally stable ones. However, the effect is smaller — or not even there — if analyzed under a different metric or for a regression setting (see Section A.2 ). The dummy baseline for classification is a stratified random sampling (using the empirical probabilities from the training dataset) and the mean prediction for the regression case.
Figure 3: Measuring performance using accuracy, “Clever Hans models” can achieve a surprisingly good performance in predicting solvent removal stability of MOFs. a MOF descriptors can be used to predict author information meaningfully better than a random baseline. b The journal in which a given MOF stability result has been published can also be predicted with non-trivial accuracy. c The publication year can be predicted with a mean average error of around two years. d A model trained on predicted bibliographic information (actual bibliometric and predicted one) can achieve non-trivial accuracy in correctly predicting the solvent stability of MOFs. Figure 7 shows the performance measured with other metrics.
Figure 4: Clever Hans models can achieve a high accuracy in predicting if a perovskite material is within the top-10% most efficient absorbers. a Composition description can predict which of the ten most prolific authors has been reporting a composition with a micro F1F_{1} score of 0.318. b The journal in which a device has been published can be predicted with a similar performance. c The publication year can be predicted meaningfully better than with a mean baseline. d A model trained on this predicted bibliometric information can achieve an accuracy in classifying if an aborber belongs to the top-10% most efficient ones with an accuracy indistinguishable from the model directly trained on composition features.
Figure 5: “Clever Hans models” perform worse than “conventional” models but better than simple baselines in predicting the maximum emission wavelengths of TADF molecules. a Using molecular descriptors, the model can predict the the authors of the paper describing a material with an accuracy of 55.434% among the 1000 most profilic authors. b The model achieves an even higher performance in predicting in which of the ten most common journals a given entry has been published. c The publication year, too, can be predicted with a high performance just based on composition descriptors. d Using predicted bibliometric information, one can achieve a mean absolute error between the “conventional” and naïve baseline models. The performance using actual bibliometric information is not much different from using the correct one.
Figure 6: “Clever Hans models” do not achieve a good performance in predicting battery capacity. a Composition descriptors can be used to determine which of the 500 most prolific authors reported this material with a mediocre F1F_{1} score. b The journal in which a material has been reported can be predicted with a similar performance. c The publication year can be predicted better than with the mean baseline. d “Clever Hans” models are not distinguishable from mean predictions ( “Dummy” ) for the prediction of the capacity of battery materials.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Performance of MOF solvent removal stability classification measured with different metrics. In all metrics, “Clever Hans models” outperform simple baselines. In some metrics, such as precision, “Clever Hans models” come close in performance to models directly trained on MOF descriptors ( “Conventional” ).
Figure 8: Parameter sweep for MOF solvent stability. Performance of proxy models as a function of the type and number of predicted bibliometric features.
Figure 9: MOF thermal stability prediction performance across different metrics. The measured difference between conventional and proxy models depends on the evaluation metric chosen.
Figure 10: Parameter sweep analysis for MOF thermal stability. Performance of proxy models as a function of author count thresholds and inclusion of temporal/journal information.
Figure 11: Performance in predicting bibliometric information and in predicting photoconversion efficiencies.
Figure 12: Impact of the number of bibliometric features on the performance of “Clever Hans” models in predicting PCE.
Figure 13: Perovskite PCE prediction performance across metrics. Comparison of conventional and proxy models using different regression evaluation metrics.
Figure 14: Perovskite top-10% classification performance across metrics. Different evaluation metrics reveal varying degrees of Clever Hans effects.
Figure 15: Parameter sweep for perovskite top-10% classification. Performance sensitivity to author count thresholds and temporal/journal features.
Figure 16: TADF wavelength prediction performance across metrics. Moderate proxy learning effects between conventional and baseline performance.
Figure 17: Parameter sweep for TADF wavelength prediction. Performance variation with different bibliometric feature configurations.
More in this topic

Spurious Correlation

See all 28 papers

Explore similar work

CardsList
May 28, 2026

OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields

As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdisciplinary, multimodal, and application-driven nature. However, existing materials benchmarks mainly focus on property prediction, knowledge QA, or characterization understanding, leaving the broader reasoning process from materials knowledge to application underexplored. To fill this gap, we present OmniMatBench, a human-calibrated multimodal reasoning benchmark for materials science. OmniMatBench contains 3,171 expert-curated QA and calculation problems across 19 materials-science subfields, spanning fundamental materials knowledge, structural and engineering materials, materials processing and manufacturing, and functional and applied materials. We evaluate 13 open-source and closed-source MLLMs and find that the best model achieves only a 0.372 overall score, revealing a substantial gap in current materials-science reasoning. Further analysis shows strong variation across subfields, fixed reasoning heuristics, uneven materials knowledge, and limited high-level knowledge application under formula-, retrieval-, and code-assisted settings. OmniMatBench provides crucial insights into the capabilities and limitations of current MLLMs and establishes a foundation for reliable AI assistants in materials-science research.

Wanhao Liu, Jiaqing Xie, Qian Tan +10

University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Fudan University +4

Benchmark DesignScientific QAMultimodal ReasoningMaterials ScienceMultimodal Model Evaluation
AskPDF
Mar 2, 2026

Probing Materials Knowledge in LLMs: From Latent Embeddings to Reliable Predictions

Large language models are increasingly applied to materials science, yet fundamental questions remain about their reliability and knowledge encoding. Evaluating 25 LLMs across four materials science tasks -- over 200 base and fine-tuned configurations -- we find that output modality fundamentally determines model behavior. For symbolic tasks, fine-tuning converges to consistent, verifiable answers with reduced response entropy, while for numerical tasks, fine-tuning improves prediction accuracy but models remain inconsistent across repeated inference runs, limiting their reliability as quantitative predictors. For numerical regression, we find that better performance can be obtained by extracting embeddings directly from intermediate transformer layers than from model text output, revealing an ``LLM head bottleneck,'' though this effect is property- and dataset-dependent. Finally, we present a longitudinal study of GPT model performance in materials science, tracking four models over 18 months and observing 9--43% performance variation that poses reproducibility challenges for scientific applications.

Vineeth Venugopal, Soroush Mahjoubi, Elsa Olivetti

Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, 02139, MA, USA.

LLM ReliabilityLLM EvaluationLanguage Model ProbingLLM Fine-TuningNumerical Reasoning in Language ModelsMaterials Property Prediction
AskPDF
Jul 22, 2026

Generative and multimodal AI for materials prediction and design: Progress, challenges, and perspectives

Artificial intelligence (AI) is accelerating materials prediction and design by enabling efficient exploration of chemical and structural spaces, with particular promise for novel materials discovery. However, novelty in materials discovery encompasses chemical plausibility, structural distinctiveness, property relevance and experimental realisability, making AI-driven novelty claims difficult to substantiate. We introduce a materials property hierarchy, from intrinsic, composition-determined properties to extrinsic, processing-dependent performance, to clarify deployment constraints and distinguish structural, physical and deployment novelty. This framework motivates an evidence-based view of multimodal materials data spanning chemical composition, microstructure, processing, and testing and characterisation, showing that current evidence remains concentrated in composition and idealised structure while heterogeneous, under-represented and weakly integrated modalities limit support for physical and deployment novelty. It also highlights the limitations of benchmarks based mainly on computational labels and proxy novelty criteria. Community-wide standards for data collection, modality alignment and evidence synthesis are needed to support multimodal data construction, process-aware multimodal modelling, feasibility-first generative modelling and deployment-aware benchmarking, so that generative and multimodal AI can design experimentally realisable materials with defensible scientific and practical novelty.

Xianyuan Liu, Charles Anjah, Benjamin E. Jolly +9

School of Computer Science, University of Sheffield, Sheffield, UK · Centre for Machine Intelligence, University of Sheffield, Sheffield, UK · School of Chemical, Materials and Biological Engineering, University of Sheffield, Sheffield, UK +3

Materials Property PredictionMaterials ScienceMultimodal GenerationMultimodal Learning
AskPDF