cs.LGNov 2, 2026

Quantifying out-of-distribution data severity with Wasserstein distances: benchmarking large language models for alloy property prediction

Authors: YuJie Liu, Dezhen Xue, Jun Sun, Xiangdong Ding, Zhibin Gao, Turab Lookman

Organizations: State Key Laboratory for Mechanical Behavior of Materials, State Key Laboratory of Porous Metal Materials, School of Materials Science and Engineering,Xi’an Jiatong University, Xi’an 710049, China · AiMaterial Research, Santa Fe, New Mexico, USA 87501

Abstract

General-purpose Large Language Models (LLMs) increasingly rival domain-specific pipelines on scientific prediction, but their performance under distribution shift is untested. Across six alloy and crystal-structure datasets, we benchmark LLMs against traditional, BERT-based, TabPFN, and text-regression baselines under random cross-validation and six out-of-distribution (OOD) protocols shifting input and target distributions, with severity quantified via a standardized Wasserstein distance. We isolate the LLM's advantage over k-nearest-neighbor (k-NN) retrieval: it beats k-NN on all eight tasks at a fixed 5% reference budget, on seven of eight when independently optimized, and in 72.5% of 120 OOD task--protocol configurations against an optimized baseline. No model family is uniformly best; the LLM is often strongest under severe shift, and error increases with target-space but not representation-space severity. Self-reported LLM confidence tracks both shift and error, more consistently than a density-only control from the same retrieved samples, and a matched-cohort check finds no substantial evidence of memorization.

Explore similar work

CardsList
  1. Probing Materials Knowledge in LLMs: From Latent Embeddings to Reliable Predictions

    Mar 2, 2026Vineeth Venugopal, Soroush Mahjoubi, Elsa OlivettiLLM ReliabilityLLM Evaluation

  2. From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals

    May 28, 2026Yeyong Yu, Wenya Hu, Xing Wu +1LLM EvaluationPairwise Preference Evaluation

  3. MOTIF: LLM-Defined Material Taxonomies for OOD Property Prediction via Invariant Feature Selection

    Nov 9, 2026Gwangseon Jang, Sieun Yun, Dong Ho Lee +4Materials Property PredictionOOD Generalization