Generative Chemical Language Models for Energetic Materials Discovery
Authors: Andrew Salij, R. Seaton Ullberg, Megan C. Davis, Marc J. Cawkwell, Christopher J. Snyder, Cristina Garcia Cardona, Ivana Matanovic, Wilton J. M. Kort-Kamp
Organizations: Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA · Weapon Stockpile Modernization Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA · Computing and Artificial Intelligence Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
The discovery of new energetic materials remains a pressing challenge hindered by limited availability of high-quality data. To address this, we have developed generative molecular language models that have been pretrained on extensive chemical data and then fine-tuned with curated energetic materials datasets. This transfer-learning strategy extends the chemical language model capabilities beyond the pharmacological space in which they have been predominantly developed, offering a framework applicable to other data-spare discovery problems. Furthermore, we discuss the benefits of fragment-based molecular encodings for chemical language models, in particular in constructing synthetically accessible structures. Together, these advances provide a foundation for accelerating the design of next-generation energetic materials with demanding performance requirements.
Figures & tables
Figure 1: a) Training pipeline for GPT models, staged into pretraining to produce a wide variety of molecules and fine-tuning, which produces many C-, N-, O-containing compounds. b) Scheme of GPT model architecture and data processing.
Model (Encoding)
Validity ( ↑ )
Uniqueness ( ↑ )
Diversity ( ↑ )
Novelty ( ↑ )
MolGPT Bagal et al. (2021) (SMILES)
0.994
1.0
0.857
0.797
GenMol Lee et al. (2025) (SMILES)
1.000±0.000
0.997±0.001
0.818±0.001
-
Group-VAE-125 Cheng et al. (2023)
1.0(0)
0.9985(4)
0.8587(1)
0.7187(11)
SELFIES-VAE-125 Cheng et al. (2023)
1.0(0)
0.9986(4)
0.8579(1)
0.7345(16)
χ hem-GPT (SELFIES)
0.999±0.001
1.000±0.000
0.820±0.001
0.998±0.001
(LARGE) (GSELFIES i )
0.911±0.006
0.996±0.002
0.821±0.001
0.995±0.001
Table 1: Benchmark metrics for the large and small χ hem-GPT models encoded via SELFIES and GroupSELFIES i compared to prior art in the literature. For χ hem-GPT models, 5000 molecules have been generated in N=5 isolated runs of 1000 molecules each. Validity, uniqueness, novelty, and diversity are determined as scalars ranging from 0−1 for each isolated run, while the other metrics have their means and variances pooled. GroupSELFIES i is GroupSELFIES that has its index alphabet enforced to be identical to SELFIES. Cells for literature models source from their respective papers, and cells are blank where the corresponding metric does not exist.
Figure 2: a) Synthetic accessibility (SA) score Ertl and Schuffenhauer (2009) distributions for unconditioned molecular outputs of pretrained χ hem- and fine-tuned X-GPT models with a) number of heavy atoms generated and b) predicted detonation velocities via ChemProp Heid et al. (2023) surrogate. All subfigures are normalized such that the highest histogram bin is 1.
Temp
Validity ( ↑ )
Diversity ( ↑ )
SA Score ( ↓ )
N Radicals ( ↓)
N Heavy Atoms
0.8
0.999±0.001
0.811±0.003
4.27±1.16
0.13±0.47
25.04±11.79
1.0
0.999±0.001
0.820±0.001
4.63±1.18
0.33±0.85
23.57±9.03
1.2
0.994±0.003
0.839±0.003
4.97±1.18
0.67±1.21
22.35±9.37
2.0
0.892±0.005
0.943±0.002
6.56±1.19
3.27±3.45
13.84±9.40
Table 2: Temperature screening for the large χ hem-GPT model trained using SELFIES over benchmark metrics. Runs and statistics performed equivalently to in Table 1 .
Table 3: Comparison of generated outputs between the base and fine-tuned GPT models. a) General metrics from all generated molecular texts across multiple machines ( N=5000 ). b) Kamlet-Jacobs Kamlet and Jacobs (1968) (KJ) metrics as characterized on a subset of the molecules in (a), filtered by converged calculations ( 185≤N≤530 ). The condition for high P , v corresponds to (p,v)=(40GPa,10km/s) . Standard fine-tuning is where the first and final transformer layers are unfrozen alongside the token output layer, while LoRA defines low-rank adaptation matrices with rank dimension r=8 and α=32 . Statistics follow the procedure outlined for Table 1 .
Figure 3: Comparison of estimated detonation velocities and pressures from Kamlet-Jacobs equations Kamlet and Jacobs (1968) for unconditioned (left column) and conditioned generation (right) for fine-tuned GroupSELFIES models compared to the base model. All subfigures have been normalized to an identical maximum value.
Figure 4: Distributions of a) number of nitrogen-oxygen bonds, b) number of nitrogen-nitrogen bonds, c) quantitative estimation of druglikeness (QED), and d) synthetic accessibility score (SA Score) in large SELFIES-based GPT models.
Figure 5: Common substructures of generated output from chemical language models that have been manually selected for diversity. Substructures were chosen as representative samples from the 200 most common subgraphs between size 5−10 as obtained via RDKit RDKit development team (2025) .
Name
SELFIES Size
GroupSELFIES Size
Sources
SAFE-8M
7897563
7905990
Noutahi et al. (2024) ; Irwin and Shoichet (2005) ; Chambers et al. (2013)
X-17K
17407
17389
Allen (2002)
Table S1: Overview of datasets used in this study. Note that for the dataset size, this is of the entire dataset prior to train/val/test splitting, and final batches smaller than the batch size are dropped during training. Note that datasets differ trivially in exact size and composition due to rare parsing issues in constructing the datasets from initial SMILES.
Figure S1: Distribution of synthetic accessibility to heavy atoms in molecule for SAFE-8M.
Figure S2: Distribution of druglikeness to molecular weight in molecule for SAFE-8M.
Figure S3: Calculated velocities and pressures from CHEETAH Fried (1994) for X-17K.
Model
Small (SELFIES)
Small (GSELFIES)
Large (SELFIES)
Large (GSELFIES)
Parameters
40 M
40 M
150 M
160 M
N Layers
12
12
12
12
dembed
512
512
1028
1028
wcontext
256
256
512
512
nhead
4
4
4
4
vocab. size
975
1617
975
1617
Table S2: Model architecture parameters. Parameters are rounded to the nearest 10 million and obtained via a summation of all registered parameters and vocabulary size includes special tokens.
Figure S4: Training and validation loss for χ hem-GPT (LARGE, SELFIES).
Figure S5: Training and validation loss for χ hem-GPT (LARGE, GroupSELFIES).
Figure S6: Training and validation loss for χ hem-GPT (SMALL, SELFIES).
Figure S7: Training and validation loss for χ hem-GPT (SMALL, GroupSELFIES).
Figure S8: Validation of ChemProp model for detonation a) velocities and b) pressures for the X-17K dataset.
Figure S9: Distributions of a) number of nitrogen-oxygen bonds, b) number of nitrogen-nitrogen bonds, c) quantitative estimation of druglikeness (QED), and d) synthetic accessibility score (SA Score) in large GroupSELFIES i -based GPT models.
Figure S10: Comparison of estimated detonation velocities and pressures via XChemProp for unconditioned generation of χ hem- and X-GPT (standard fine-tuning) models. All subfigures normalized to an identical maximum value.
Figure S11: Comparison of CHEETAH calculated (not estimated) detonation properties for χ hem- and X-GPT GroupSELFIES i models. All subfigures normalized to an identical maximum value.
Figure S12: Comparison of CHEETAH calculated (not estimated) detonation properties for χ hem- and X-GPT SELFIES models. All subfigures normalized to an identical maximum value.
Figure S13: Comparison of Kamlet-Jacobs calculated (not estimated) detonation properties for χ hem- and X-GPT SELFIES models. All subfigures normalized to an identical maximum value.
Molecular generation develops computational methods to create chemical structures and conformations for discovery, optimization, and structure elucidation.
CO2 reduction requires efficient catalysts, yet materials discovery remains bottlenecked by 10-20 year development cycles requiring deep domain expertise. This paper demonstrates how large language models can assist the catalyst discovery process by helping researchers explore chemical spaces and interpret results when augmented with retrieval-based grounding. We introduce a retrieval-augmented generation framework that enables GPT-4 to navigate chemical space by accessing a database of 50,000+ known materials, adapting general-purpose language understanding for high-throughput materials design. Our approach generated over 250 catalyst candidates with an 82% thermodynamic stability rate while addressing multi-objective constraints: 68% achieved <100/kgcostwithmetallicconductivity(bandgap<0.1eV)andmechanicalstability(B/G>1.75).Thebest−performingFe0.2Co0.2Ni0.2Ir0.1Ru0.3achieves0.285Vlimitingpotential(2518/kg. Volcano plot analysis confirms that 78% of LLM-generated catalysts cluster near the theoretical activity optimum, while our system achieves 200x computational efficiency compared to traditional high-throughput screening. By demonstrating that retrieval-augmented generation can ground AI creativity in physical constraints without sacrificing exploration, this work demonstrates an approach where natural language interfaces can streamline materials discovery workflows, enabling researchers to explore chemical spaces more efficiently while the LLM assists in result interpretation and hypothesis generation.
AI Scientists, Xinyi Lin, Danqing Yin +1
School of Biomedical Sciences, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong SAR, 999077, China · Laboratory of Data Discovery for Health Limited (D24H), Pak Shek Kok, Hong Kong SAR, 999077, China · Room 312, Lau Chung Him Building, 8 Castle Peak Road, Tuen Mun, Hong Kong
Atomistic structure and natural language have long been modeled separately, with language models either calling atomistic models as tools or being fine-tuned on lossy textual encodings that discard atomistic information. We introduce Atomistic Language Models (ALMs) to pursue native multimodality, in which a single language backbone understands atomistic structures, generates materials from natural language, and optimizes crystal structures as instructed by text. By unifying a pretrained atomistic encoder, large language model, and denoising diffusion model through purely continuous projectors and staged training, ALMs achieve state-of-the-art results on crystal structure prediction and de novo generation. ALMs are enabled by a continuous bridge that maps language model embeddings directly into the steering space of atomistic diffusion, and are assisted by Text-to-Crystal Feynman-Kac (T2C-FK), a particle-based sampler that scores partial denoising trajectories to enforce stoichiometric targets at inference time. To evaluate the ability of ALMs to optimize and generate materials from natural-language prompts and 3D atom-coordinate inputs, we introduce ALM Bench, the first benchmark for text-conditioned crystal generation and optimization. Code, training data, and model weights will be released soon.
Sathya Edamadaka, Krithik Ramesh, Ju Li +1
Massachusetts Institute of Technology · Lyra Labs · Lila Sciences
Chemical language models (CLMs) are trained with linearized representations such as SMILES, yet it remains unclear which chemically meaningful substructures they encode. To foster a better understanding of CLMs, we conduct a systematic study and probe for 78 molecular substructures across eight pre-trained and six randomly initialized models. We furthermore study how fine-tuning on chemical downstream tasks affects the learned representations of molecular substructures. Our results show that pre-training generally improves molecular structure awareness of CLMs, particularly in the upper layers. Moreover, randomly initialized models already encode ring structures well in the first layer. Our analysis on two chemical downstream tasks further reveals that, interestingly, fine-tuning affects task-relevant molecular substructures more than others, indicating that the changes in the representations follow chemical theory.
Anna Karnysheva, Dietrich Klakow, Ji-Ung Lee
1RTG Neuroexplicit Models · 2Spoken Language Systems · Saarland University +1