SIGMA: Semantic Identifier Grouping for Molecular Autoregression

Xinyu Wang; Fei Dou; Jinbo Bi; Minghu Song

cs.LGMar 26, 2026

SIGMA: Semantic Identifier Grouping for Molecular Autoregression

Authors: Xinyu Wang, Fei Dou, Jinbo Bi, Minghu Song

Organizations: School of Computing, University of Connecticut, Storrs, CT, USA · Institute for Artificial Intelligence, University of Georgia, Athens, GA, USA · Institute of Hydrobiology, Chinese Academy of Sciences, Wuhan, China

Abstract

Autoregressive molecular models assign probability to molecular serializations even though chemical identity is invariant to serialization. Equivalent serializations can therefore represent a common molecular identity yet induce inconsistent next-token decisions. Randomized strings broaden exposure, but do not reveal which intermediate decisions should agree. We introduce SIGMA, a dense suffix-position objective built from chemically certified same-suffix triplets: two equivalent histories, one non-equivalent history, and a shared suffix. SIGMA aligns corresponding pre-token hidden states along the continuation and separates the negative to a finite relative margin, leaving the language-model objective, decoder, and inference procedure unchanged. We compare SIGMA with canonical training, randomized-serialization training, and last-token alignment across four datasets under SMILES and SELFIES. Across the eight representation-dataset blocks, SIGMA yields clear test-reference Frechet ChemNet Distance reductions in six: all four SELFIES domains and QM9 and ZINC under SMILES, with paired 95% confidence intervals below zero against every control. Position-wise analyses show improved state correspondence, chemical discrimination, and next-token agreement while preserving between-molecule information. On two full-corpus ZINC blocks, compute-matched ablations identify chemically correct state correspondence as the effective ingredient. Beyond generation, SIGMA improves mean predictive performance on all six molecular property benchmarks and reduces sensitivity to equivalent molecular serializations on every task.

Figures & tables

Figure 1: Last-token alignment constrains only the endpoint, whereas SIGMA aligns every position of an exact shared suffix and retains a margin-gated non-equivalent history. The artwork uses positive B for the upper example and positive C for the lower example; these are alternative illustrations, not simultaneous inputs. Each implemented triplet contains one anchor, one equivalent positive, and one non-equivalent negative. Artwork D therefore corresponds to the negative denoted C in the Method notation.

	SMILES				SELFIES
Dataset	Canon.	Random.	Last-token	SIGMA	Canon.	Random.	Last-token	SIGMA
QM9	.6655 $\pm$ .0610	.8013 $\pm$ .0286	.6474 $\pm$ .0568	.5832 $\pm$ .0441	.8557 $\pm$ .0644	1.0510 $\pm$ .1080	.8491 $\pm$ .0418	.6846 $\pm$ .0120
ZINC	1.9655 $\pm$ .2463	2.3486 $\pm$ .2065	1.9960 $\pm$ .2080	1.8348 $\pm$ .1975	6.8046 $\pm$ .1125	8.1759 $\pm$ .1164	6.9863 $\pm$ .2009	6.2527 $\pm$ .1241
ChEMBL	1.5914 $\pm$ .0851	1.7327 $\pm$ .0958	1.6034 $\pm$ .0999	1.6090 $\pm$ .0879	3.1631 $\pm$ .0683	3.7631 $\pm$ .0756	3.2637 $\pm$ .1064	3.1095 $\pm$ .0644
MOSES-300k	5.7967 $\pm$ .1501	6.1355 $\pm$ .0387	5.8048 $\pm$ .0690	5.7502 $\pm$ .0851	9.6606 $\pm$ .2699	11.7648 $\pm$ .1037	9.9398 $\pm$ .0523	9.1999 $\pm$ .1524

Table 1: Test-reference FCD under SMILES and SELFIES (mean

\pm

standard deviation across three seeds; lower is better; bold is best).

Figure 2: The final margin-gated model changes the whole shared-suffix prediction path, not only its endpoint. Curves aggregate the four SMILES datasets and three training seeds (12 dataset–seed conditions); markers are condition means and ribbons show one standard deviation across conditions. The horizontal axes run from the beginning (0) to the end (1) of the shared suffix. At every normalized suffix bin, SIGMA identifies the aligned state of the certified equivalent prefix more often and is less likely to rank the certified non-equivalent prefix above it.

Task	Metric	Randomized serialization	SIGMA
ESOL	RMSE	$1.013\pm.081$	$\mathbf{.970\pm.082}$
FreeSolv	RMSE	$2.320\pm.155$	$2.246\pm.108$
Lipophilicity	RMSE	$.758\pm.021$	$\mathbf{.722\pm.005}$
BACE	AUROC	$.823\pm.012$	$.834\pm.013$
BBBP	AUROC	$.920\pm.026$	$.943\pm.005$
ClinTox	AUROC	$.761\pm.155$	$.854\pm.061$

Table 2: Transfer across six molecular-property benchmarks. Predictions average two paired serializations per molecule; values are mean

\pm

training-seed standard deviation. Lower RMSE and higher AUROC are better. Complete paired 95% intervals appear in Supplementary Section 5.

Training objective	S–ZINC	F–ZINC
Full-corpus LM	1.094 $\pm$ .044	6.330 $\pm$ .114
Output-distribution alignment	1.137 $\pm$ .043	6.401 $\pm$ .226
SIGMA-Pos	1.062 $\pm$ .069	5.725 $\pm$ .061
Shuffled-pair compute control	1.087 $\pm$ .023	6.333 $\pm$ .068
SIGMA-Margin	1.043 $\pm$ .064	5.788 $\pm$ .126

Table 3: Separating compute and alignment on two full-corpus ZINC validation blocks (FCD

\downarrow

, mean

\pm

SD over three training seeds). Every row uses the same full language-model corpus, initialization, optimizer schedule, and five paired generation repetitions. S denotes SMILES and F denotes SELFIES; Supplementary Section 7 reports the complete three-block table.

Molecular Generation

Molecular generation develops computational methods to create chemical structures and conformations for discovery, optimization, and structure elucidation.

See all 234 papers

Augmenting Molecular Language Models with Local $n$ -gram Memory
Jun 10, 2026Xinni Zhang, Zijing Liu, He Cao +2Memory-Augmented Language Models Language Modeling
1The Chinese University of Hong Kong · 2International Digital Economy Academy
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
Jul 6, 2026Hunter HeidenreichSubword Tokenization Byte-Pair Encoding
Independent Researcher
SMolLM: Small Language Models Learn Small Molecular Grammar
May 7, 2026Akhil Jindal, Harang JuSmall Language Models Mechanistic Interpretability
Moku

Jun 10, 2026J/K move · Enter open · S save

Augmenting Molecular Language Models with Local $n$ -gram Memory

Xinni Zhang, Zijing Liu, He Cao, Yu Li+1

1The Chinese University of Hong Kong · 2International Digital Economy Academy

Memory-Augmented Language Models Language Modeling Molecular Generation

Open paper AskPDF arXiv

Transformer-based language models for SMILES strings suffer from a locality gap: standard character-level tokenization fragments chemically meaningful motifs, forcing models to repeatedly learn local syntax at the expense of long-range dependencies. To address this without disrupting standard tokenizers, we propose MolGram, which integrates a conditional

n

-gram memory module into molecular language models. MolGram maps local string patterns to learned embeddings via scalable hash lookups and dynamically injects this regional context into hidden states. Evaluations across three tasks, including unconditional molecule generation, forward reaction prediction, and single-step retrosynthesis, show that MolGram consistently improves performance. Crucially, our analyses demonstrate that MolGram outperforms baselines with 3

\times

more parameters, establishing explicit local pattern memory as a highly efficient inductive bias.

Abstract

Molecular Generation

Explore similar work

Augmenting Molecular Language Models with Local nnn-gram Memory

Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

SMolLM: Small Language Models Learn Small Molecular Grammar

Augmenting Molecular Language Models with Local $n$ -gram Memory