SIGMA: Semantic Identifier Grouping for Molecular Autoregression
Authors: Xinyu Wang, Fei Dou, Jinbo Bi, Minghu Song
Organizations: School of Computing, University of Connecticut, Storrs, CT, USA · Institute for Artificial Intelligence, University of Georgia, Athens, GA, USA · Institute of Hydrobiology, Chinese Academy of Sciences, Wuhan, China
Autoregressive molecular models assign probability to molecular serializations even though chemical identity is invariant to serialization. Equivalent serializations can therefore represent a common molecular identity yet induce inconsistent next-token decisions. Randomized strings broaden exposure, but do not reveal which intermediate decisions should agree. We introduce SIGMA, a dense suffix-position objective built from chemically certified same-suffix triplets: two equivalent histories, one non-equivalent history, and a shared suffix. SIGMA aligns corresponding pre-token hidden states along the continuation and separates the negative to a finite relative margin, leaving the language-model objective, decoder, and inference procedure unchanged. We compare SIGMA with canonical training, randomized-serialization training, and last-token alignment across four datasets under SMILES and SELFIES. Across the eight representation-dataset blocks, SIGMA yields clear test-reference Frechet ChemNet Distance reductions in six: all four SELFIES domains and QM9 and ZINC under SMILES, with paired 95% confidence intervals below zero against every control. Position-wise analyses show improved state correspondence, chemical discrimination, and next-token agreement while preserving between-molecule information. On two full-corpus ZINC blocks, compute-matched ablations identify chemically correct state correspondence as the effective ingredient. Beyond generation, SIGMA improves mean predictive performance on all six molecular property benchmarks and reduces sensitivity to equivalent molecular serializations on every task.
Figures & tables
Figure 1: Last-token alignment constrains only the endpoint, whereas SIGMA aligns every position of an exact shared suffix and retains a margin-gated non-equivalent history. The artwork uses positive B for the upper example and positive C for the lower example; these are alternative illustrations, not simultaneous inputs. Each implemented triplet contains one anchor, one equivalent positive, and one non-equivalent negative. Artwork D therefore corresponds to the negative denoted C in the Method notation.
SMILES
SELFIES
Dataset
Canon.
Random.
Last-token
SIGMA
Canon.
Random.
Last-token
SIGMA
QM9
.6655 ± .0610
.8013 ± .0286
.6474 ± .0568
.5832 ± .0441
.8557 ± .0644
1.0510 ± .1080
.8491 ± .0418
.6846 ± .0120
ZINC
1.9655 ± .2463
2.3486 ± .2065
1.9960 ± .2080
1.8348 ± .1975
6.8046 ± .1125
8.1759 ± .1164
6.9863 ± .2009
6.2527 ± .1241
ChEMBL
1.5914 ± .0851
1.7327 ± .0958
1.6034 ± .0999
1.6090 ± .0879
3.1631 ± .0683
3.7631 ± .0756
3.2637 ± .1064
3.1095 ± .0644
MOSES-300k
5.7967 ± .1501
6.1355 ± .0387
5.8048 ± .0690
5.7502 ± .0851
9.6606 ± .2699
11.7648 ± .1037
9.9398 ± .0523
9.1999 ± .1524
Table 1: Test-reference FCD under SMILES and SELFIES (mean ± standard deviation across three seeds; lower is better; bold is best).
Figure 2: The final margin-gated model changes the whole shared-suffix prediction path, not only its endpoint. Curves aggregate the four SMILES datasets and three training seeds (12 dataset–seed conditions); markers are condition means and ribbons show one standard deviation across conditions. The horizontal axes run from the beginning (0) to the end (1) of the shared suffix. At every normalized suffix bin, SIGMA identifies the aligned state of the certified equivalent prefix more often and is less likely to rank the certified non-equivalent prefix above it.
Task
Metric
Randomized serialization
SIGMA
ESOL
RMSE
1.013±.081
.970±.082
FreeSolv
RMSE
2.320±.155
2.246±.108
Lipophilicity
RMSE
.758±.021
.722±.005
BACE
AUROC
.823±.012
.834±.013
BBBP
AUROC
.920±.026
.943±.005
ClinTox
AUROC
.761±.155
.854±.061
Table 2: Transfer across six molecular-property benchmarks. Predictions average two paired serializations per molecule; values are mean ± training-seed standard deviation. Lower RMSE and higher AUROC are better. Complete paired 95% intervals appear in Supplementary Section 5.
Training objective
S–ZINC
F–ZINC
Full-corpus LM
1.094 ± .044
6.330 ± .114
Output-distribution alignment
1.137 ± .043
6.401 ± .226
SIGMA-Pos
1.062 ± .069
5.725 ± .061
Shuffled-pair compute control
1.087 ± .023
6.333 ± .068
SIGMA-Margin
1.043 ± .064
5.788 ± .126
Table 3: Separating compute and alignment on two full-corpus ZINC validation blocks (FCD ↓ , mean ± SD over three training seeds). Every row uses the same full language-model corpus, initialization, optimizer schedule, and five paired generation repetitions. S denotes SMILES and F denotes SELFIES; Supplementary Section 7 reports the complete three-block table.
Molecular generation develops computational methods to create chemical structures and conformations for discovery, optimization, and structure elucidation.
Transformer-based language models for SMILES strings suffer from a locality gap: standard character-level tokenization fragments chemically meaningful motifs, forcing models to repeatedly learn local syntax at the expense of long-range dependencies. To address this without disrupting standard tokenizers, we propose MolGram, which integrates a conditional n-gram memory module into molecular language models. MolGram maps local string patterns to learned embeddings via scalable hash lookups and dynamically injects this regional context into hidden states. Evaluations across three tasks, including unconditional molecule generation, forward reaction prediction, and single-step retrosynthesis, show that MolGram consistently improves performance. Crucially, our analyses demonstrate that MolGram outperforms baselines with 3× more parameters, establishing explicit local pattern memory as a highly efficient inductive bias.
Xinni Zhang, Zijing Liu, He Cao +2
1The Chinese University of Hong Kong · 2International Digital Economy Academy
Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.
Language models for molecular design have scaled to hundreds of millions of parameters, yet how they learn chemical grammar is poorly understood. We train SMolLM, a 53K-parameter weight-shared transformer, to generate novel SMILES with 95% validity on the ZINC-250K drug-like-molecule benchmark, outperforming a standard GPT with 10 times more parameters. Mechanistically, the same block resolves SMILES constraints across passes in a fixed hierarchy: brackets first, rings second, and valence last, as shown by error classification and linear probing, with ablation isolating the bracket-matching head. Together, these results yield a compact, mechanistically interpretable molecular generator and a testbed for studying iterative computation in formal-language domains.