cs.LGOct 13, 2026
SavePairIT: Autoregressive Transformers for Low-Data Molecule Optimization
Organizations: Genentech, South San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · Prescient Design, Genentech, Basel, Switzerland · Prescient Design, Genentech, New York, NY, USA
Abstract
Generating novel molecules with desired properties is a central challenge in drug discovery. Although foundational work focused on de novo generation, practical use cases require a more complex setting: conditioning on an initial seed molecule, optimizing multiple properties, and operating in data-sparse regimes. We introduce PairIT, a flexible training framework for autoregressive transformers over SMILES sequences that enables seeded generation by learning from pairs of structurally similar molecules. By leveraging matched pairs, the model learns to generate local, chemically meaningful edits that improve a property of the seed molecule. Prompt tokens extend this framework by specifying the target property, desired direction, and approximate magnitude of change. Multiple prompt tokens can be concatenated to enable multi-property optimization at inference time. For properties where pair supervision alone is insufficient, reinforcement learning post-training can further improve optimization, providing a tunable trade-off between proximity to the seed and the optimization objective. PairIT naturally supports pretraining on a large dataset and fine-tuning on a smaller dataset, which is critical for performance in data-sparse regimes. Surprisingly, we observe diminishing returns to the scale of the pretraining dataset. We demonstrate empirically on MoleculeNet and Therapeutics Data Commons datasets that, with appropriate paired-data curation, a standard transformer model of modest size (68M parameters) can achieve robust seeded molecular optimization in low-data regimes.
Figures & tables
Figure 1: Overview of PairIT. We (1) construct training pairs of structurally similar molecules with differing property values and (2) train an autoregressive transformer to predict the target SMILES given the seed SMILES and structured property-control prompt tokens. (3) At inference time, the model generates property-improved candidates from a new seed and prompt. Multiple prompt blocks can be concatenated for multi-property control.
Figure 2: MoleculeNet seeded bioactivity optimization. (a) Per-seed VIN (valid, novel, improved) distribution: each stacked bar gives the percentage of seeds for which of 5 retained generations are valid, novel, and improved. (b) Aggregate VIN–SNN (similarity to nearest neighbor in the target class) tradeoff across methods.
Figure 3: Example generations from PairIT and baselines. Leftmost column: the class-0 seed, rightmost column: the corresponding ground-truth class-1 MMP partner from the test set. Green = predicted class probability (success); red = seed duplicate.
Figure 4: Scale of ZINC-250K pretraining dataset vs. downstream fine-tuned BACE performance. Saturation before one full epoch of pretraining is visible.
Figure 5: QED controllability on MoleculeNet, with upward prompts over delta bins in . Error bars show standard deviation across 5 generations per seed per bin. Realized changes scale with prompted magnitude, indicating coarse magnitude control. The corresponding logP results are in Appendix G.2 (Figure 37 ).
| Setting | Dataset | Combined VIN (%) | Per-property VIN (%) | Random (%) | |
| Low-correlation pair | BACE | 12 | 13 / 16 | 17 | -4 |
| BBBP | 32 | 38 / 35 | 21 | +11 | |
| HIV | 64 | 70 / 76 | 32 | +32 | |
| QED + logP | BACE | 19 | 22 / 21 | 15 | +3 |
| BBBP | 30 | 33 / 43 | 17 | +13 | |
| HIV | 42 | 55 / 63 | 30 | +12 |
Table 1: Two-property controllability on MoleculeNet, using 64 raw generations per held-out matched pair without top-five selection. Per-property VIN : is the fraction of all attempts with an applicable nonzero request that are valid, novel, and satisfy that property’s criterion. Combined VIN : fraction of generations that are valid, novel, and move in the requested direction on both prompted properties; zero-delta properties impose no condition, only pairs with at least one non-zero change requested are counted. For bioactivity, success means an inactive seed yields a generation with train-only random-forest activity probability greater than 0.5. Random : independently resample each property’s signed-bin index uniformly from 00–15 while retaining property identities and block order. Both arms are scored against the original requests to serve as a baseline. is prompted minus randomized Combined VIN in percentage points, positive in bold.
| Method | BACE | BBBP | HIV |
| PairIT (10-property prompt) | 0.773 | 0.739 | 0.739 |
| Randomized prompt (control) | 0.433 | 0.499 | 0.559 |
Table 2: Direction adherence under multi-property prompting: fraction of generations whose property delta matches the prompted direction, averaged across the 9 non-activity properties. Higher is better.
Appendix figures & tables51 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Seeded | Single Step | Multi- Property | Small Datasets | Property Model-Free | Optimization Mechanism |
| GraphDiT [ Liu et al., 2024 ] | AdaLN | |||||
| PropEn [ Tagasovska et al., 2024 ] | Pairs | |||||
| LLOME [ Chen et al., 2025 ] | LLM + online black-box optimization | |||||
| NF-BO [ Lee et al., 2025 ] | Bayesian optimization | |||||
| HierG2G [ Jin et al., 2020 ] | Graph translation | |||||
| DDSBM [ Kim et al., 2025 ] | Schrödinger bridge |
Table 3: Comparison of molecular generation and optimization methods.
| Setting | Value |
| Layers | 12 |
| Hidden size | 648 |
| Attention heads | 12 |
| KV heads | 12 (MHA) |
| FFN size | 2,048 |
| Vocabulary size | 1,225 |
Table 4: PairIT architecture and training hyperparameters for the 68M model used in the main experiments.
Figure 6: We observe negligible gains from scaling the model size; larger models show increased memorization, indicating overfitting without performance benefits.
Figure 7: Inference-time temperature sweep ( – ) showing the trade-off between diversity and validity: lower temperatures favor similarity to the seed, while higher temperatures increase novelty and property improvement at the cost of validity.
Figure 8: Examples of generations with different temperature settings.
| BACE | BBBP | HIV | hERG | |
| Train/Test | Train/Test | Train/Test | Train/Test | |
| Molecules | 798 / 267 | 517 / 175 | 1,420 / 475 | 414 / 111 |
| Pairs | 2,546 / 217 | 512 / 40 | 1,009 / 165 | 2,865 / 764 |
| Usable pairs | 662 / 51 | 88 / 14 | 256 / 46 | 76 / – |
| Test seeds | – / 135 | – / 88 | – / 231 | – / 85 |
Table 5: Dataset statistics for fine-tuning and evaluation. Usable pairs are direction-improving transformations retained after filtering and used for fine-tuning. BACE, BBBP, and HIV use matched molecular pairs. For hERG, MMPs are too sparse (41 direction-improving training pairs), so we use Tanimoto-neighbor pairs, yielding 76 usable pairs after filtering. Test seeds are class-0 molecules for MoleculeNet and class-1 molecules (blockers) for hERG
| Metric | BACE | BBBP | HIV | hERG | |
| Benchmark test | molecules | 261 | 172 | 475 | 111 |
| Pearson | 0.791 | 0.819 | 0.576 | 0.783 | |
| Spearman | 0.787 | 0.825 | 0.616 | 0.685 | |
| Classification agreement (%) | 83.5 | 84.9 | 69.7 | 87.6 | |
| Generated | molecules | 1618 | 1389 | 3751 | 1353 |
| Pearson | 0.438 | 0.631 | 0.290 | 0.595 |
Table 6: Agreement between the random-forest and MiniMol property oracles. Classification agreement is the fraction of molecules assigned the same active/inactive label by both models.
Figure 9: Aggregate MoleculeNet seeded-generation metrics corresponding to Figure 2 . Bars show mean performance across test seeds after within-seed de-duplication and final top-5 selection; error bars are bootstrap 95% confidence intervals over seeds. VIN is the fraction of retained generations that are valid, novel, and improved. Similarity to nearest neighbor (SNN) is computed over novel and valid generations only. QED is the quantitative estimate of drug-likeness.
| Model | VIN | QED | Valid | Novel | Unique | Diverse | Improved | SNN |
| MMP | 43.56 | 0.52 | 100.00 | 99.11 | 100.00 | 38.00 | 44.30 | 0.50 |
| GraphDiT | – | – | 87.32 | 100.00 | 11.28 | – | 86.60 | – |
| DDSBM-Bridge | 32.74 | 0.58 | 100.00 | 99.85 | 100.00 | 79.00 | 32.89 | 0.35 |
| DDSBM-SB | 36.59 | 0.55 | 92.89 | 97.77 | 92.15 | 46.00 | 41.16 | 0.44 |
| HierG2G | 30.37 | 0.50 | 79.85 | 93.69 | 74.81 | 62.00 | 40.59 | 0.37 |
| NF-BO | 32.15 | 0.51 | 100.00 | 100.00 | 100.00 | 74.00 | 32.15 | 0.29 |
Table 7: MoleculeNet BACE bioactivity improvement ( ) under seeded generation. Best 5 of 64 per seed selected after within-seed canonical-SMILES de-duplication, so each retained molecule is distinct within its seed. Per-seed bootstrap 95% CIs for VIN and SNN are shown in Fig. 9 . SNN is the mean per-generation Tanimoto to the nearest class-1 training molecule, computed over novel valid generations only.
| Model | VIN | QED | Valid | Novel | Unique | Diverse | Improved | SNN |
| MMP | 26.59 | 0.44 | 99.55 | 100.00 | 99.55 | 43.00 | 26.71 | 0.29 |
| GraphDiT | – | – | 83.93 | 100.00 | 20.33 | – | 82.30 | – |
| DDSBM-Bridge | 45.91 | 0.51 | 91.14 | 100.00 | 91.14 | 88.00 | 50.37 | 0.21 |
| DDSBM-SB | 33.41 | 0.55 | 88.86 | 95.91 | 85.91 | 57.00 | 39.68 | 0.28 |
| HierG2G | 0.00 | 0.47 | 20.00 | 0.00 | 0.00 | – | 0.00 | – |
| NF-BO | 68.86 | 0.65 | 90.45 | 98.74 | 89.32 | 70.00 | 77.10 | 0.29 |
Table 8: MoleculeNet BBBP bioactivity improvement ( ) under seeded generation. Best 5 of 64 per seed selected after within-seed canonical-SMILES de-duplication, so each retained molecule is distinct within its seed. Per-seed bootstrap 95% CIs for VIN and SNN are shown in Fig. 9 . SNN is the mean per-generation Tanimoto to the nearest class-1 training molecule, computed over novel valid generations only.
| Model | VIN | QED | Valid | Novel | Unique | Diverse | Improved | SNN |
| MMP | 40.43 | 0.47 | 92.03 | 99.91 | 91.95 | 41.00 | 43.97 | 0.29 |
| GraphDiT | – | – | 80.16 | 100.00 | 16.55 | – | 95.55 | – |
| DDSBM-Bridge | 40.95 | 0.54 | 97.66 | 99.91 | 97.66 | 85.00 | 42.02 | 0.23 |
| DDSBM-SB | 33.94 | 0.56 | 87.53 | 97.03 | 86.15 | 61.00 | 40.70 | 0.29 |
| HierG2G | 0.69 | 0.53 | 21.04 | 7.00 | 1.47 | 71.00 | 47.06 | 0.24 |
| NF-BO | 37.40 | 0.57 | 87.01 | 98.31 | 85.54 | 70.00 | 43.72 | 0.27 |
Table 9: MoleculeNet HIV bioactivity improvement ( ) under seeded generation. Best 5 of 64 per seed selected after within-seed canonical-SMILES de-duplication, so each retained molecule is distinct within its seed. Per-seed bootstrap 95% CIs for VIN and SNN are shown in Fig. 9 . SNN is the mean per-generation Tanimoto to the nearest class-1 training molecule, computed over novel valid generations only.
Figure 10: Scatter plots of VIN vs. SimSeed showing the distribution of all generations for all seeds on BACE.
Figure 11: Scatter plots of VIN vs. SimSeed showing the distribution of all generations for all seeds on BBBP. HierG2G generations are excluded because they all contain duplicates of the seed.
Figure 12: Scatter plots of VIN vs. SimSeed showing the distribution of all generations for all seeds on HIV. The majority of HierG2G generations contain duplicates of the seed.
Figure 13: Failure-mode breakdown on MoleculeNet BACE. Bars show VIN success in green and the sources of error for generations not counted as VIN.
Figure 14: Failure-mode breakdown on MoleculeNet BBBP. Bars show VIN success in green and the sources of error for generations not counted as VIN.
Figure 15: Failure-mode breakdown on MoleculeNet HIV. Bars show VIN success in green and the sources of error for generations not counted as VIN.
Figure 16: Additional example generations for MoleculeNet seeded optimization. Generations with predicted class probability greater than 0.5 are highlighted in green and seed duplicates in red.
| Method | Valid % | Improved % | Novel % | VIN % | SimSeed |
| MMP | 91.76 | 10.77 | 100.00 | 9.88 | 0.74 |
| DDSBM-Bridge | 100.00 | 19.76 | 100.00 | 19.76 | 0.30 |
| DDSBM-SB | 93.88 | 87.91 | 97.74 | 80.71 | 0.19 |
| HierG2G | 99.06 | 81.67 | 99.76 | 80.71 | 0.14 |
| NF-BO | 98.59 | 22.30 | 99.52 | 21.88 | 0.47 |
| GeLLM4O-C | 100.00 | 29.18 | 100.00 | 29.18 | 0.60 |
Table 13: Seeded molecular optimization on hERG. SimSeed measures Tanimoto similarity to the input seed among VIN generations.
| Method | Valid % | Improved % | VIN % | SimSeed | Novel % |
| No pretraining | 40.9 | 25.3 | 20.0 | 0.42 | 36.3 |
| Single-molecule (uniform) | 75.2 | 45.5 | 40.0 | 0.20 | 74.8 |
| Single-molecule ( seeds) | 69.3 | 37.0 | 44.0 | 0.21 | 70.0 |
| Tanimoto (1 neighbor) | 87.7 | 44.6 | 28.0 | 0.49 | 63.5 |
| Tanimoto (2 neighbor) | 89.2 | 42.9 | 45.0 | 0.52 | 71.0 |
| Tanimoto (3 neighbor) | 89.0 | 39.6 | 32.0 | 0.52 | 73.2 |
Table 14: Downstream BACE molecule-improvement with varying ZINC pretraining objectives. Valid, Improved, and Novel are computed over all 64 raw generations per seed, while VIN and SimSeed are computed over the top-5 selected candidates per seed.
Figure 17: Effect of ZINC-250K pretraining on PairIT. Top row: VIN; bottom row: SNN computed over VIN molecules. The no-pretraining model uses the same architecture and MoleculeNet directed pairs as PairIT, but is trained from random initialization.
Figure 18: Failure-mode breakdown for the pretraining ablation. Without ZINC-250K pretraining, PairIT is dominated by invalid SMILES; pretrained PairIT variants rarely fail validity.
Figure 19: Variance in overfitting behavior, and the number of unique molecules seen at the time of overfitting, for different ZINC-based data distributions (pairs with and without property-based filtering, and unconditional single-molecule generation). Dotted lines indicate the point of early stopping, i.e. minimal validation loss.
Figure 20: ZINC pretraining validation loss is well-correlated with chemical metrics, specifically its own generation rate of valid, logP-improved molecules, and novel molecules (VIN), justifying our early stopping criterion based on validation loss.
Figure 21: ZINC pretraining validation loss is well-correlated with downstream fine-tuned performance on BACE, further justifying our early stopping criterion based on validation loss. Here, each point is a separate checkpoint, pretrained on ZINC pair datasets of varying sizes and fine-tuned on the full BACE dataset.
Figure 22: Filtering with more aggressive property cutoffs results in corresponding learned behavior. (This is in the basic setting without any prompting.)
Figure 23: ZINC-250K logP optimization. Bars report joint valid–novel–improved success (V,N,I), validity, novelty, improvement, and SimSeed. PairIT variants sweep the pair-construction threshold ; larger corresponds to training on pairs with larger logP increases. Error bars are bootstrap 95% confidence intervals over seeds.
Figure 24: A model pretrained to optimize logP on ZINC transfers zero-shot to logP optimization on MoleculeNet. We track this ability over the course of training, and see that it correlates well with the pretraining validation loss.
Figure 25: BACE single-property example, seed 1 of 2. Top row: seed and ground-truth improved partner from the test pair. Each subsequent row: top-5 deduplicated generations for one method. Box color: green = V N I (valid novel improved); orange = V I but training-memorized; red = V I but seed-copy.
Figure 26: BACE single-property example, seed 2 of 2. Top row: seed and ground-truth improved partner from the test pair. Each subsequent row: top-5 deduplicated generations for one method. Box color: green = V N I (valid novel improved); orange = V I but training-memorized; red = V I but seed-copy.
Figure 27: BBBP single-property example, seed 1 of 2.
Figure 28: BBBP single-property example, seed 2 of 2.
Figure 29: HIV single-property example, seed 1 of 2.
Figure 30: HIV single-property example, seed 2 of 2.
Figure 31: BACE multi-property example, seed 1 of 2. Methods compared: SFT, RL sim=1 , RL sim=3 , RL sim=5,gate=0.2 , RL sim=10 .
Figure 32: BACE multi-property example, seed 2 of 2. Methods compared: SFT, RL sim=1 , RL sim=3 , RL sim=5,gate=0.2 , RL sim=10 .
Figure 33: BBBP multi-property example, seed 1 of 2.
Figure 34: BBBP multi-property example, seed 2 of 2.
Figure 35: HIV multi-property example, seed 1 of 2.
Figure 36: HIV multi-property example, seed 2 of 2.
| Property | Dataset | VIN% | MSE |
| QED ( only) | BACE | 52 | 0.041 |
| BBBP | 80 | 0.034 | |
| HIV | 77 | 0.034 | |
| logP (bidirectional) | BACE | 61 | 1.82 |
| BBBP | 69 | 4.09 | |
| HIV | 70 | 2.34 |
Table 15: Single-property controllability on MoleculeNet after fine-tuning. VIN is the fraction of valid, novel generations whose property moves in the prompted direction. MSE is computed relative to the midpoint of the prompted delta bin.
Figure 37: Single-property controllability on MoleculeNet for both QED and logP. Top: QED, upward prompts over . Bottom: logP, bidirectional prompts over . Error bars show standard deviation across 5 generations per seed per bin. Realized changes scale with prompted magnitude in both cases, indicating coarse magnitude control. The QED-only version of the top row appears in the main body as Figure 5 .
Figure 38: Prompted versus realized absolute property change under multi-property conditioning for four selected low-correlation property pairs from ZINC-250K. Each panel corresponds to one property within a pair and plots the prompted absolute delta on the x-axis against the realized absolute delta of the generated molecule on the y-axis; the dashed diagonal indicates perfect agreement. Across all pairs, larger requested edits produce larger realized changes, showing that approximate magnitude control is retained for both properties under simultaneous conditioning, although dispersion increases for large requested deltas and bounded properties such as QED show some saturation.
Figure 39: Selected low-correlation two-property control on MoleculeNet. Each column is one dataset; the top and bottom rows show the two prompted properties. Column titles report combined VIN% (valid, novel outputs satisfying every nonzero requested direction, divided by all attempts with at least one nonzero request). The model achieves 12–64% combined VIN, demonstrating independent control of selected property pairs with low empirical correlation.
| Dataset | Props | Combined VIN% | Prop1 VIN% | Prop2 VIN% |
| BACE | MW+HBD | 12 | 13 | 16 |
| BBBP | TPSA+logP | 32 | 38 | 35 |
| HIV | HBA+logP | 64 | 70 | 76 |
Table 16: Two-property controllability for selected low-correlation property pairs on MoleculeNet, using 64 raw generations per matched test pair without top-five selection. Per-property VIN divides valid, novel, direction-matching outputs by all attempts with a nonzero requested change for that property. Combined VIN divides valid, novel outputs satisfying every nonzero requested direction by all attempts with at least one nonzero request. Invalid and non-novel attempts remain denominator failures; zero-delta properties impose no condition.
Figure 40: Joint QED+logP control, with both properties prompted using natural signed deltas from matched test pairs. The model achieves 19–42% combined VIN (valid, novel outputs satisfying every nonzero requested direction, divided by all attempts with at least one nonzero request). QED+logP is a harder joint-control setting than uncorrelated descriptor pairs.
| Dataset | Combined VIN% | QED VIN% | logP VIN% |
| BACE | 19 | 22 | 21 |
| BBBP | 30 | 33 | 43 |
| HIV | 42 | 55 | 63 |
Table 17: Two-property controllability for joint QED and logP control on MoleculeNet, using signed-bin requests from held-out matched pairs and 64 raw generations per pair without top-five selection. Per-property VIN requires validity, novelty, and the correct direction, using all attempts with a nonzero requested change for that property as its denominator. Combined VIN uses all attempts with at least one nonzero request and requires validity, novelty, and agreement with every nonzero requested direction. Zero-delta properties impose no condition.
| Dataset | VIN% | Valid% | Novel% | Active% | Property-model Up% | SimSeed |
| BACE | 21.7 | 95.8 | 40.9 | 53.0 | 60.4 | 0.491 |
| BBBP | 83.0 | 99.8 | 91.6 | 90.6 | 72.2 | 0.290 |
| HIV | 12.3 | 92.3 | 89.4 | 13.8 | 47.9 | 0.162 |
Table 18: Full multi-property prompted model evaluated with all 10 properties on MoleculeNet test pairs. Each property is prompted with the signed delta bin corresponding to the ground-truth pair. Active% is the fraction of valid, novel generations predicted active (class 1) by the property model; property model Up% is the fraction for which the property model score moves in the correct direction relative to the seed. SimSeed is the mean Tanimoto similarity to the input seed.
| Dataset | VIN% | Valid% | Novel% | Active% | property model Up% | SimSeed |
| BACE | 25.5 | 95.8 | 61.3 | 41.4 | 67.1 | 0.401 |
| BBBP | 79.3 | 90.0 | 90.0 | 88.1 | 95.2 | 0.818 |
| HIV | 19.8 | 84.6 | 84.6 | 23.4 | 75.5 | 0.841 |
Table 19: Full multi-property prompted model evaluated with the target bioactivity property plus logP, SA, and QED (three-property subset) on MoleculeNet test pairs.
| Dataset | Mean VIN% | Best VIN% | Worst VIN% | Spread (pp) |
| BACE | 30.4 | 21.1 | 9.3 | |
| BBBP | 85.0 | 57.1 | 27.9 | |
| HIV | 25.0 | 5.5 | 19.5 |
Table 20: Property-subset sensitivity for the full multi-property prompted model. A single trained model is evaluated across all combinations of three RDKit properties prompted alongside the target bioactivity property (84 evaluation runs per dataset, not 84 trained models).
| Property | BACE | BBBP | HIV |
| logP | +0.022 | -0.037 | +0.709 |
| QED | -0.032 | +0.018 | +0.033 |
| SA | +0.126 | -0.132 | -0.595 |
| MW | +17.78 | +3.11 | +18.56 |
| TPSA | +6.20 | -7.09 | -15.51 |
| HBD | -0.041 | -0.354 | -0.719 |
Table 21: Mean property delta (generated minus seed) for the full multi-property prompted model under 10-property prompting.