DELBERT-2: Pretrained Fingerprint Language Models for DEL Protein Binder Prediction

Bing Hu; Sun Sun; Shaik Salman Basha; Anita Layton; Helen Chen

cs.LGOct 30, 2026

DELBERT-2: Pretrained Fingerprint Language Models for DEL Protein Binder Prediction

Authors: Bing Hu, Sun Sun, Shaik Salman Basha, Anita Layton, Helen Chen

Organizations: Computer Science, University of Waterloo, Canada · National Research Council of Canada, Canada · Applied Mathematics, University of Waterloo, Canada · School of Public Health, University of Waterloo, Canada

Abstract

We present DELBERT-2, a pretrained fingerprint language model that converts sparse molecular fingerprints into unified token sequences using a ModernBERT encoder trained with masked language modelling on 2.5M molecules from the AIRCHECK DEL corpus. We evaluate across six targets (WDR91, WDR12, SETDB1, PLCZ1, LRRK2, DCAF7) under three out-of-distribution (OOD) protocols: hierarchical cluster splits probe chemical novelty, library splits test cross-library transfer, and building-block splits assess compositional generalization. DELBERT-2 consistently improves PR-AUC and NDCG@1000 relative to LightGBM ensemble baselines and transformers trained from scratch, with the largest gains in stringent OOD regimes. DELBERT-2 achieves 13.28+/-3.72 enrichment factor at top-100 vs. 12.36+/-3.75 for no-pretraining (p=0.040), representing 7.4% improvement and 13x enrichment over random selection. These results demonstrate that fingerprint-centric self-supervised learning effectively improves hit prioritization under distribution shift, enabling practical DEL virtual screening. Code is available at \hyperlink{https://github.com/GenerativeDrugDiscovery/DELBERT-2}{DELBERT-2}.

Figures & tables

Figure 1: DELBERT-2 pre-training and fine-tuning workflow. Multiple sparse fingerprints are tokenized into a unified sequence: each active bit at position pos in fingerprint type fp becomes a discrete token fp_pos_val , transforming sparse binary vectors into variable-length token sequences. The model is pre-trained with masked language modeling (MLM) on 2.5M DEL molecules, then fine-tuned with LoRA for target-specific binder prediction.

Figure 2: Out-of-distribution (OOD) evaluation protocols for DEL virtual screening. Left: Hierarchical clustering split constructs nested ID/OOD partitions at multiple Tanimoto similarity ( Tanimoto, 1958 ) cutoffs (0.65/0.75/0.85); higher cutoffs create tighter clusters, making OOD test molecules more chemically distant from training. Middle: Building-block split holds out 40% of building blocks (BBs) and stratifies test molecules by BB overlap with training ( 0bb / 1bb / 2bb ), directly probing compositional generalization. Right: Library split holds out entire DEL libraries, testing transfer to unseen synthesis designs (UMAP colored by library; $n$ = number of molecules per library).

Panel of Figure 3 — Figure 3: Hierarchy split performance vs. OOD severity (Tanimoto similarity cutoff). (a) Per-target Test PR-AUC degrades as the cutoff increases (0.65 $\rightarrow$ 0.85), reflecting increasing chemical novelty. DELBERT-2 consistently outperforms LGBM and No-PT across all targets. (b) Mean $\pm$ std across 6 targets: DELBERT-2 maintains higher performance at all severity levels, with the largest advantage at the strictest cutoff (0.85), demonstrating that unified pre-training improves robustness to chemical novelty.

		PR-AUC
		Cutoff 0.65		Cutoff 0.75		Cutoff 0.85
Data	Model	ID	OOD	ID	OOD	ID	OOD
W91	LGBM	0.768	0.532	0.775	0.586	0.757	0.575
	NPT	0.827	0.684	0.828	0.685	0.784	0.588
	DBT-2	0.881	0.737	0.878	0.735	0.865	0.676
W12	LGBM	0.689	0.601	0.713	0.606	0.681	0.564

Table 1: Hierarchy split results. Left: PR-AUC. Right: NDCG@1000. Higher is better. OOD severity increases with a higher clustering cutoff. DELBERT-2 consistently outperforms LGBM and NPT across targets, with the largest gains in stricter OOD regimes.

Figure 4: Cross-library transfer performance. Left: Per-target breakdown showing DELBERT-2 consistently above LGBM and No-PT across exclusion fractions. Right: Mean $\pm$ std Test PR-AUC across 4 targets (WDR91, WDR12, SETDB1, LRRK2) as the held-out library fraction increases (0.1 $\rightarrow$ 0.3). DELBERT-2 maintains more consistent performance than baselines, demonstrating better cross-library transfer from unified pre-training.

		PR-AUC
		Split 0.10		Split 0.20		Split 0.30
Data	Model	ID	OOD	ID	OOD	ID	OOD
LK2	LGBM	0.719	0.623	0.723	0.702	0.721	0.652
	NPT	0.807	0.565	0.810	0.754	0.814	0.614
	DBT-2	0.874	0.721	0.855	0.776	0.856	0.745
ST1	LGBM	0.896	0.889	0.896	0.893	0.894	0.872

Table 2: Library split results. Left: PR-AUC. Right: NDCG@1000. Split

s\in\{0.10,0.20,0.30\}

denotes the approximate fraction of libraries held out. DELBERT-2 consistently improves over baselines, demonstrating robust cross-library transfer.

Figure 5: Enrichment analysis across targets. Top row: enrichment factors for DCAF7, LRRK2, PLCZ1, and mean across all 6 targets. Bottom row: SETDB1, WDR12, WDR91, and mean Precision@K. DELBERT-2 consistently achieves higher enrichment than baselines, with mean EF@100 of 13.28 $\pm$ 3.72 vs. 12.36 $\pm$ 3.75 (No-PT) and 12.69 $\pm$ 3.11 (LGBM), representing 7.4% improvement over No-PT ( $p{=}0.040$ ). All three models are evaluated at K $\in$ {10, 20, 50, 100, 200, 500, 1000}.

		Validation			2BB Test			1BB Test			0BB Test
Target	Metric	LGBM	NPT	DBT-2	LGBM	NPT	DBT-2	LGBM	NPT	DBT-2	LGBM	NPT	DBT-2
WDR91	PR-AUC	0.875	0.874	0.875	0.538	0.518	0.542	0.298	0.207	0.295	0.174	0.124	0.174
WDR91	Relative	100%	100%	100%	61%	59%	62%	34%	24%	34%	20%	14%	20%
WDR12	PR-AUC	0.731	0.798	0.735	0.481	0.350	0.461	0.230	0.159	0.211	0.161	0.145	0.142
WDR12	Relative	100%	100%	100%	66%	44%	63%	32%	20%	29%	22%	18%	19%
SETDB1	PR-AUC	0.943	0.969	0.932	0.604	0.655	0.601	0.293	0.344	0.309	0.218	0.187	0.228

Table 3: Building-block split results. 40% of building blocks are held out; test molecules are bucketed by overlap: 2BB (both BBs seen), 1BB (one seen), 0BB (neither seen—strictest). Performance degrades as overlap decreases. DELBERT-2 often improves in stricter buckets, suggesting pre-training reduces building-block memorization. Relative: performance as % of ID validation.

	Validation			2BB Test			1BB Test			0BB Test
Target	LGBM	NPT	DBT-2	LGBM	NPT	DBT-2	LGBM	NPT	DBT-2	LGBM	NPT	DBT-2
WDR91	0.875	0.874	0.875	0.538	0.518	0.542	0.298	0.207	0.295	0.174	0.124	0.174
WDR12	0.731	0.798	0.735	0.481	0.350	0.461	0.230	0.159	0.211	0.161	0.145	0.142
SETDB1	0.943	0.969	0.932	0.604	0.655	0.601	0.293	0.344	0.309	0.218	0.187	0.228
PLCZ1	0.772	0.704	0.776	0.706	0.629	0.699	0.624	0.580	0.611	0.547	0.540	0.515
LRRK2	0.846	0.878	0.836	0.636	0.636	0.610	0.418	0.351	0.390	0.223	0.147	0.220

Table 4: Building-block split results. 40% of building blocks are held out; test molecules are bucketed by overlap: 2BB (both BBs seen), 1BB (one seen), 0BB (neither seen—strictest). Performance degrades as overlap decreases. DELBERT-2 often improves in stricter buckets, suggesting pre-training reduces building-block memorization. Relative: performance as % of ID validation.

Panel of Figure 6 — Figure 6: Building-block compositional generalization analysis. (a) Test PR-AUC degrades as building-block overlap decreases (Val $\rightarrow$ 2BB $\rightarrow$ 1BB $\rightarrow$ 0BB), with DELBERT-2 showing better retention than baselines in stricter regimes. Lines show mean $\pm$ std across 6 targets. (b) Relative performance normalized to validation baseline. DELBERT-2 maintains higher retention in 0BB (20%) and 1BB (34%) buckets compared to NPT (14% and 24%), demonstrating that pretraining reduces building-block memorization.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
LoRA rank	8
LoRA $\alpha$	16
LoRA dropout	0.1
LoRA target module	Wqkv (attention)
Focal loss $\gamma$	2
Optimizer	AdamW

Appendix

Table 5: Fine-tuning hyperparameters for DELBERT-2.

Component	DELBERT	DELBERT-2
Backbone	ModernBERT	ModernBERT
Parameters	$\sim$ 70M	$\sim$ 100M
Hidden size	640	1024
Layers	16	12
Attention heads	16	16
Feed-forward size	1,152	1,152

Appendix

Table 6: Architectural and training comparison between DELBERT ( Seyed-Ahmadi et al., 2026 ) and DELBERT-2. Both use ModernBERT with Flash Attention and RoPE. Key improvements in DELBERT-2 include larger model capacity, expanded fingerprint coverage, unified pre-training on 5

\times

more molecules, and comprehensive OOD evaluation.

	DELBERT OOD PR-AUC			DELBERT-2 OOD PR-AUC
	full K-fold CV			Split 0.10		Split 0.20		Split 0.30
Target	$K$	Binary Baselines	DBT	LGBM	DBT-2	LGBM	DBT-2	LGBM	DBT-2
WDR91	20	0.219 $\pm$ 0.150	0.349 $\pm$ 0.162	0.557	0.650	0.639	0.770	0.580	0.681
LRRK2	15	0.198 $\pm$ 0.153	0.336 $\pm$ 0.215	0.623	0.721	0.702	0.776	0.652	0.745
SETDB1	11	0.240 $\pm$ 0.173	0.585 $\pm$ 0.250	0.889	0.887	0.893	0.972	0.872	0.938
DCAF7	18	0.430 $\pm$ 0.183	0.405 $\pm$ 0.192	-	-	-	-	-	-

Appendix

Table 7: Library split PR-AUC comparison between DELBERT ( Seyed-Ahmadi et al., 2026 ) and DELBERT-2 on the four overlapping targets. DELBERT values are mean

\pm

std across full K-fold library CV (all libraries cycled as test);

K

= number of CV folds per target. DELBERT-2 values are OOD test PR-AUC at three library exclusion levels (10%, 20%, 30%). Note: Baseline models differ: DELBERT uses XGBoost+LightGBM ensembles on binarized FPs (20 models/fold); DELBERT-2 uses LightGBM ensembles on count-based FPs (5 models/fold). DCAF7 is not available in DELBERT-2’s library split evaluation.

Hyperparameter	Value
MLM masking probability	15%
Batch size (per GPU)	8
Gradient accumulation steps	8 (effective batch = 64)
Learning rate	$1\times 10^{-4}$
Optimizer	AdamW
Weight decay	0.01

Appendix

Table 8: Pre-training hyperparameters for DELBERT-2.

Drug Discovery

Drug discovery identifies, designs, and tests molecules that can safely affect disease biology.

See all 173 papers

Domain-Adapted Molecular Language Models for Efficient Search of Make-on-Demand Libraries
Aug 18, 2026Henrik Wille, Luis-Finley Schütz, Felix Strieth-KalthoffMolecular Representation Learning Molecular Foundation Models
University of Wuppertal, School of Mathematics and Natural Sciences, Gaußstr. 20, 42119 Wuppertal, Germany · University of Wuppertal, Interdisciplinary Center of Machine Learning and Data Analytics, Gaußstr. 20, 42119 Wuppertal, Germany
ProtModernBERT: Compute-Efficient Protein Language Modeling
Oct 14, 2026Alberto Martínez, Natalia Flechas Manrique, Llorenç Espinosa-Portalés +1Protein Language Models Efficient Language Model Training
Multiverse Computing, Parque Científico y Tecnológico de Gipuzkoa, Paseo de Miramón 170, 20014 Donostia-San Sebastián, Spain. · Multiverse Computing, 7 rue de la Croix Martre, 91120 Palaiseau, Paris, France
ProtSent: Protein Sentence Transformers
May 7, 2026Dan Ofer, Oriel Perets, Michal Linial +1Contrastive Learning Protein Language Models
Department of Biological Chemistry The Hebrew University of Jerusalem · Department of Computer and Information Science Ben-Gurion University of the Negev

Aug 18, 2026J/K move · Enter open · S save

Domain-Adapted Molecular Language Models for Efficient Search of Make-on-Demand Libraries

Henrik Wille, Luis-Finley Schütz, Felix Strieth-Kalthoff

University of Wuppertal, School of Mathematics and Natural Sciences, Gaußstr. 20, 42119 Wuppertal, Germany · University of Wuppertal, Interdisciplinary Center of Machine Learning and Data Analytics, Gaußstr. 20, 42119 Wuppertal, Germany

Molecular Representation Learning Molecular Foundation Models Virtual Screening Language Modeling Molecular Property Prediction

Open paper AskPDF arXiv

Pretrained molecular language models are increasingly used as molecular encoders for learning structure-property relationships. However, their practical suitability for molecular discovery within and beyond their pretraining domain remains unclear. Herein, we systematically benchmark four molecular language models across six virtual molecular libraries spanning drug discovery, organic materials, and catalysis. Native molecular language model embeddings show substantial variation in discovery performance across libraries, whereas molecular fingerprints provide a consistently strong and robust baseline. Consistent with a potential domain-representation mismatch, we show that explicit domain adaptation substantially improves representation performance. Fine-tuning molecular language model encoders on structures from the target virtual library consistently improves sample efficiency, with several adapted encoders emerging as the top-performing representations across the benchmark tasks. These results show that molecular representation quality depends strongly on the target domain and that explicit adaptation can improve the practical utility of molecular foundation models. More broadly, our findings establish domain-adapted molecular representations as a promising strategy for sample-efficient adaptive decision making in virtual screening and self-driving laboratories.

Abstract

Drug Discovery

Explore similar work

Domain-Adapted Molecular Language Models for Efficient Search of Make-on-Demand Libraries

ProtModernBERT: Compute-Efficient Protein Language Modeling

ProtSent: Protein Sentence Transformers