DELBERT-2: Pretrained Fingerprint Language Models for DEL Protein Binder Prediction

Pretrained molecular language models are increasingly used as molecular encoders for learning structure-property relationships. However, their practical suitability for molecular discovery within and beyond their pretraining domain remains unclear. Herein, we systematically benchmark four molecular language models across six virtual molecular libraries spanning drug discovery, organic materials, and catalysis. Native molecular language model embeddings show substantial variation in discovery performance across libraries, whereas molecular fingerprints provide a consistently strong and robust baseline. Consistent with a potential domain-representation mismatch, we show that explicit domain adaptation substantially improves representation performance. Fine-tuning molecular language model encoders on structures from the target virtual library consistently improves sample efficiency, with several adapted encoders emerging as the top-performing representations across the benchmark tasks. These results show that molecular representation quality depends strongly on the target domain and that explicit adaptation can improve the practical utility of molecular foundation models. More broadly, our findings establish domain-adapted molecular representations as a promising strategy for sample-efficient adaptive decision making in virtual screening and self-driving laboratories.

Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especially in low-compute regimes of relevance for resource-constrained research laboratories. We show that the training cost for encoder-only transformer architectures can be reduced by leveraging modern architectural refinements, offering an encouraging performance / compute-efficiency trade-off. We introduce ProtModernBERT, a modernized BERT-style model pre-trained on amino acid sequences, and compare it with ProtBert, Amplify, and ESM2 across a set of supervised structural tasks -encompassing secondary structure prediction, disorder prediction, and subcellular localization- and ProteinGym's zero-shot functional benchmark. We find that ProtModernBERT matches or closely tracks the performance of Amplify-120M and ProtBert with up to 80-90% reduction in pre-training FLOPs, and narrows the gap to ESM2 when controlling for pre-training. Our analysis using a cluster-based validation protocol for hyperparameter selection and randomly initialized models disentangles the contributions of architecture and data, showing that modern components provide stronger inductive biases that are more consequential than parameter count in low-compute regimes.

Protein language models produce rich per-residue representations, but the sequence embeddings obtained by mean-pooling them are never trained to place related proteins near one another, so the geometry that retrieval, clustering and nearest-neighbor transfer depend on is never optimized. We present ProtSent, a contrastive post-training framework that supervises a protein language model with three types of biological relation at once, Pfam family co-membership, AlphaFold DB structural clusters and STRING interactions, and with deep mutational scanning fitness ordering as an optional fourth. It needs no structure at training or inference, trains on corpora filtered at 40% sequence identity against the benchmark test sets they most resemble, and is applied to ESM-2 (35M and 150M parameters) and ESM-C (300M). Post-training makes structural family membership recoverable from the frozen embedding without a trained readout: on SCOPe-40, the adjusted Rand index of the 35M model rises from 0.054 (near chance) to 0.507, and family retrieval gains +0.185, +0.190 and +0.424 Recall@1 over the three frozen backbones, with paired bootstrap intervals excluding zero. On the CATH midnight-zone benchmark, homologous-superfamily accuracy rises from 40.7% to 56.7% at 35M and from 43.3% to 62.7% at 150M, more than the 12 points a CATH-supervised model gains over its own base. Against a maximally sensitive HMMER search, alignment keeps the best single hit at both ESM-2 scales, but the embedding leads at ranking depth (MAP 0.705 against 0.607 at 150M) and returns a ranking for every query, including those where alignment finds none. Under a trained linear probe, however, the same embeddings fall below the untuned backbone, as do structure-distilled embeddings against theirs, so the gain in neighborhood structure comes at a cost in linear-probe accuracy. Code and models: https://github.com/oriel9p/ProtSent