Beyond Random Splits: Evaluating Drug-Target Affinity Models Under Chemically and Biologically Motivated Distribution Shifts Copy

Minjae Chung; Clara Li; Malar Paavai Muthukumaran; Shaunna Wang; Aniket Ramkrishnan Iyer; Shaun Qien Yeau Tan; Harinishree Sathu; Micky C. Nnamdi

cs.LGOct 2, 2026

Beyond Random Splits: Evaluating Drug-Target Affinity Models Under Chemically and Biologically Motivated Distribution Shifts Copy

Authors: Minjae Chung, Clara Li, Malar Paavai Muthukumaran, Shaunna Wang, Aniket Ramkrishnan Iyer, Shaun Qien Yeau Tan, Harinishree Sathu, Micky C. Nnamdi, +3 more

Organizations: Georgia Institute of Technology

Abstract

Drug-target affinity (DTA) prediction is widely used to prioritize candidate compounds before costly experimental screening. DTA models are often compared under a single data split, even though deployment may require extrapolation to new chemical series, new protein targets, or both. We ask whether the distribution shift used for evaluation changes which architecture appears best. We curate 718,800 unique drug-protein pairs from the ChEMBL and BindingDB datasets. We compare a Morgan-fingerprint + protein-CNN baseline with 12 controlled architectures that combine four drug representations with three ESM-2 interaction modes. Mean validation RMSE increases from 0.950 and 0.945 under scaffold and fingerprint-cluster OOD to 1.299 and 1.321 under protein-cluster and dual OOD. Model rankings are similar across the two chemical shifts (tau = 0.79), but agreement with scaffold OOD falls under protein OOD (tau = 0.39) and reverses under dual OOD (tau = -0.55). Held-out evaluation, repeated seeds, group-aware bootstrap analysis, and a size-matched control support the same conclusion: architecture selection depends on the form of extrapolation, not only on average error or training-set size. DTA benchmarks should therefore match the chemical and target shifts expected at deployment.

Figures & tables

Figure 1: Benchmark design. The same architecture family is evaluated under four generalization regimes to independently analyze absolute error and model-selection stability.

Figure 2: Architecture-rank trajectories under RMSE. Architecture rankings are similar under scaffold and fingerprint OOD but shift substantially under protein and dual OOD. This figure visualizes the paper’s central model-selection result.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Distribution of $pK_{i}$ values in the curated ChEMBL and BindingDB dataset. A reference line at $pK_{i}=5$ may be included to show the corresponding $K_{i}=10\,\mu\mathrm{M}$ location without defining it as a universal activity threshold.

Regime pair	Spearman $\rho$
Scaffold vs. Fingerprint	0.895
Scaffold vs. Protein	0.448
Scaffold vs. Dual OOD	-0.727
Fingerprint vs. Protein	0.434
Fingerprint vs. Dual OOD	-0.769
Protein vs. Dual OOD	-0.476

Appendix

Table 1: Spearman rank correlations (

\rho

) between architecture rankings across evaluation regimes. Positive values indicate similar architecture orderings and negative values indicate inverted orderings.

Split	Validation RMSE	Held-out test RMSE
Scaffold	1.0355	1.1175
Fingerprint	1.0968	1.1802
Protein	1.2505	1.3359
Dual OOD	1.4700	1.4853

Appendix

Table 2: Morgan-fingerprint + protein-CNN baseline. Values are RMSE under the same split definitions and training protocol as the core matrix.

Model	Protein 511k	Protein 352k	Dual 352k
ECFP+AttnPool	1.2728	$\mathbf{1.2804\pm 0.0246}$	1.3652
GCN+CrossAttn	1.3595	$1.2917\pm 0.0315$	1.2728

Appendix

Table 3: Training-size ablation for ECFP+AttnPool and GCN+CrossAttn. Protein-matched values are mean

\pm

SD across five independent 352k training subsets. Values are RMSE.

Architecture	Drug encoder	Protein / fusion mode	Batch size	Trainable params
ECFP+MeanPool	ECFP4	MeanPool MLP	2048	1.04M
ChemBERTa+MeanPool	ChemBERTa	MeanPool MLP	2048	0.62M
ECFP+AttnPool	ECFP4	Dot-product AttnPool	128	1.17M
ECFP+CrossAttn	ECFP4	CrossAttn	128	2.09M
ChemBERTa+AttnPool	ChemBERTa	Dot-product AttnPool	128	0.74M
ChemBERTa+CrossAttn	ChemBERTa	CrossAttn	128	1.66M

Appendix

Table 4: Architecture-specific batch sizes and trainable parameter counts. Pretrained ESM-2 and ChemBERTa representations are frozen. Parameter counts refer only to trainable task-specific modules.

Architecture	Scaffold	Fingerprint	Protein	Dual OOD
ECFP + MeanPool	0.9523	0.9511	1.2977	1.3221
ChemBERTa + MeanPool	0.9633	0.9490	1.2858	1.3286
ECFP + AttnPool	0.8815	0.9034	1.2728	1.3652
ECFP + CrossAttn	0.9286	0.9441	1.2749	1.3604
ChemBERTa + AttnPool	0.9362	0.9250	1.2925	1.3112
ChemBERTa + CrossAttn	0.9192	0.9186	1.3091	1.3341

Appendix

Table 5: Validation RMSE across the architecture-by-split matrix. Lower is better. Bold indicates the best model within each regime; the final row gives the column mean across all 12 architectures.

Architecture	Scaffold	Fingerprint	Protein	Dual OOD
ECFP + MeanPool	0.9872	0.9615	1.3823	1.3599
ECFP + AttnPool	0.9234	0.8905	1.3713	1.4948
ECFP + CrossAttn	0.9738	0.9551	1.3703	1.4349
ChemBERTa + MeanPool	0.9969	0.9469	1.3360	1.4143
ChemBERTa + AttnPool	0.9622	0.9307	1.4080	1.3625
ChemBERTa + CrossAttn	0.9594	0.9228	1.3098	1.3883

Appendix

Table 6: Held-out test RMSE across the architecture-by-split matrix. Lower is better. Bold indicates the best model within each regime; the final row gives the column mean across all 12 architectures.

Explore similar work

LaPro-DTA: Latent Dual-View Drug Representations and Salient Protein Feature Extraction for Generalizable Drug--Target Affinity Prediction
Mar 16, 2026Zihan Dun, Liuyi Xu, An-Yang Lu +2Protein-Ligand Binding Affinity Prediction Molecular Representation Learning
College of Information Science and Engineering, Northeastern University, Shenyang 110819, China · School of Artificial Intelligence, Beihang University, Beijing 100000, China · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China
Chemical and geometric representation fidelity improves drug--target affinity prediction
Sep 2, 2026Yixiao Li, Yining Qian, Yefan Chen +5Protein-Ligand Binding Affinity Prediction Representation Learning
College of Information Science and Engineering, Northeastern University, Shenyang 110819, China · School of Computer and Communication Engineering, Northeastern University at Qinhuangdao, Qinhuangdao 066004, China · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China +1
Co-Diffusion: An Affinity-Aware Two-Stage Latent Diffusion Framework for Generalizable Drug-Target Affinity Prediction
Mar 11, 2026Yining Qian, Pengjie Wang, Yixiao Li +4Drug Discovery Latent Diffusion Models
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · College of Information Science and Engineering, Northeastern University, Shenyang 110819, China · Westlake University, Hangzhou 310000, China +2

Mar 16, 2026J/K move · Enter open · S save

LaPro-DTA: Latent Dual-View Drug Representations and Salient Protein Feature Extraction for Generalizable Drug--Target Affinity Prediction

Zihan Dun, Liuyi Xu, An-Yang Lu, Shuang Li+1

College of Information Science and Engineering, Northeastern University, Shenyang 110819, China · School of Artificial Intelligence, Beihang University, Beijing 100000, China · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China

Protein-Ligand Binding Affinity Prediction Molecular Representation Learning Protein Representation Learning

Open paper AskPDF arXiv

Drug--target affinity prediction is pivotal for accelerating drug discovery, yet existing methods suffer from significant performance degradation in realistic cold-start scenarios (unseen drugs/targets/pairs), primarily driven by overfitting to training instances and information loss from irrelevant target sequences. In this paper, we propose LaPro-DTA, a framework designed to achieve robust and generalizable DTA prediction. To tackle overfitting, we devise a latent dual-view drug representation mechanism. It synergizes an instance-level view to capture fine-grained substructures with stochastic perturbation and a distribution-level view to distill generalized chemical scaffolds via semantic remapping, thereby enforcing the model to learn transferable structural rules rather than memorizing specific samples. To mitigate information loss, we introduce a salient protein feature extraction strategy using pattern-aware top-

k

pooling, which effectively filters background noise and isolates high-response bioactive regions. Furthermore, a cross-view multi-head attention mechanism fuses these purified features to model comprehensive interactions. Extensive experiments on benchmark datasets demonstrate that LaPro-DTA significantly outperforms state-of-the-art methods, achieving an 8% MSE reduction on the Davis dataset in the challenging unseen-drug setting, while offering interpretable insights into binding mechanisms.

Abstract

Explore similar work

LaPro-DTA: Latent Dual-View Drug Representations and Salient Protein Feature Extraction for Generalizable Drug--Target Affinity Prediction

Chemical and geometric representation fidelity improves drug--target affinity prediction

Co-Diffusion: An Affinity-Aware Two-Stage Latent Diffusion Framework for Generalizable Drug-Target Affinity Prediction