Benchmarking Machine-Learning Classifiers for Neoantigen Immunogenicity Prediction and the Limits of SHAP Interpretability
Organizations: Mentomy AI
Abstract
Personalized cancer vaccines require prioritizing, among hundreds of candidate mutant peptides, the few that will induce a T-cell response. This study asks what general-purpose machine learning achieves on that task, and whether the features such models identify as important can be trusted. We make three contributions. The first is a benchmark of sixty models: twelve supervised classifier families, each in five configurations, evaluated on the Immunogenic Tumor Specific Neoantigen database (ITSNdb; 199 experimentally validated neoantigens, 129 immunogenic and 70 non-immunogenic). All sixty are compared under one stratified k-fold cross-validation protocol, in which the data used to choose a configuration are never the data used to score it. The second is an interpretability study that combines SHAP with feature ablation. The third is a shuffled-label control that tests whether a SHAP value has been earned. Two results follow, and both run against current practice. First, the classical linear models lead the ranking. The gradient-boosted ensembles and neural networks that dominate recent work bring no advantage. Second, SHAP can fail on high-cardinality categorical features. Ablation establishes the presenting HLA allele as the dominant input and the source gene as dispensable. SHAP reports the opposite, assigning 68.7% to 81.7% of its total value to the source gene in every model that gives each gene its own weight. Shuffling the gene labels at random among the peptides resolves the contradiction: the meaningless column earns the same SHAP value as the real one. That value measures only that ITSNdb contains 173 genes for 199 peptides. We therefore recommend that SHAP values for high-cardinality categorical annotations in cohorts of this size always be reported against such a shuffled-label control, and we describe one in full.
Figures & tables
| Study (tool) | Year | Dataset | ML algorithms | Reported performance |
|---|---|---|---|---|
| Wu et al. (2019) (DeepHLApan) | 2019 | IEDB: HLA–peptide pairs, immunogenic | Recurrent neural network (RNN) | No AUC reported for the immunogenicity model; as a filter on NetMHCpan, precision |
| Wells et al. (2020) (TESLA) | 2020 | screened epitopes; in an independent cohort | Feature-integrated immunogenicity model | Removed of non-immunogenic peptides at precision above , at a recall of |
| Li et al. (2021) (DeepImmuno) | 2021 | IEDB: peptide–HLA pairs ( immunogenic, not) | DeepImmuno-CNN | 10-fold CV: ROC-AUC , PR-AUC |
| Schmidt et al. (2021) (PRIME) | 2021 | Curated dataset ( peptides, immunogenic, not) | Predictor combining antigen presentation with TCR recognition | PRIME outperforms known methods on ROC-AUC and PR-AUC |
| Müller et al. (2023) | 2023 | NCI (112 patients), TESLA (8) and HiTIDE (11), harmonized: variants, neo-peptides, immunogenic | LR, SVM, XGBoost, CatBoost; LR + XGBoost voting ensemble | LR best; of immunogenic neo-peptides in the top 20 (NCI, leave-one-out CV); more in the top 20 than the previous ranking |
| Nibeyro et al. (2023) (ITSNdb) | 2023 | ITSNdb: neoantigens ( immunogenic, not) | Evaluation of seven published predictors; no new model | ROC-AUC across all seven |
| Family | Varied | S1 | S2 | S3 | S4 | S5 | Fixed settings |
| LR | 0.01 | 0.1 | 1.0 | 10 | 100 | balanced class weights; liblinear solver; max_iter 5000; features standardized | |
| LDA | shrink. | 0.1 | 0.25 | 0.5 | 0.75 | auto | lsqr solver; equal class priors; features standardized |
| SGD | log loss; L2 penalty; balanced class weights; averaged coefficients; max_iter 5000; standardized | ||||||
| KNN | (5,2) | (9,2) | (15,2) | (9,1) | (15,1) | distance weighting; numeric columns standardized, one-hot columns left at 0/1 | |
| DT | depth | 2 | 3 | 4 | 5 | 8 | min_samples_leaf 5; balanced class weights |
| SVM | (kern., ) | (lin,0.1) | (lin,1) | (rbf,0.1) | (rbf,1) | (rbf,10) | gamma = scale; balanced class weights; probability estimates enabled; standardized |
| Feature | Type | Role | Distribution in ITSNdb ( ) |
|---|---|---|---|
| Length | numeric | Fit within the MHC-I groove | 2 values: 9 ( ), 10 ( ) |
| mutPosition | numeric | TCR-facing versus MHC-facing mutation ( Capietto et al., 2020 ) | 10 values, 1–10; peaks at 8 ( ), 2 and 9 ( ) |
| IsAnchor | binary | Mutation buried in the MHC groove | anchor ( ), non-anchor ( ) |
| RelativePosition | numeric | Length-normalized mutation location | derived from Length and mutPosition |
| Tumor | categorical | Tumor-type context; cohort structure | 16 types; melanoma ( ), breast (15), lung (7), colorectal (6) |
| HLA | categorical | Presenting allele ( Reynisson et al., 2020 ) | 31 alleles; A02:01 ( ), B27:05 and A*03:01 (21 each) |
| Family | Accuracy | Balanced acc. | Sn | Sp | F1 | ROC-AUC | PR-AUC |
|---|---|---|---|---|---|---|---|
| LDA | 0.794 0.082 | 0.760 0.100 | 0.876 0.063 | 0.643 0.175 | 0.847 0.059 | 0.802 0.098 | 0.880 0.075 |
| SGD | 0.784 0.087 | 0.735 0.108 | 0.899 0.043 | 0.571 0.182 | 0.845 0.060 | 0.817 0.100 | 0.901 0.067 |
| ETC | 0.768 0.083 | 0.733 0.106 | 0.852 0.052 | 0.614 0.193 | 0.828 0.059 | 0.785 0.108 | 0.856 0.079 |
| SVM | 0.764 0.067 | 0.733 0.085 | 0.838 0.073 | 0.629 0.171 | 0.821 0.051 | 0.799 0.092 | 0.880 0.067 |
| RFC | 0.769 0.061 | 0.730 0.072 | 0.860 0.070 | 0.600 0.139 | 0.828 0.045 | 0.777 0.100 | 0.848 0.081 |
| ADA | 0.784 0.077 | 0.729 0.089 | 0.914 0.051 | 0.543 0.130 | 0.846 0.054 | 0.782 0.113 | 0.837 0.084 |
| Family | S1 | S2 | S3 | S4 | S5 | Mean | SD | Min | Max |
|---|---|---|---|---|---|---|---|---|---|
| LR | 0.716 | 0.730 | 0.725 | 0.706 | 0.683 | 0.712 | 0.019 | 0.683 | 0.730 |
| LDA | 0.722 | 0.728 | 0.733 | 0.725 | 0.719 | 0.725 | 0.005 | 0.719 | 0.733 |
| SGD | 0.649 | 0.641 | 0.661 | 0.632 | 0.732 | 0.663 | 0.040 | 0.632 | 0.732 |
| KNN | 0.617 | 0.592 | 0.591 | 0.621 | 0.627 | 0.610 | 0.017 | 0.591 | 0.627 |
| DT | 0.652 | 0.672 | 0.682 | 0.665 | 0.668 | 0.668 | 0.011 | 0.652 | 0.682 |
| SVM | 0.721 | 0.649 | 0.500 | 0.693 | 0.692 | 0.651 | 0.088 | 0.500 | 0.721 |
| Family | Setup | Gene | HLA | Tumor | Length | IsAnch. | mutPos. | RelPos. |
|---|---|---|---|---|---|---|---|---|
| one coefficient per category level | ||||||||
| LDA | LDA-S3 | 81.7 | 8.4 | 5.3 | 2.7 | 1.6 | 0.2 | 0.1 |
| MLP | MLP-S5 | 71.8 | 13.0 | 6.5 | 4.2 | 3.6 | 0.2 | 0.7 |
| SVM | SVM-S1 | 70.7 | 12.7 | 10.3 | 3.5 | 1.5 | 0.5 | 0.9 |
| SGD | SGD-S5 | 69.1 | 13.5 | 8.0 | 4.9 | 3.4 | 0.7 | 0.3 |
| LR | LR-S2 | 68.7 | 14.3 | 7.2 | 5.7 | 3.6 | 0.1 | 0.4 |
| Family | All | Gene | Gene HLA | HLA only |
|---|---|---|---|---|
| LDA | 0.760 | 0.713 | 0.649 | 0.681 |
| SGD | 0.735 | 0.737 | 0.654 | 0.650 |
| ETC | 0.733 | 0.736 | 0.684 | 0.701 |
| SVM | 0.733 | 0.713 | 0.664 | 0.681 |
| RFC | 0.730 | 0.722 | 0.666 | 0.701 |
| ADA | 0.729 | 0.737 | 0.542 | 0.701 |
SHAP Feature Attribution
SHAP (SHapley Additive exPlanations) feature attribution assigns model predictions to input features using Shapley values from cooperative game theory.
See all 44 papers