cs.AIOct 31, 2026

Benchmarking Machine-Learning Classifiers for Neoantigen Immunogenicity Prediction and the Limits of SHAP Interpretability

Authors: Ramon Gonzalez

Organizations: Mentomy AI

Abstract

Personalized cancer vaccines require prioritizing, among hundreds of candidate mutant peptides, the few that will induce a T-cell response. This study asks what general-purpose machine learning achieves on that task, and whether the features such models identify as important can be trusted. We make three contributions. The first is a benchmark of sixty models: twelve supervised classifier families, each in five configurations, evaluated on the Immunogenic Tumor Specific Neoantigen database (ITSNdb; 199 experimentally validated neoantigens, 129 immunogenic and 70 non-immunogenic). All sixty are compared under one stratified k-fold cross-validation protocol, in which the data used to choose a configuration are never the data used to score it. The second is an interpretability study that combines SHAP with feature ablation. The third is a shuffled-label control that tests whether a SHAP value has been earned. Two results follow, and both run against current practice. First, the classical linear models lead the ranking. The gradient-boosted ensembles and neural networks that dominate recent work bring no advantage. Second, SHAP can fail on high-cardinality categorical features. Ablation establishes the presenting HLA allele as the dominant input and the source gene as dispensable. SHAP reports the opposite, assigning 68.7% to 81.7% of its total value to the source gene in every model that gives each gene its own weight. Shuffling the gene labels at random among the peptides resolves the contradiction: the meaningless column earns the same SHAP value as the real one. That value measures only that ITSNdb contains 173 genes for 199 peptides. We therefore recommend that SHAP values for high-cardinality categorical annotations in cohorts of this size always be reported against such a shuffled-label control, and we describe one in full.

Figures & tables

Explore similar work

Aug 3, 2026

TransNRank: Towards Accurate Neoantigen Ranking with Transformer

Personalized neoantigen prediction is challenging due to the scarcity of positive samples, the noise of the experimental data, the severe class imbalance trait and the complex of immunogenicity features. Prior arts, such as linear regression and XGBoost fail to model long-range dependencies and contextual relationships within peptide features, therefore the performance of neoantigen positive recall rate is limited. In this paper, we present a novel deep learning framework based on Transformer, coined as TransNRank. By leveraging the self-attention mechanism, our model captures both local and global feature contexts, enabling more accurate recognition of immunogenic neoantigens. A positive-aware training objective is utilized to handle the class imbalance problem, assigning more weights to those few positive samples. Extensive experiments are performed on NCI, TESLA and HiTIDE datasets. Notably, our TransNRank can push the upper bound top 20 recall rate of neoantigen prediction from 46.9% (45 from 96) to 53.1% (51 from 96), while reducing the training epochs from 200 epochs to 20 epochs. Furthermore, we analyze the features contribution based on TransNRank and find that the mutation at anchor and TCGA expression level play an unexpected important role in neoantigen prediction, and removing insignificant features to reduce the input dimensionality of peptides does not drastically impair the overall performance of the model. Our paradigm not only streamlines the prediction pipeline but also sets a new state-of-the-art for neoantigen discovery, with broad implications for accurate immuno-oncology.
Oct 21, 2026

Mastering Model Fit: Applications of the InterModel Vigorish for Interpretable Machine Learning

Accuracy alone gives little information about which features or model components improve probabilistic fit, which outcomes benefit, or how fit changes during training. We introduce the InterModel Vigorish (IMV) into the Machine Intelligence space in order to express differences in fit as an expected betting advantage over a reference model. SHAP-IMV attributes gains to features through Shapley-weighted subset comparisons, Multi-IMV evaluates one-vs-rest and pairwise class discrimination, and Ablate-IMV assesses architectural components and feature groups by ablation. We evaluate these applications over ten random seeds using tabular data, images, sensor sequences, and graphs. Feature rankings vary across estimators. High one-vs-rest accuracy can accompany negligible improvement over a null, while ablation can preserve or improve fit under a fixed training budget. On six graph-classification benchmarks, we time-profile IMV over 200 training epochs, reporting seed means and ranges. The open-source imvpy library and accompanying notebooks provide code to replicate our work and to use the IMV more widely.
Oct 13, 2026

Evaluating the Causal Faithfulness of SHAP as a Candidate Attribution Method for SAE Features

Sparse autoencoders decompose language model representations into interpretable features, but feature interpretation in this literature remains predominantly correlational. Attribution methods offer more direct estimates of feature influence, yet their faithfulness to actual causal effects has not been systematically evaluated in the SAE setting. We apply SHAP-based attribution to SAE latent features and evaluate it against direct causal intervention, using sentiment classification on SST-2 with GPT-2 Small and a 32,768-feature layer-7 SAE as a testbed. We compare SHAP to Integrated Gradients, Gradient×\timesInput, and probe-weight baselines across linear and MLP probe arms, scoring each against feature ablation and activation steering measured through the language model's own logit difference. Most faithfulness associations we measure do not survive multiple-comparison correction, and attribution methods that agree strongly with one another (ρ=0.42ρ= 0.42--0.800.80) agree poorly with measured causal effects, indicating that inter-method agreement is not evidence of causal faithfulness. Two effects hold under control. First, local per-example attribution retains a directional advantage over global averaged attribution, while its apparent importance advantage is explained by feature activity rather than superior causal ranking. Second, under known-ground-truth minimal pairs, attribution recovers inserted sentiment signal well above chance (AUROC 0.7920.792) but fails to localize deletions (AUROC 0.5200.520), suggesting that these methods detect the presence of a feature's influence more reliably than its absence. We argue that importance faithfulness and directional faithfulness are separable properties requiring separate evaluation, and that attribution agreement should not substitute for causal validation.