cs.LGApr 1, 2026
SaveBioCOMPASS: Integrating Biomarkers into Transformer-Based Immunotherapy Response Prediction
Organizations: University of York, UK
Abstract
Datasets used in immunotherapy response prediction are typically small in size, as well as diverse in cancer type, drug administered, and sequencer used. Models often drop in performance when tested on patient cohorts that are not included in the training process. Recent work has shown that transformer-based models along with self-supervised learning show better generalisation performance than threshold-based biomarkers, but is still suboptimal. We present BioCOMPASS, an extension of a transformer-based model called COMPASS, that integrates biomarkers and treatment information to further improve its generalisability. Instead of feeding biomarker data as input, we built loss components to align them with the model's intermediate representations. We found that components such as treatment gating and pathway consistency loss improved generalisability when evaluated with Leave-one-cohort-out, Leave-one-cancer-type-out and Leave-one-treatment-out strategies. Results show that building components that exploit biomarker and treatment information can help in generalisability of immunotherapy response prediction. Careful curation of additional components that leverage complementary clinical information and domain knowledge represents a promising direction for future research.
Figures & tables
Figure 1: BioCOMPASS architecture: Gene expression data is first fed into the COMPASS encoder to generate embeddings. Minimising the pathway consistency loss makes sure that the pathway scores predicted from embeddings are aligned with external pathway scores. The embeddings are then fed into the COMPASS concept bottleneck to generate 44 biological concepts. These are aligned with cell-type biomarker scores using the concept alignment objective. They are also used to predict immunotherapy response prediction biomarkers such as TIDE & IPRES and other immune phenotypes. The concepts are also scaled based on the specific treatment type using the treatment gating module. The scaled concepts are then used to predict response using a classifier head. Components from COMPASS are in blue colour while BioCOMPASS components are in green.
| Cohorts | Model | Accuracy | ROC-AUC | F1 | Precision | Recall |
|---|---|---|---|---|---|---|
| All | C | 63.10 ± 5.43 | 70.99 ± 2.88 | 46.65 ± 2.15 | 48.33 ± 6.24 | 56.93 ± 6.51 |
| BC | 70.00 ± 1.76 | 73.58 ± 1.29 | 54.01 ± 2.81 | 56.00 ± 2.64 | 58.55 ± 6.55 | |
| Small (<50) | C | 63.03 ± 6.15 | 72.05 ± 4.30 | 39.55 ± 3.98 | 40.82 ± 8.86 | 51.88 ± 3.91 |
| BC | 69.77 ± 1.86 | 74.44 ± 2.46 | 52.93 ± 1.67 | 52.51 ± 4.30 | 61.95 ± 6.96 | |
| Large (>50) | C | 63.18 ± 6.74 | 69.94 ± 2.75 | 53.74 ± 6.33 | 55.84 ± 6.25 | 61.98 ± 15.36 |
| BC | 70.24 ± 4.84 | 72.72 ± 0.80 | 55.08 ± 3.96 | 59.48 ± 5.83 | 55.14 ± 9.82 |
Table 1: LOCO validation of COMPASS (C) and BioCOMPASS (BC). This table shows the average performance (in %) across all left-out cohorts. The first two rows show results averaged across all 8 test cohorts; the ones below show the same across 4 small cohorts (less than 50 samples) and 4 large cohorts (more than 50 samples). The 95% confidence intervals (CI) show variation across 4 seeds.
Figure 2: Points show mean performance across four random seeds with 95% CI error bars. Circles: COMPASS; Squares: BioCOMPASS. BioCOMPASS shows consistent improvements in accuracy and ROC-AUC across most cohorts. Please note that the subplots have different scales on x-axis.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Cohort | Cancer Type | Drug | Number of Samples | Size Category |
|---|---|---|---|---|
| Mariathasan et al. (2018) | BLCA | Atezolizumab | 298 | Large (>50) |
| McDermott et al. (2018) | KIRC | Atezolizumab | 247 | Large (>50) |
| Liu et al. (2019) | SKCM | Nivolumab | 121 | Large (>50) |
| Gide et al. (2019) | SKCM | Nivolumab | 73 | Large (>50) |
| Riaz et al. (2017) | SKCM | Nivolumab | 49 | Small (<50) |
| Kim et al. (2018) | STAD | Pembrolizumab | 45 | Small (<50) |
Table 2: Cohorts used for LOCO evaluation along with the cancer type, drug used, number of samples and size category as well as a citation to the publication.
| Config | Acc (%) | AUC (%) | F1 (%) | Prec (%) | Recall (%) |
|---|---|---|---|---|---|
| No Gating | 72.13 ± 3.32 | 64.16 ± 4.52 | 48.23 ± 5.07 | 51.07 ± 6.48 | 58.65 ± 8.37 |
| No Pathway | 72.24 ± 2.88 | 67.53 ± 3.51 | 52.05 ± 3.74 | 53.96 ± 5.88 | 57.01 ± 5.52 |
| No Auxiliary | 72.55 ± 3.13 | 68.32 ± 3.06 | 51.94 ± 4.04 | 53.58 ± 5.36 | 57.10 ± 6.15 |
| No Alignment | 72.52 ± 3.02 | 68.09 ± 3.48 | 52.50 ± 4.20 | 54.78 ± 5.76 | 58.83 ± 6.35 |
Table 3: Ablation study of various components in BioCOMPASS. Each row shows performance when one component is disabled. Values are mean ± 95% CI margin across five random seeds across all cohorts in LOCO setting. Treatment gating contributes most to performance improvement, followed by pathway consistency. Lower is better here as it shows that taking way that component reduced performance the most.
| Setting | Model | Accuracy | ROC AUC | F1 | Precision | Recall |
|---|---|---|---|---|---|---|
| LOCTO | C | 61.69 ± 6.99 | 67.72 ± 3.87 | 42.41 ± 5.40 | 42.61 ± 13.69 | 51.50 ± 8.44 |
| BC | 70.05 ± 1.63 | 71.65 ± 3.02 | 49.16 ± 5.72 | 51.66 ± 2.38 | 50.37 ± 10.22 | |
| LOTO | C | 68.93 ± 2.95 | 74.49 ± 1.56 | 57.02 ± 2.16 | 55.44 ± 2.04 | 63.53 ± 4.86 |
| BC | 73.49 ± 1.36 | 76.85 ± 3.56 | 60.53 ± 2.66 | 61.01 ± 1.84 | 63.36 ± 4.28 |
Table 4: Performance of COMPASS (C) and BioCOMPASS (BC) in LOCTO and LOTO settings. This table shows the average performance (in %) across 4 left-out cancer types (BLCA, KIRC, SKCM, STAD) and 4 immunotherapy treatment targets (PD-1, PD-L1, CTLA-4, CTLA-4 + PD-1). The 95% confidence intervals (CI) show variation across 4 seeds.
Figure 3: Points show mean performance across four random seeds with 95% CI error bars. Circles: COMPASS; Squares: BioCOMPASS. Please note that the subplots have different scales on x-axis.
Figure 4: Points show mean performance across four random seeds with 95% CI error bars. Circles: COMPASS; Squares: BioCOMPASS. Please note that the subplots have different scales on x-axis.
| Method | Description | Reference |
|---|---|---|
| GeneBio | Combined score of immunotherapy target markers PD1/PDL1/CTLA4 | Kong et al. (2022) |
| CTLA4 | Expression of CTLA4 as a single ICI target marker | Kong et al. (2022) |
| PD1 | Expression of PDCD1 as a single ICI target marker | Kong et al. (2022) |
| PDL1 | Expression of CD274 as a single ICI target marker | Kong et al. (2022) |
| CD8 | CD8 + T cell score derived from average expression of CD8A and CD8B | Chen et al. (2016) Kong et al. (2022) |
| CIS | Cytotoxic immune signature score averaging cytotoxic immune genes | Davoli et al. (2017) |
Table 5: Description of biomarker-based baseline methods. These were used based on the implementation by COMPASS.
| Method | AUC (%) | Accuracy (%) | F1 (%) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| C | CT | T | C | CT | T | C | CT | T | |
| Logistic regression on baseline methods | |||||||||
| CKS | 61.88 | 63.96 | 63.05 | 57.70 | 59.99 | 59.30 | 48.23 | 47.95 | 48.78 |
| GEP | 62.83 | 65.50 | 65.42 | 57.27 | 57.05 | 60.02 | 48.30 | 46.60 | 47.77 |
| IFNG | 61.97 | 64.82 | 62.89 | 65.57 | 68.51 | 65.23 | 38.85 | 38.36 | 39.31 |
| CD8 | 61.13 | 64.39 | 61.99 | 57.27 | 59.74 | 62.65 | 44.61 | 42.89 | 48.80 |
Table 6: Performance of baseline methods in LOCO (C), LOCTO (CT) and LOTO (T) settings. It can be seen that all baseline methods show poor generalisability. Immune score based methods were used based on the implementation by COMPASS. LR: Logistic regression, GBM: Gradient boosting machine, RF: Random forest, PCA: Principal component analysis