cs.AISep 11, 2026
SaveCausal multi-modal AI for personalized chemosensitivity prediction
Organizations: Ataraxis AI, New York, NY, USA. · Centre for Inflammation Biology and Cancer Immunology, King’s College London, London, UK. · Department of Medical Oncology, Guy’s Hospital, London, UK. · School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, UK. ·
Abstract
Chemotherapy improves survival for some patients with breast cancer, but doctors cannot reliably predict who. Current guidelines rely on recurrence scores as a proxy for treatment benefit, which may contribute to the overprescription of chemotherapy. Here we present a causal multi-modal AI model that predicts personalized chemosensitivity using routinely collected pathology and clinical information. We developed our model on a multi-national dataset of 9,141 patients (twelve cohorts, nine countries) and evaluated it on another 1,994 patients (five cohorts, three countries). The model generated treatment-specific recurrence probabilities for each patient, with near-perfect calibration and strong prognostic discrimination across both 5- and 10-year follow-up horizons. Moreover, its chemotherapy benefit predictions demonstrated robust predictive performance, and out-performed existing recurrence-score-based tests. Compared to the standard of care, using the model to support personally tailored therapeutic decisions could reduce the number of patients receiving chemotherapy by 30% while achieving the same recurrence-free rate. Tumors predicted to be highly chemosensitive displayed concordant molecular and morphological programs of proliferation, cell cycle progression, and replication stress. The model's predictive capabilities transferred zero-shot to non-breast cancers, indicating our causal multi-modal AI approach may provide a universal strategy to predict treatment outcomes across cancer types.
Figures & tables
Figure 1: Causal multi-modal AI model for personalized chemosensitivity prediction. a , Survival curves are shown for a hypothetical patient with HR+/HER2 breast cancer to illustrate baseline risk with endocrine therapy (ET, green line) and residual risk with chemoendocrine therapy (CET, gold line). The difference is the personalized chemotherapy benefit (black arrow). b , Three scenarios for how prognosis and chemotherapy benefit predictions could provide personally tailored therapeutic decision support for patients with HR+/HER2 breast cancer: (1) low risk and low-benefit, for whom ET is sufficient; (2) high risk and high-benefit, for whom CET is warranted; (3) high risk but low-benefit from standard chemotherapy, for whom escalation with additional agents may be required. c , Development and evaluation cohorts. Marker area is proportional to cohort size. Blue indicates the development set (12 cohorts, n = 9,141) and purple the held-out evaluation set (5 cohorts, n = 1,994). d , Pathology foundation model training. The foundation model learns general-purpose representations of tissue morphology through self-supervised pretraining on over two billion H&E image patches. Training follows a similar procedure to DINOv2 [ 61 ] , a state-of-the-art self-supervised learning method in which the model is taught to produce consistent representations of the same tissue region viewed under different augmentations, using a giant vision transformer (ViT-g). Falcon, the pathology foundation model used here, is a successor to Kestrel [ 41 ] . e , Causal AI model training. A counterfactual regression network (CFRNet [ 59 , 60 ] ) uses a maximum mean discrepancy penalty to balance the representation distributions of the ET and CET treatment groups, then predicts recurrence separately under each treatment. f , Architecture of the multi-modal AI. Whole-slide images are divided into 224 224 patches, encoded by the pathology foundation model, and aggregated by multiple instance learning into a single slide-level embedding; clinical features are encoded separately by a tabular residual network. The two embeddings are fused with a gated attention layer and passed to CFRNet, the causal model. The causal model outputs CTX-prognostic — the predicted risk of recurrence under the treatment a patient actually received, baseline risk under ET or residual risk under CET — and CTX- , the predicted increase in recurrence-free probability with CET versus ET, which is the personalized estimate of chemotherapy benefit.
Figure 2: Evaluation of CTX as a predictor of prognosis and chemosensitivity. Evaluations performed in the evaluation set (n=1,994). a , Calibration plot compares CTX-prognostic predictions versus observed event probability for five-year recurrence-free interval (RFI). Circles denote observed event probabilities within binned strata of predicted risk, with vertical error bars representing 95% CIs; the diagonal indicates perfect calibration. b, c , Kaplan–Meier curves for the probability of meeting the RFI endpoint in the CTX- predicted low-benefit ( b ) and high-benefit ( c ) groups, stratified by adjuvant therapy (chemoendocrine therapy [CET, gold] vs. endocrine therapy [ET, green]), with risk tables underneath. Risk tables report IPW-adjusted numbers at risk, not raw subgroup sizes. Within each subgroup, the HR, 95% CI, and p-value are from an IPW-adjusted Cox model containing chemotherapy treatment status as the only covariate, fit to that subgroup; the p-value is from the score test. d , Forest plot of the hazard ratio (HR) for CTX- , chemotherapy treatment status (ET or CET), and the interaction of chemotherapy and CTX- in an IPW-adjusted Cox model. The center box indicates the HR, and error bars represent 95% confidence intervals (CIs). p-values are indicated for the treatment-by-biomarker interaction term. e , HR for chemotherapy benefit across the range of continuous CTX- , with 95% CI (shaded region). Estimates were derived by the delta method applied to an IPW-adjusted Cox model including an interaction term between continuous CTX- score and chemotherapy treatment, with robust standard errors. The dashed reference line indicates no chemotherapy benefit (HR = 1). f , Lollipop plot comparing the predictive performance of clinical, pathological, molecular, and CTX- for five-year RFI. For each marker, the vertical axis shows the treatment-by-biomarker interaction p-value from a Cox model containing the biomarker, chemotherapy treatment status, and their interaction. The dashed horizontal line indicates p = 0.05. Points are colored by feature category: patient (green), clinical (purple), molecular (blue), and CTX (gold).
Figure 3: CTX separates chemosensitivity from risk to enable personalized therapeutic decision support. a , Scatter plots compare CTX- against four risk estimates: CTX-prognostic, tumor size, number of positive lymph nodes, and Oncotype DX recurrence score. The blue line indicates the median (50th-percentile) quantile regression fit, with the shaded ribbon representing the interquartile range (25th–75th percentile) band. Pearson correlation coefficients (r) are shown for each comparison. b , Predicted five-year recurrence rate as a function of the proportion of patients receiving chemotherapy (CET rather than ET). Patients are ranked according to those most likely to benefit from chemotherapy, as judged by CTX- predictions. The curve shows the IPW-adjusted Kaplan–Meier estimated outcomes if the proportion of patients receiving chemotherapy increases from 0% to 60%, starting with the patient with the greatest predicted benefit. The x-axis ends at 60%, within one standard deviation of the mean observed treatment rate across cohorts. The proportion treated with CET and recurrence rate is indicated for the observed population (grey circle). The arrows indicate the same values for CTX- -guided strategies reducing chemotherapy allocation while matching the recurrence-free rate (horizontal arrow), or improving recurrence-free rate while matching the observed treatment rate (vertical arrow).
Figure 4: Personalized predictions capture heterogeneity obscured at the (sub)population level. a , Illustration of the hierarchy of populations and subpopulations from which we may estimate prognosis and chemotherapy benefit. Estimates can be derived from the full population (left), subpopulations defined by clinicopathological features (middle), or the individual patient (right). Moving from left to right narrows the group being characterized, recovering precision that coarser estimates average away. b , Full population IPW-adjusted Kaplan–Meier (KM) curves estimating the probability of recurrence in the evaluation set (n = 1,994), stratified by adjuvant therapy status (chemoendocrine therapy [CET] in gold vs. endocrine therapy [ET] in green). Average benefit is the difference in the KM-estimated five-year recurrence probability between the two arms (ET minus CET), with positive values denoting lower recurrence with the addition of chemotherapy. The average benefit across the entire evaluation set is 0.30%. c , Subpopulations defined by clinicopathological features: node-positive disease, large tumor (tumor size median), and Oncotype DX category (high-risk RS 26, low- and intermediate-risk RS < 26, the split used for chemotherapy allocation). d , Case study 1. Patient 1 and 2 are both premenopausal, node-positive with large tumors. The CTX-predicted probability of recurrence under ET and under CET are annotated. CTX recurrence probabilities assigned both patients a high baseline risk under ET, in agreement with standard clinicopathological risk indicators. However, CTX- separated them, predicting 0.71% benefit for patient 1 and 4.41% benefit for patient 2. In practice, both patients received CET, but their outcomes differed. Patient 1 recurred at approximately 3 years, whereas Patient 2 remained recurrence-free (censored at approximately 3.5 years). These patient outcomes are aligned with their CTX- scores, which predicted patient 1 to be significantly less chemosensitive than patient 2. e , Case study 2. Patient 3 and 4 both have node-negative disease and intermediate Oncotype DX recurrence scores. CTX predicted patient 3 to be at low risk with negligible chemotherapy benefit (CTX- = 0.17%) and patient 4 to be at high risk with substantial benefit (CTX- = 3.28%). The treatment each patient received ran counter to these predictions: patient 3 received CET and patient 4 received ET alone. Patient 3 remained recurrence-free across approximately 17 years of follow-up, suggesting that chemotherapy de-escalation might have been safe — as predicted by CTX. Patient 4 recurred at approximately 4.8 years, suggesting that chemotherapy escalation may have been prudent — as predicted by CTX. In both case studies, patients who are indistinguishable based on the clinicopathological features experienced divergent outcomes which were tracked by divergent CTX predictions. Panels b – e share a common y-axis (probability of recurrence) and x-axis (time since diagnosis, years). Individual counterfactual outcomes are never observed, so these examples illustrate the interpretation of CTX- at the level of the individual patient rather than constituting a formal evaluation of predictive accuracy; the latter is presented in Figure 2 .
Figure 5: Morphological and molecular correlates of CTX- predictions. a , Two-dimensional UMAP projection [ 74 ] of embeddings from the pathology foundation model, Falcon, for a cohort in the evaluation set (UNIRAD, n = 362). Each point represents a single H&E image patch. Patches were partitioned into 9 clusters by -means applied in the embedding space. Clusters are colored by their average CTX- score, from low- (dark blue) to high- (bright yellow) predicted chemotherapy benefit. b , Representative H&E patches selected from clusters in a , ordered by average CTX- score (low to high). c , Bar plot shows gene-set enrichment analysis [ 75 ] of MSigDB Hallmark pathways for CTX- low- versus high-benefit tumors in the TCGA breast cancer cohort (n=348). Bars show normalized enrichment scores. Only pathways significant at false discovery rate (FDR) < 0.05 are shown. d , Pathways enriched in CTX- low-benefit tumors include early estrogen response, epithelial-mesenchymal transition, and angiogenesis. Pathways enriched in CTX- high-benefit tumors include proliferative (E2F targets, G2M checkpoint, MYC targets, MTORC1 signaling), DNA damage response (DNA repair), immune activation (IFN- and IFN- response), and cGAS-STING signatures.
Figure 6: Zero-shot transfer of CTX prognosis and chemosensitivity predictions across cancer types. The frozen CTX model, trained only on breast cancer, was applied without fine-tuning or re-calibration to patients spanning 16 cancer types from The Cancer Genome Atlas (TCGA), using overall survival as the endpoint. Performance of CTX in the evaluation set is also displayed (as “breast”) for reference. Each cancer type is colored by adjuvant systemic therapy class — anthracycline/platinum/taxane (gold), alkylator (purple), or other cytotoxic (light grey); targeted and/or immunotherapy (blue); or none (green). CTX was trained to predict benefit from anthracycline-, platinum- and/or taxane-based regimens, therefore prognostic and predictive transfer is expected primarily in cancer types treated with mechanistically comparable cytotoxic regimens. a , Box plot shows prognostic transfer of CTX-risk, assessed independently in each cancer type by univariable Cox regression (n = 6,692 patients across 16 cancer types, and n = 1,994 CTX breast evaluation set). Points show the hazard ratio for CTX-risk per cancer type; vertical bars are 95% CIs. The horizontal dotted line marks the null (hazard ratio = 1). b , Bar plot shows predictive transfer of CTX- , evaluated in the cancer types routinely treated with adjuvant systemic therapy (n = 5,783 patients across 16 cancer types, and n = 1,994 CTX breast evaluation set). Within each cancer type, patients were ranked by CTX- and split into high- and low-benefit subgroups at the 90th percentile; in an IPW-adjusted cohort we computed the difference in restricted mean survival time (RMST) between chemotherapy-treated and untreated patients within each subgroup. Each bar is the high-benefit minus low-benefit RMST (years). Positive RMST values indicate that the subgroup CTX- predicts will benefit most gained more survival time from chemotherapy than the low-benefit subgroup (correct predictive ordering); negative values indicate the reverse.
Extended Data Fig. 1 : Propensity adjustment. a , Histogram of propensity score estimates colored by treatment status. Scores are clipped so that no patient has an inverse propensity weight (IPW) greater than 10. b , Standardized mean differences between chemoendocrine and endocrine therapy patients, before and after IPW-adjustment.
Extended Data Fig. 2 : Pathology foundation model benchmarking. The pathology foundation model used by CTX ( Figure 1 d ), Falcon, is a ViT-Giant (1.1B parameters) pretrained with DINOv2 [ 61 ] on over two billion H&E patches from 180,000 whole-slide images. Falcon was benchmarked on a slide-level task suite against its predecessor Kestrel [ 41 ] (ViT-Large, 303M parameters) and two publicly available pathology foundation models, UNI 2 [ 42 ] (681M parameters) and Prov-GigaPath [ 43 ] (1.1B parameters). No pretraining slide was used in any benchmark task, and no benchmark task overlapped with any datasets used for evaluating CTX. a , Benchmarking protocol. Each foundation model extracts a patch-level embedding for all patches in each slide. All embeddings from a slide are mean-pooled, and this mean-pooled embedding is used as a feature vector to evaluate the performance of the foundation model on survival or classification tasks. b , Aggregate performance on a suite of 24 slide-level histopathology evaluation tasks. Performance is the C-index for survival tasks and AUC for mutation and biomarker tasks. Falcon achieved the highest mean performance across all tasks. The gain over Kestrel, trained using similar protocol on an order of magnitude fewer data, indicates that generalization performance scales with model capacity and training duration. c , Per-task slide-level performance underlying b . Axis labels describe the cohort, and the target for classification tasks; in the case of survival tasks, the target was disease-specific survival. d , Performance of Falcon on a suite of 24 patch-level histopathology evaluation tasks. Hyperparameters for Falcon were tuned by optimizing performance on this task suite. Each task tests the model’s ability to identify a specific property of cancerous tissue. All tasks are classification tasks, except for BreastPathQ and CoNIC which are regression tasks. The performance of the final Falcon model is shown.
Extended Data Fig. 3 : Evaluation of CTX as a prognostic marker. CTX-prognostic predictions for recurrence-free interval (RFI) in the evaluation set (n = 1,994). a , Receiver operating characteristic (ROC) curves illustrate the sensitivity and specificity of CTX-prognostic for five-year (blue) and 10-year (red) RFI. b , Forest plot shows the adjusted hazard ratio (HR) for CTX-prognostic in a multivariable Cox model adjusting for age at diagnosis, tumor stage, and nodal stage, at the five-year (blue) and 10-year (red) horizons. The center box indicates the HR, and error bars represent 95% confidence intervals (CIs). p-values are indicated for each predictor in the multivariable models.
Extended Data Fig. 4 : Evaluation of CTX as a predictive marker of chemotherapy benefit. Analyses performed in the evaluation set (n = 1,994) unless otherwise stated. Forest plots show the hazard ratio (HR; center box) with 95% confidence intervals (CIs; error bars) for each covariate. a , Treatment-by-biomarker interaction for chemotherapy and CTX- in an unweighted multivariable Cox model adjusting for CTX- category (low- vs. high-benefit), adjuvant treatment (endocrine therapy [ET] vs. chemoendocrine therapy [CET]), age at diagnosis, tumor stage, nodal stage, grade, and histological subtype (invasive ductal carcinoma [IDC], invasive lobular carcinoma [ILC]), together with the CTX- and chemotherapy interaction term (CTX- :CET). The interaction HR quantifies how much the effect of chemotherapy differs between low- and high-benefit patients. b , Ablation analysis quantifying the contribution of each input modality to the predictive signal. CTX- scores from the clinical-only and pathology-only models were each dichotomized at the 2% threshold and entered into a single IPW-adjusted Cox model with chemotherapy treatment status and both treatment-by-biomarker interactions (CTX- clinical:CET, CTX- pathology:CET). The likelihood ratio test (LRT) p-value compares this model against a nested model omitting the pathology interaction, testing whether histopathology adds predictive information beyond clinical features. c , Head-to-head comparison of the strength of treatment-effect modification by CTX- and the Oncotype DX recurrence score, in the subset of patients with a recorded Oncotype DX score (n = 983). Bars show the treatment-by-biomarker interaction p-value from an IPW-adjusted Cox model containing the standardized biomarker, chemotherapy treatment status, and their interaction, fit separately for each biomarker in the same patients. The dashed horizontal line indicates p = 0.05.
Extended Data Fig. 5 : Patch-level annotation of morphological correlates of CTX- predictions. Representative H&E image patches (224 224 pixels, 0.5 µm per pixel) from each of the nine -means clusters shown in Figure 5 a , derived from Falcon patch embeddings in the UNIRAD evaluation cohort (n = 362). Columns correspond to clusters, ordered left to right by increasing cluster-level CTX- coefficient, from clusters enriched in CTX- low-benefit tumors (dark blue) to those enriched in CTX- high-benefit tumors (bright yellow); the colored square above each column matches the cluster coloring in Figure 5 a . Five randomly sampled patches are shown per cluster. The label beneath each patch gives the dominant morphological pattern annotated by a board-certified pathologist with subspecialty expertise in breast pathology. Clusters associated with low predicted benefit are dominated by fibrotic collagen, adipose tissue, normal ducts, and lymphocytes, with sparse invasive carcinoma; clusters associated with high predicted benefit consist predominantly of nests of invasive carcinoma with high cellularity, alone or admixed with lymphocytes.
Extended Data Fig. 6 : Data filtering flow diagrams. Data filtering flow diagrams show the sequential inclusion and exclusion criteria applied to assemble the development set (left, 12 cohorts) and the held-out evaluation set (right, 5 cohorts). Blue boxes indicate the number of patients retained at each step. Orange boxes indicate the number excluded and the reason for exclusion. Of 23,912 patients in the development cohorts, 9,141 with at least one whole-slide image, a known recurrence endpoint, early-stage female breast cancer, no prior breast cancer history, and known adjuvant chemotherapy status were retained for model development. Development cohorts were deliberately not restricted by receptor subtype or treatment. Of 2,877 patients in the evaluation cohorts, 1,994 were retained after additionally restricting to the intended deployment population — HR+/HER2 disease, receipt of adjuvant endocrine therapy (ET) with known adjuvant chemotherapy status, no adjuvant targeted therapy — and excluding slides that failed quality review.
| Characteristic | Development cohort (N=9141) | Evaluation cohort (N=1994) |
|---|---|---|
| Age at diagnosis , median [iqr] | 56.0 [47.0–65.1] | 57.0 [49.0–65.0] |
| Race | ||
| Asian | 1181 (12.92%) | 26 (1.30%) |
| Black or African American | 96 (1.05%) | 154 (7.72%) |
| White | 1114 (12.19%) | 331 (16.60%) |
| Other/Unknown | 6750 (73.84%) | 1483 (74.37%) |
Extended Data Table 1 : Baseline characteristics of the development and evaluation cohorts.
| Hyperparameter | Search space |
|---|---|
| Optimization | |
| Modality branch weights ( ) | Uniform (0.0, 1.0) |
| Elastic-net mixing weight ( ) | Uniform (0.0, 1.0) |
| Elastic-net regularization weight ( ) | Log-uniform ( , ) |
| Learning rate | Log-uniform ( , ) |
| Balancing penalty (MMD) | |
Extended Data Table 3 : CTX hyperparameter search space. Continuous parameters were sampled from Uniform or Log-uniform distributions over the specified ranges; discrete sets were sampled uniformly.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Dordrecht | Roswell Park | UChicago | UNIRAD | Karmanos | |
|---|---|---|---|---|---|
| Low | 0 (0%) | 102 (22%) | 73 (17%) | 0 (0%) | 33 (22%) |
| Intermediate | 0 (0%) | 297 (63%) | 231 (53%) | 0 (0%) | 85 (58%) |
| High | 0 (0%) | 69 (15%) | 65 (15%) | 0 (0%) | 28 (19%) |
| Not tested | 582 (100%) | 0 (0%) | 66 (15%) | 362 (100%) | 1 (1%) |
Supplemental Table 1: Oncotype DX score category distributions across the evaluation datasets.
| Challenge | Our solution | |
|---|---|---|
| 1 | Only unlabelled H&E WSIs are commonly available, so robust and generalizable representations must be learned without supervision | A multi-modal architecture built on a self-supervised pathology foundation model ( Sections A.3.1 and 4.2 ) |
| 2 | Treatment assignment is not randomized in observational data, so patients who received chemotherapy differ systematically from those who did not | Treatment-specific output heads and a between-arm balancing penalty that improve extrapolation across treatment groups ( Sections A.3.1 and A.3.2 ) |
| 3 | The outcome is a censored, time-to-event endpoint | A tailored training objective ( Section A.3.2 ) |
| 4 | The data are pooled across heterogeneous cohorts | Multi-source cross-validation ( Section A.3.3 ) |
Supplemental Table 2: Methodological challenges and solutions. H&E, hematoxylin and eosin; WSI, whole slide image.
Supplemental Fig. 1 : Architecture of the proposed Multi-modal model for estimating chemotherapy benefit. Our model takes two input modalities for each patient: clinical variables and histopathology image. The clinical variables are encoded into a clinical embedding using a tabular ResNet, while patches of the image are converted into patch embeddings and aggregated into a slide-level embedding through gated attention. Then the clinical and slide representations are fused through gated attention, and passed to output heads estimating recurrence-free probability under chemoendocrine therapy and endocrine therapy alone . The predicted chemotherapy benefit ( ) is the difference . Auxiliary output heads of the clinical and pathology branches are omitted for clarity. FC denotes a single fully connected (linear) layer, whereas the MLP consists of two fully connected layers with a ReLU activation between them.