cs.AISep 11, 2026

Causal multi-modal AI for personalized chemosensitivity prediction

Authors: Dhruva Biswas, Jeroen Berrevoets, Alec McClean, Linus Bao, Jungkyu Park, Ken G. Zeng, Joseph Cappadona, Cerise Tang, 

Organizations: Ataraxis AI, New York, NY, USA. · Centre for Inflammation Biology and Cancer Immunology, King’s College London, London, UK. · Department of Medical Oncology, Guy’s Hospital, London, UK. · School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, UK. · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Chemotherapy improves survival for some patients with breast cancer, but doctors cannot reliably predict who. Current guidelines rely on recurrence scores as a proxy for treatment benefit, which may contribute to the overprescription of chemotherapy. Here we present a causal multi-modal AI model that predicts personalized chemosensitivity using routinely collected pathology and clinical information. We developed our model on a multi-national dataset of 9,141 patients (twelve cohorts, nine countries) and evaluated it on another 1,994 patients (five cohorts, three countries). The model generated treatment-specific recurrence probabilities for each patient, with near-perfect calibration and strong prognostic discrimination across both 5- and 10-year follow-up horizons. Moreover, its chemotherapy benefit predictions demonstrated robust predictive performance, and out-performed existing recurrence-score-based tests. Compared to the standard of care, using the model to support personally tailored therapeutic decisions could reduce the number of patients receiving chemotherapy by 30% while achieving the same recurrence-free rate. Tumors predicted to be highly chemosensitive displayed concordant molecular and morphological programs of proliferation, cell cycle progression, and replication stress. The model's predictive capabilities transferred zero-shot to non-breast cancers, indicating our causal multi-modal AI approach may provide a universal strategy to predict treatment outcomes across cancer types.

Figures & tables

Figure 1: Causal multi-modal AI model for personalized chemosensitivity prediction. a , Survival curves are shown for a hypothetical patient with HR+/HER2 −- breast cancer to illustrate baseline risk with endocrine therapy (ET, green line) and residual risk with chemoendocrine therapy (CET, gold line). The difference is the personalized chemotherapy benefit (black arrow). b , Three scenarios for how prognosis and chemotherapy benefit predictions could provide personally tailored therapeutic decision support for patients with HR+/HER2 −- breast cancer: (1) low risk and low-benefit, for whom ET is sufficient; (2) high risk and high-benefit, for whom CET is warranted; (3) high risk but low-benefit from standard chemotherapy, for whom escalation with additional agents may be required. c , Development and evaluation cohorts. Marker area is proportional to cohort size. Blue indicates the development set (12 cohorts, n = 9,141) and purple the held-out evaluation set (5 cohorts, n = 1,994). d , Pathology foundation model training. The foundation model learns general-purpose representations of tissue morphology through self-supervised pretraining on over two billion H&E image patches. Training follows a similar procedure to DINOv2 [ 61 ] , a state-of-the-art self-supervised learning method in which the model is taught to produce consistent representations of the same tissue region viewed under different augmentations, using a giant vision transformer (ViT-g). Falcon, the pathology foundation model used here, is a successor to Kestrel [ 41 ] . e , Causal AI model training. A counterfactual regression network (CFRNet [ 59 , 60 ] ) uses a maximum mean discrepancy penalty to balance the representation distributions of the ET and CET treatment groups, then predicts recurrence separately under each treatment. f , Architecture of the multi-modal AI. Whole-slide images are divided into 224 ×\times 224 patches, encoded by the pathology foundation model, and aggregated by multiple instance learning into a single slide-level embedding; clinical features are encoded separately by a tabular residual network. The two embeddings are fused with a gated attention layer and passed to CFRNet, the causal model. The causal model outputs CTX-prognostic — the predicted risk of recurrence under the treatment a patient actually received, baseline risk under ET or residual risk under CET — and CTX- τ\tau , the predicted increase in recurrence-free probability with CET versus ET, which is the personalized estimate of chemotherapy benefit.
Figure 2: Evaluation of CTX as a predictor of prognosis and chemosensitivity. Evaluations performed in the evaluation set (n=1,994). a , Calibration plot compares CTX-prognostic predictions versus observed event probability for five-year recurrence-free interval (RFI). Circles denote observed event probabilities within binned strata of predicted risk, with vertical error bars representing 95% CIs; the diagonal indicates perfect calibration. b, c , Kaplan–Meier curves for the probability of meeting the RFI endpoint in the CTX- τ\tau predicted low-benefit ( b ) and high-benefit ( c ) groups, stratified by adjuvant therapy (chemoendocrine therapy [CET, gold] vs. endocrine therapy [ET, green]), with risk tables underneath. Risk tables report IPW-adjusted numbers at risk, not raw subgroup sizes. Within each subgroup, the HR, 95% CI, and p-value are from an IPW-adjusted Cox model containing chemotherapy treatment status as the only covariate, fit to that subgroup; the p-value is from the score test. d , Forest plot of the hazard ratio (HR) for CTX- τ\tau , chemotherapy treatment status (ET or CET), and the interaction of chemotherapy and CTX- τ\tau in an IPW-adjusted Cox model. The center box indicates the HR, and error bars represent 95% confidence intervals (CIs). p-values are indicated for the treatment-by-biomarker interaction term. e , HR for chemotherapy benefit across the range of continuous CTX- τ\tau , with 95% CI (shaded region). Estimates were derived by the delta method applied to an IPW-adjusted Cox model including an interaction term between continuous CTX- τ\tau score and chemotherapy treatment, with robust standard errors. The dashed reference line indicates no chemotherapy benefit (HR = 1). f , Lollipop plot comparing the predictive performance of clinical, pathological, molecular, and CTX- τ\tau for five-year RFI. For each marker, the vertical axis shows the treatment-by-biomarker interaction p-value from a Cox model containing the biomarker, chemotherapy treatment status, and their interaction. The dashed horizontal line indicates p = 0.05. Points are colored by feature category: patient (green), clinical (purple), molecular (blue), and CTX (gold).
Figure 3: CTX separates chemosensitivity from risk to enable personalized therapeutic decision support. a , Scatter plots compare CTX- τ\tau against four risk estimates: CTX-prognostic, tumor size, number of positive lymph nodes, and Oncotype DX recurrence score. The blue line indicates the median (50th-percentile) quantile regression fit, with the shaded ribbon representing the interquartile range (25th–75th percentile) band. Pearson correlation coefficients (r) are shown for each comparison. b , Predicted five-year recurrence rate as a function of the proportion of patients receiving chemotherapy (CET rather than ET). Patients are ranked according to those most likely to benefit from chemotherapy, as judged by CTX- τ\tau predictions. The curve shows the IPW-adjusted Kaplan–Meier estimated outcomes if the proportion of patients receiving chemotherapy increases from 0% to 60%, starting with the patient with the greatest predicted benefit. The x-axis ends at 60%, within one standard deviation of the mean observed treatment rate across cohorts. The proportion treated with CET and recurrence rate is indicated for the observed population (grey circle). The arrows indicate the same values for CTX- τ\tau -guided strategies reducing chemotherapy allocation while matching the recurrence-free rate (horizontal arrow), or improving recurrence-free rate while matching the observed treatment rate (vertical arrow).
Figure 4: Personalized predictions capture heterogeneity obscured at the (sub)population level. a , Illustration of the hierarchy of populations and subpopulations from which we may estimate prognosis and chemotherapy benefit. Estimates can be derived from the full population (left), subpopulations defined by clinicopathological features (middle), or the individual patient (right). Moving from left to right narrows the group being characterized, recovering precision that coarser estimates average away. b , Full population IPW-adjusted Kaplan–Meier (KM) curves estimating the probability of recurrence in the evaluation set (n = 1,994), stratified by adjuvant therapy status (chemoendocrine therapy [CET] in gold vs. endocrine therapy [ET] in green). Average benefit is the difference in the KM-estimated five-year recurrence probability between the two arms (ET minus CET), with positive values denoting lower recurrence with the addition of chemotherapy. The average benefit across the entire evaluation set is 0.30%. c , Subpopulations defined by clinicopathological features: node-positive disease, large tumor (tumor size ≥\geq median), and Oncotype DX category (high-risk RS ≥\geq 26, low- and intermediate-risk RS < 26, the split used for chemotherapy allocation). d , Case study 1. Patient 1 and 2 are both premenopausal, node-positive with large tumors. The CTX-predicted probability of recurrence under ET and under CET are annotated. CTX recurrence probabilities assigned both patients a high baseline risk under ET, in agreement with standard clinicopathological risk indicators. However, CTX- τ\tau separated them, predicting 0.71% benefit for patient 1 and 4.41% benefit for patient 2. In practice, both patients received CET, but their outcomes differed. Patient 1 recurred at approximately 3 years, whereas Patient 2 remained recurrence-free (censored at approximately 3.5 years). These patient outcomes are aligned with their CTX- τ\tau scores, which predicted patient 1 to be significantly less chemosensitive than patient 2. e , Case study 2. Patient 3 and 4 both have node-negative disease and intermediate Oncotype DX recurrence scores. CTX predicted patient 3 to be at low risk with negligible chemotherapy benefit (CTX- τ\tau = 0.17%) and patient 4 to be at high risk with substantial benefit (CTX- τ\tau = 3.28%). The treatment each patient received ran counter to these predictions: patient 3 received CET and patient 4 received ET alone. Patient 3 remained recurrence-free across approximately 17 years of follow-up, suggesting that chemotherapy de-escalation might have been safe — as predicted by CTX. Patient 4 recurred at approximately 4.8 years, suggesting that chemotherapy escalation may have been prudent — as predicted by CTX. In both case studies, patients who are indistinguishable based on the clinicopathological features experienced divergent outcomes which were tracked by divergent CTX predictions. Panels b – e share a common y-axis (probability of recurrence) and x-axis (time since diagnosis, years). Individual counterfactual outcomes are never observed, so these examples illustrate the interpretation of CTX- τ\tau at the level of the individual patient rather than constituting a formal evaluation of predictive accuracy; the latter is presented in Figure 2 .
Figure 5: Morphological and molecular correlates of CTX- τ\tau predictions. a , Two-dimensional UMAP projection [ 74 ] of embeddings from the pathology foundation model, Falcon, for a cohort in the evaluation set (UNIRAD, n = 362). Each point represents a single H&E image patch. Patches were partitioned into 9 clusters by kk -means applied in the embedding space. Clusters are colored by their average CTX- τ\tau score, from low- (dark blue) to high- (bright yellow) predicted chemotherapy benefit. b , Representative H&E patches selected from clusters in a , ordered by average CTX- τ\tau score (low to high). c , Bar plot shows gene-set enrichment analysis [ 75 ] of MSigDB Hallmark pathways for CTX- τ\tau low- versus high-benefit tumors in the TCGA breast cancer cohort (n=348). Bars show normalized enrichment scores. Only pathways significant at false discovery rate (FDR) < 0.05 are shown. d , Pathways enriched in CTX- τ\tau low-benefit tumors include early estrogen response, epithelial-mesenchymal transition, and angiogenesis. Pathways enriched in CTX- τ\tau high-benefit tumors include proliferative (E2F targets, G2M checkpoint, MYC targets, MTORC1 signaling), DNA damage response (DNA repair), immune activation (IFN- α\alpha and IFN- γ\gamma response), and cGAS-STING signatures.
Figure 6: Zero-shot transfer of CTX prognosis and chemosensitivity predictions across cancer types. The frozen CTX model, trained only on breast cancer, was applied without fine-tuning or re-calibration to patients spanning 16 cancer types from The Cancer Genome Atlas (TCGA), using overall survival as the endpoint. Performance of CTX in the evaluation set is also displayed (as “breast”) for reference. Each cancer type is colored by adjuvant systemic therapy class — anthracycline/platinum/taxane (gold), alkylator (purple), or other cytotoxic (light grey); targeted and/or immunotherapy (blue); or none (green). CTX was trained to predict benefit from anthracycline-, platinum- and/or taxane-based regimens, therefore prognostic and predictive transfer is expected primarily in cancer types treated with mechanistically comparable cytotoxic regimens. a , Box plot shows prognostic transfer of CTX-risk, assessed independently in each cancer type by univariable Cox regression (n = 6,692 patients across 16 cancer types, and n = 1,994 CTX breast evaluation set). Points show the hazard ratio for CTX-risk per cancer type; vertical bars are 95% CIs. The horizontal dotted line marks the null (hazard ratio = 1). b , Bar plot shows predictive transfer of CTX- τ\tau , evaluated in the cancer types routinely treated with adjuvant systemic therapy (n = 5,783 patients across 16 cancer types, and n = 1,994 CTX breast evaluation set). Within each cancer type, patients were ranked by CTX- τ\tau and split into high- and low-benefit subgroups at the 90th percentile; in an IPW-adjusted cohort we computed the difference in restricted mean survival time (RMST) between chemotherapy-treated and untreated patients within each subgroup. Each bar is the high-benefit minus low-benefit Δ\Delta RMST (years). Positive Δ\Delta RMST values indicate that the subgroup CTX- τ\tau predicts will benefit most gained more survival time from chemotherapy than the low-benefit subgroup (correct predictive ordering); negative values indicate the reverse.
Extended Data Fig. 1 : Propensity adjustment. a , Histogram of propensity score estimates colored by treatment status. Scores are clipped so that no patient has an inverse propensity weight (IPW) greater than 10. b , Standardized mean differences between chemoendocrine and endocrine therapy patients, before and after IPW-adjustment.
Extended Data Fig. 2 : Pathology foundation model benchmarking. The pathology foundation model used by CTX ( Figure 1 d ), Falcon, is a ViT-Giant (1.1B parameters) pretrained with DINOv2 [ 61 ] on over two billion H&E patches from 180,000 whole-slide images. Falcon was benchmarked on a slide-level task suite against its predecessor Kestrel [ 41 ] (ViT-Large, 303M parameters) and two publicly available pathology foundation models, UNI 2 [ 42 ] (681M parameters) and Prov-GigaPath [ 43 ] (1.1B parameters). No pretraining slide was used in any benchmark task, and no benchmark task overlapped with any datasets used for evaluating CTX. a , Benchmarking protocol. Each foundation model extracts a patch-level embedding for all patches in each slide. All embeddings from a slide are mean-pooled, and this mean-pooled embedding is used as a feature vector to evaluate the performance of the foundation model on survival or classification tasks. b , Aggregate performance on a suite of 24 slide-level histopathology evaluation tasks. Performance is the C-index for survival tasks and AUC for mutation and biomarker tasks. Falcon achieved the highest mean performance across all tasks. The gain over Kestrel, trained using similar protocol on an order of magnitude fewer data, indicates that generalization performance scales with model capacity and training duration. c , Per-task slide-level performance underlying b . Axis labels describe the cohort, and the target for classification tasks; in the case of survival tasks, the target was disease-specific survival. d , Performance of Falcon on a suite of 24 patch-level histopathology evaluation tasks. Hyperparameters for Falcon were tuned by optimizing performance on this task suite. Each task tests the model’s ability to identify a specific property of cancerous tissue. All tasks are classification tasks, except for BreastPathQ and CoNIC which are regression tasks. The performance of the final Falcon model is shown.
Extended Data Fig. 3 : Evaluation of CTX as a prognostic marker. CTX-prognostic predictions for recurrence-free interval (RFI) in the evaluation set (n = 1,994). a , Receiver operating characteristic (ROC) curves illustrate the sensitivity and specificity of CTX-prognostic for five-year (blue) and 10-year (red) RFI. b , Forest plot shows the adjusted hazard ratio (HR) for CTX-prognostic in a multivariable Cox model adjusting for age at diagnosis, tumor stage, and nodal stage, at the five-year (blue) and 10-year (red) horizons. The center box indicates the HR, and error bars represent 95% confidence intervals (CIs). p-values are indicated for each predictor in the multivariable models.
Extended Data Fig. 4 : Evaluation of CTX as a predictive marker of chemotherapy benefit. Analyses performed in the evaluation set (n = 1,994) unless otherwise stated. Forest plots show the hazard ratio (HR; center box) with 95% confidence intervals (CIs; error bars) for each covariate. a , Treatment-by-biomarker interaction for chemotherapy and CTX- τ\tau in an unweighted multivariable Cox model adjusting for CTX- τ\tau category (low- vs. high-benefit), adjuvant treatment (endocrine therapy [ET] vs. chemoendocrine therapy [CET]), age at diagnosis, tumor stage, nodal stage, grade, and histological subtype (invasive ductal carcinoma [IDC], invasive lobular carcinoma [ILC]), together with the CTX- τ\tau and chemotherapy interaction term (CTX- τ\tau :CET). The interaction HR quantifies how much the effect of chemotherapy differs between low- and high-benefit patients. b , Ablation analysis quantifying the contribution of each input modality to the predictive signal. CTX- τ\tau scores from the clinical-only and pathology-only models were each dichotomized at the 2% threshold and entered into a single IPW-adjusted Cox model with chemotherapy treatment status and both treatment-by-biomarker interactions (CTX- τ\tau clinical:CET, CTX- τ\tau pathology:CET). The likelihood ratio test (LRT) p-value compares this model against a nested model omitting the pathology interaction, testing whether histopathology adds predictive information beyond clinical features. c , Head-to-head comparison of the strength of treatment-effect modification by CTX- τ\tau and the Oncotype DX recurrence score, in the subset of patients with a recorded Oncotype DX score (n = 983). Bars show the treatment-by-biomarker interaction p-value from an IPW-adjusted Cox model containing the standardized biomarker, chemotherapy treatment status, and their interaction, fit separately for each biomarker in the same patients. The dashed horizontal line indicates p = 0.05.
Extended Data Fig. 5 : Patch-level annotation of morphological correlates of CTX- τ\tau predictions. Representative H&E image patches (224 ×\times 224 pixels, 0.5 µm per pixel) from each of the nine kk -means clusters shown in Figure 5 a , derived from Falcon patch embeddings in the UNIRAD evaluation cohort (n = 362). Columns correspond to clusters, ordered left to right by increasing cluster-level CTX- τ\tau coefficient, from clusters enriched in CTX- τ\tau low-benefit tumors (dark blue) to those enriched in CTX- τ\tau high-benefit tumors (bright yellow); the colored square above each column matches the cluster coloring in Figure 5 a . Five randomly sampled patches are shown per cluster. The label beneath each patch gives the dominant morphological pattern annotated by a board-certified pathologist with subspecialty expertise in breast pathology. Clusters associated with low predicted benefit are dominated by fibrotic collagen, adipose tissue, normal ducts, and lymphocytes, with sparse invasive carcinoma; clusters associated with high predicted benefit consist predominantly of nests of invasive carcinoma with high cellularity, alone or admixed with lymphocytes.
Extended Data Fig. 6 : Data filtering flow diagrams. Data filtering flow diagrams show the sequential inclusion and exclusion criteria applied to assemble the development set (left, 12 cohorts) and the held-out evaluation set (right, 5 cohorts). Blue boxes indicate the number of patients retained at each step. Orange boxes indicate the number excluded and the reason for exclusion. Of 23,912 patients in the development cohorts, 9,141 with at least one whole-slide image, a known recurrence endpoint, early-stage female breast cancer, no prior breast cancer history, and known adjuvant chemotherapy status were retained for model development. Development cohorts were deliberately not restricted by receptor subtype or treatment. Of 2,877 patients in the evaluation cohorts, 1,994 were retained after additionally restricting to the intended deployment population — HR+/HER2 −- disease, receipt of adjuvant endocrine therapy (ET) with known adjuvant chemotherapy status, no adjuvant targeted therapy — and excluding slides that failed quality review.
CharacteristicDevelopment cohort (N=9141)Evaluation cohort (N=1994)
Age at diagnosis , median [iqr]56.0 [47.0–65.1]57.0 [49.0–65.0]
Race  
Asian1181 (12.92%)26 (1.30%)
Black or African American96 (1.05%)154 (7.72%)
White1114 (12.19%)331 (16.60%)
Other/Unknown6750 (73.84%)1483 (74.37%)
Extended Data Table 1 : Baseline characteristics of the development and evaluation cohorts.
HyperparameterSearch space
Optimization
Modality branch weights ( λ1,λ2\lambda_{1},\lambda_{2} )Uniform (0.0, 1.0)
Elastic-net mixing weight ( γ\gamma )Uniform (0.0, 1.0)
Elastic-net regularization weight ( α\alpha )Log-uniform ( 10−610^{-6} , 10−310^{-3} )
Learning rateLog-uniform ( 10−310^{-3} , 3×10−23\times 10^{-2} )
Balancing penalty (MMD)
Extended Data Table 3 : CTX hyperparameter search space. Continuous parameters were sampled from Uniform or Log-uniform distributions over the specified ranges; discrete sets were sampled uniformly.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

 DordrechtRoswell ParkUChicagoUNIRADKarmanos
Low0 (0%)102 (22%)73 (17%)0 (0%)33 (22%)
Intermediate0 (0%)297 (63%)231 (53%)0 (0%)85 (58%)
High0 (0%)69 (15%)65 (15%)0 (0%)28 (19%)
Not tested582 (100%)0 (0%)66 (15%)362 (100%)1 (1%)
Appendix
Supplemental Table 1: Oncotype DX score category distributions across the evaluation datasets.
 ChallengeOur solution
1Only unlabelled H&E WSIs are commonly available, so robust and generalizable representations must be learned without supervisionA multi-modal architecture built on a self-supervised pathology foundation model ( Sections A.3.1 and 4.2 )
2Treatment assignment is not randomized in observational data, so patients who received chemotherapy differ systematically from those who did notTreatment-specific output heads and a between-arm balancing penalty that improve extrapolation across treatment groups ( Sections A.3.1 and A.3.2 )
3The outcome is a censored, time-to-event endpointA tailored training objective ( Section A.3.2 )
4The data are pooled across heterogeneous cohortsMulti-source cross-validation ( Section A.3.3 )
Appendix
Supplemental Table 2: Methodological challenges and solutions. H&E, hematoxylin and eosin; WSI, whole slide image.
Supplemental Fig. 1 : Architecture of the proposed Multi-modal model for estimating chemotherapy benefit. Our model takes two input modalities for each patient: clinical variables and histopathology image. The clinical variables are encoded into a clinical embedding using a tabular ResNet, while patches of the image are converted into patch embeddings and aggregated into a slide-level embedding through gated attention. Then the clinical and slide representations are fused through gated attention, and passed to output heads estimating recurrence-free probability under chemoendocrine therapy s^(1)\hat{s}^{(1)} and endocrine therapy alone s^(0)\hat{s}^{(0)} . The predicted chemotherapy benefit ( τ^\hat{\tau} ) is the difference s^(1)−s^(0)\hat{s}^{(1)}-\hat{s}^{(0)} . Auxiliary output heads of the clinical and pathology branches are omitted for clarity. FC denotes a single fully connected (linear) layer, whereas the MLP consists of two fully connected layers with a ReLU activation between them.
More in this topic

Cancer Drug Response Prediction

See all 21 papers

Explore similar work

CardsList
Jun 1, 2026

Multi-Modal Machine Learning for Breast Cancer Recurrence Prediction

Breast cancer recurrence, a leading cause of long-term mortality among survivors, requires timely and accurate risk assessment to guide follow-up care and treatment planning. Traditional predictive models, often limited to either structured or unstructured data alone, struggle to capture the full clinical context. This study examines the impact of integrating multi-modal clinical data, including treatment records, pathology reports, and clinician notes, on recurrence prediction. By integrating a rule-based regular expression extraction mechanism with a rigorous precedence-based conflict reconciliation strategy, our approach effectively recovers definitive tumor characteristics from free-text pathology narratives to augment structured records. We also benchmark performance against commonly used feature sets from prior breast cancer studies to assess the added value of multi-modal integration. Single-source and multi-modal inputs are evaluated across a range of machine learning models. Results show that multi-modal integration consistently improves predictive accuracy compared to single-modal methods.

Jiahao Shao, Xudong Wang, Anam Nawaz Khan +3

Department of Industrial & Systems Engineering, The University of Tennessee, Knoxville, TN 37996, USA · The University of Tennessee Medical Center, Knoxville, TN 37920, USA

Breast CancerClinical Outcome PredictionMultimodal Learning
AskPDF
Mar 12, 2026

Leveraging Large Language Models and Survival Analysis for Early Prediction of Chemotherapy Outcomes

Chemotherapy for cancer treatment is costly and accompanied by severe side effects, highlighting the critical need for early prediction of treatment outcomes to improve patient management and informed decision-making. Predictive models for chemotherapy outcomes using real-world data face challenges, including the absence of explicit phenotypes and treatment outcome labels such as cancer progression and toxicity. This study addresses these challenges by employing Large Language Models (LLMs) and ontology-based techniques for phenotypes and outcome label extraction from patient notes. We focused on one of the most frequently occurring cancers, breast cancer, due to its high prevalence and significant variability in patient response to treatment, making it a critical area for improving predictive modeling. The dataset included features such as vitals, demographics, staging, biomarkers, and performance scales. Drug regimens and their combinations were extracted from the chemotherapy plans in the EMR data and shortlisted based on NCCN guidelines, verified with NIH standards, and analyzed through survival modeling. The proposed approach significantly reduced phenotypes sparsity and improved predictive accuracy. Random Survival Forest was used to predict time-to-failure, achieving a C-index of 73%, and utilized as a classifier at a specific time point to predict treatment outcomes, with accuracy and F1 scores above 70%. The outcome probabilities were validated for reliability by calibration curves. We extended our approach to four other cancer types. This research highlights the potential of early prediction of treatment outcomes using LLM-based clinical data extraction enabling personalized treatment plans with better patient outcomes.

Muhammad Faisal Shahid, Asad Afzal, Abdullah Faiz +5

CureMD Research, 80 Pine St 21st Floor, New York, NY 10005, United States

Clinical Outcome PredictionCancer Survival PredictionBreast Cancer
AskPDF
Aug 21, 2026

CHIMERA Challenge: Biochemical Recurrence Prediction in Prostate Cancer Patients using multimodal datasets

Biochemical recurrence (BCR), defined as any detectable prostate-specific antigen level after prostatectomy with confirmatory elevation, is widely used as a surrogate endpoint and typically assessed using clinical and pathological variables. Currently, no standardized benchmark exists for multimodal prognostic modeling in urological cancers, partly because curating heterogeneous multimodal data remains challenging. We developed the CHIMERA Challenge, a multimodal benchmark integrating preoperative mpMRI, post-prostatectomy histopathology, patient characteristics, and clinician-derived variables from 267 patients across two institutions. The dataset comprises 801 MRI sequences, 13 clinical variables per case, and 942 WSIs. Training (n=95), validation (n=23), and test (n=149) splits were established and hosted on the Grand Challenge platform. Baseline clinical and pathological characteristics did not differ significantly across splits. Models were evaluated on predicting time to BCR using the C-index. Post-challenge analyses tested how each model type performed when clinician-derived variables were withheld or randomized. Unimodal clinical models achieved the highest test C-index of 0.7402 but proved sensitive to the integrity of these variables, with performance collapsing toward chance (C approximately 0.50) when they were randomized. Multimodal models retained near-baseline performance when these variables were withheld (delta C at most 0.04), indicating their ability to recover prognostic signal directly from imaging data. CHIMERA is the first public, standardized multimodal benchmark for prostate cancer prognosis. Although models using only patient characteristics and clinician-derived variables yielded the highest leaderboard performance, multimodal models demonstrated greater robustness in clinically realistic scenarios where complete expert annotation is not guaranteed.

Robert N. Spaans, Catherine Chia, Tongjie Wang +7

Radboud University Medical Center, Department of Pathology, Research Institute for Medical Innovation, Nijmegen, The Netherlands · Canisius Wilhelmina Hospital, Department of Urology, Nijmegen, The Netherlands · Oncode Institute, Utrecht, The Netherlands +6

Clinical Outcome PredictionMedical Image BenchmarksCancer Survival PredictionMultimodal Survival AnalysisMultimodal Robustness
AskPDF