cs.AIOct 2, 2026

Transcriptome-informed multi-modal AI for predicting neoadjuvant therapy response from breast cancer biopsies

Authors: Jungkyu Park, Dhruva Biswas, Joseph Cappadona, Cerise Tang, Ken G. Zeng, Bartosz Machura, Chuwen Liu, Paolo Tarantino, 

Organizations: Ataraxis AI, New York, NY, USA. · Breast Oncology Program, Dana-Farber Cancer Institute, Boston, MA, USA. · Harvard Medical School, Boston, MA, USA. · Division of Medical Oncology, Rutgers Cancer Institute, New Brunswick, NJ, USA. · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scarcity of labeled data limits development of deep learning biomarkers in oncology. We develop a two-stage AI model predicting pathological complete response (pCR) to neoadjuvant therapy in breast cancer. The first stage learns the transcriptome from histopathology using 8,742 patients across 32 cancer types, corroborated by pathologist review and spatial agreement with measured expression. This simplifies the second stage to predicting pCR from inferred expression and clinical variables. Developed using 1,080 patients (five cohorts) and evaluated in 1,412 patients (nine cohorts), the model achieves a pooled AUROC of 0.79 (95% CI, 0.73-0.85), discriminating responders within molecular subtypes. It outperforms histopathological biomarkers, remaining stable across intratumoral sampling and with minimal biopsy tissue. Ablations show transcriptome-wide inference improves discrimination over clinical variables alone or one-stage pathology models, and robustness by avoiding genomic assays' gene selection constraints. These results indicate that biologically informed compression may generalize to data-sparse applications in precision oncology.

Figures & tables

Figure 1: Overview of the study and NEO architecture. a , Clinical setting and prediction task. The tumor’s gene expression reflects its biological state, and that state also manifests in tissue morphology. The patient’s tumor is sampled by core needle biopsy and digitized as an H&E-stained whole-slide image (WSI). From this biopsy WSI, it is possible to infer gene expression, as a substrate for predicting the probability of pathological complete response (pCR) to neoadjuvant therapy (NAT). b , Development and evaluation cohorts. Left, the contributing institutions, with marker area proportional to cohort size. Right, two bar charts: cohort size and molecular subtype composition. In the size chart, bar length is the number of patients and the solid segment is those achieving pCR. Throughout, blue denotes the development set (5 cohorts, N = 1,080) and red denotes the held-out evaluation set (9 cohorts, N = 1,412). “Unknown” subtype covers patients whose HER2 status is equivocal or whose receptor status is not recorded. c , Distribution of the number of 224 ×\times 224-pixel patches per slide across all 3,606 slides in both splits (log scale; median 4,123 per slide). d , Cohorts used to build and evaluate MORPHEUS. TCGA supplied the paired H&E slides and bulk RNA-seq on which MORPHEUS was developed; CPTAC was held out and used only for evaluation. Counts are patients, whole-slide images and distinct cancer types. e , MORPHEUS training and inference. Trained on TCGA cases with paired H&E slides and bulk RNA-seq, MORPHEUS infers expression from WSIs alone, requiring no transcriptomic assay at inference. f , NEO inference pipeline. MORPHEUS converts the biopsy WSI into H&E-inferred expression, which is combined with baseline clinical variables and mapped to a calibrated probability of pCR ( Section 4.2 ).
Figure 2: NEO performance on independent evaluation cohorts ( N=1,412N=1{,}412 ). a , Receiver operating characteristic curves for pCR in all patients (blue), TNBC (purple), HER2 ++ (green), and HR ++ /HER2 −- (gold); each curve is the pooled average of the per-cohort curves, and the per-cohort estimates behind it are shown in Figure 1 . b , Observed pCR rate against mean predicted probability in four equal-count bins of predicted probability, for all patients (blue circles), TNBC (purple squares), HER2 ++ (green triangles), and HR ++ /HER2 −- (gold diamonds). c , Adjusted odds ratio of pCR per 0.1-unit increase in the score ( Section 4.11 ), from a Firth-penalized logistic regression within each cohort: one blue circle per cohort, area proportional to its weight, with the pooled estimate as a blue diamond beneath. n=1,031n=1{,}031 patients in seven cohorts; IMPRESS and Providence do not record a required covariate, contribute no complete case, and appear as excluded rows. The axis is logarithmic, and arrowheads mark intervals continuing beyond the axis; exact values are given on the right. d , AUROC of NEO (blue bars) and of four computational TIL biomarkers, the computational tumor-infiltrating lymphocytes, the lymphocytes-to-tumor ratio, the inflamed tumor ratio [ 2 ] , and muTILs [ 66 ] (gray bars, darkest to lightest in that order), by cohort and pooled, over the slides on which all five methods return a value ( n=1,386n=1{,}386 patients; Section A.4.5 ). e , AUROC of NEO (blue bars) and pathologist-assessed Ki-67 (red bars), by cohort and pooled, in the n=237n=237 patients for whom Ki-67 was recorded. Statistics: pooled AUROCs and odds ratios are random-effects estimates over the contributing cohorts ( Section 4.9 ). Error bars are 95% confidence intervals: bootstrap for cohort AUROCs in d , e ; Wilson in b ; Wald for odds ratios within cohort in c ; and Hartung–Knapp for pooled estimates in c – e . Dashed lines mark the null reference: chance discrimination in a , optimal calibration in b , OR =1=1 in c , AUROC =0.5=0.5 in d , e . The pp value in c is from the random-effects model; paired comparisons between methods in d and e , with their pp values, are given in Figure 2 a–e.
Figure 3: Transcriptional programs recovered from H&E by MORPHEUS. a , Hallmark pathway enrichment of MORPHEUS prediction fidelity within each TCGA project. Columns, the 23 projects with at least 20 validation patients ( Table 3 ); rows, the 25 hallmark sets of 50 that reached significance in at least six of the 23 projects. Color, normalized enrichment score (NES) from genes ranked by the within-project Pearson correlation between inferred and measured expression; a positive NES indicates that the set’s genes are predicted better than the transcriptome-wide background. White dots, Benjamini–Hochberg adjusted p<0.05p<0.05 within that project. b , Highest- and lowest-scoring patches among the 100 sampled at random from each slide of the Jagiellonian evaluation cohort, for three hallmark programs: epithelial-mesenchymal transition (EMT), angiogenesis, and the mitotic spindle. Each patch was scored by the mean H&E-inferred expression of that hallmark set’s members ( Section 4.13 ). Eight patches per block; high and low are the two ends of the same ranking. The italic label below each block summarizes what two pathologists reported for it ( Section 4.13 ). c , Spatially resolved examples on two publicly available imaging-based spatial transcriptomic slides of human breast cancer, neither seen by MORPHEUS at any stage: one Xenium v1 slide and one Xenium Prime 5K slide, one tissue section each ( Section A.6.2 ). One gene per program, each among the best-inferred members of its set ( Section A.6.2 ): TAGLN (EMT; Xenium v1, 3,614 patches) and IL7R (inflammatory response; Xenium Prime, 8,068 patches). Each slide was tiled into 224 ×\times 224-pixel patches; the measurement paired with a patch is log-transformed transcript abundance normalized to the total transcript count across all measured genes in that patch ( Section A.6.2 ). Left to right, H&E; measured expression; MORPHEUS prediction. Predictions were generated per patch with no spatial supervision, retraining, or fine-tuning of any kind. Both maps show percentile rank within the slide, one cell per patch; ρ\rho is the Spearman correlation between prediction and measurement across those patches ( Section A.6.2 ).
Figure 4: Interpretability of the H&E-inferred expression, and ablations of its contribution to pCR prediction. a , Pooled AUROC for pCR of the transcriptome-informed and direct pCR prediction models, uni-modal (H&E only), and multi-modal (H&E with the clinical variables), alongside a pCR prediction model using the clinical variables alone. Transcriptome-informed pCR prediction models use the H&E-inferred expression to predict pCR. Direct pCR prediction models receive the same foundation-model embeddings that MORPHEUS uses and predict pCR from them directly ( Section A.2.13 ). Error bars are 95% confidence intervals. Brackets give two-sided pp values for pooled paired AUROC differences (per-cohort differences in Figure 2 f–h). Panel a ablates NEO’s neural-network-based pCR prediction model rather than its full ensemble, so that the transcriptome-informed and direct pCR prediction models are neural networks matched in shared hyperparameter ranges and tuning budget. b , Global feature importance of the EBM trained on the clinical variables and the five MORPHEUS-ICs of the H&E-inferred expression (top), and the shape functions of the five MORPHEUS-ICs (bottom). Past the divider, the two bars give the importance of the five MORPHEUS-ICs taken as a group and of the eight clinical variables taken as a group ( Section A.6.1 ). Each shape function shows the amount each MORPHEUS-IC adds to the predicted log-odds of pCR as the component varies, with all other features held fixed. The band is ±\pm 1 s.d. across folds. c , Pooled AUROC of pCR prediction models trained on H&E-inferred expression alone, for gene subsets of increasing size selected four ways: at random (gray; mean and range over 25 seeds), by expression-prediction fidelity (purple), by per-gene pCR discrimination in the cross-validation cohorts (green), and as published clinical or curated signatures (red). Dashed line with star, all 14,773 genes. d , Pooled AUROC of the multi-modal pCR prediction model against the number of components the 2,427-gene H&E-inferred expression is reduced to, for three reduction methods, each fitted on the TCGA validation subset ( Section 4.6 ): ICA (purple), principal component analysis (green), and sparse principal component analysis (gold); the star and the dashed line mark the unreduced representation, all 2,427 standardized genes. AUROCs in a , c , d pool the nine evaluation cohorts ( N=1,412N=1{,}412 ) with a random-effects model ( Section 4.9 ).
Figure 5: Robustness of NEO to tumor sampling bias. a , Whole-slide predictions for every UChicago slide, patients ordered by their mean; the four examples in b are the boxed points, outlined in the same colors. b , Per-core predictions on example UChicago slides along with their whole-slide prediction (given above each column). Top : H&E slide. Bottom : the same field with each core scored independently by NEO, colored on the shared pCR scale with its score overlaid. Each column is outlined in the color of its own whole-slide prediction, keying it to panel a . c , Fraction of total variance attributable to sampling, W/TW/T , against the number of slides averaged, kk , for NEO, an imaging-only variant of NEO, and the four computational TIL biomarkers, on the UChicago multi-slide patients within the shared-slide set ( Section A.4.5 ; n=242n=242 patients with ≥2\geq 2 pre-treatment slides). The dashed line is the stability threshold W/T=0.15W/T=0.15 [ 4 ] ; the slides needed to reach it are given in the legend. d , Dependence of the prediction on the amount of tissue sampled, for the imaging-only variant of NEO shown in c ( Section 4.15 ). For each of the 70 evaluation slides with ≥\geq 10,000 patches, the model was re-run on random subsets of 50 to 5,000 of the slide’s patches (30 subsets per size). For each subset, prediction change was calculated as the absolute difference from the all-patch predicted pCR probability, divided by the all-patch prediction. Thin lines: one per slide, the mean change across its 30 random subsets at each subset size; purple line, median of these per-slide means across slides; dashed line, a 5% change shown for reference. The shaded span marks the patch count of a typical core-needle biopsy slide in our cohorts, its lower edge (400 patches) being about the 1st percentile of per-slide patch counts across our cohorts. The top axis gives the corresponding tissue area (0.0125 mm 2 per patch).
Figure 1: NEO subgroup analyses by cohort and subtype. Model discrimination, measured using area under the receiver operating characteristic curve (AUROC) for pathological complete response (pCR), was assessed within individual evaluation cohorts and molecular subtypes. Cohort estimates are shown as squares and pooled estimates as diamonds. Square area is proportional to the cohort’s random-effects weight; horizontal bars indicate 95% confidence intervals (CIs). The right-hand column reports pCR events and patient counts. Cohorts contributed to pooling only if they contained at least 30 positive–negative patient pairs ( Section 4.9 ); excluded cohorts and reasons are indicated. a , All patients (nine cohorts), individual AUROCs 0.658–0.880 and pooled AUROC 0.792 (95% CI, 0.732–0.852); I2I^{2} = 74%, τ2\tau^{2} = 0.0040. b , Triple-negative breast cancer (TNBC, seven cohorts), individual AUROCs 0.640–0.803 and pooled AUROC 0.699 (95% CI, 0.643–0.755); I2I^{2} = 0%, τ2\tau^{2} = 0. Łódź and Providence were excluded because they contributed only 15 and 22 positive–negative pairs, respectively. c , Human epidermal growth factor receptor 2-positive (HER2 ++ , eight cohorts), individual AUROCs 0.540–0.825 and pooled AUROC 0.669 (95% CI, 0.576–0.763); I2I^{2} = 58%, τ2\tau^{2} = 0.0077. Dordrecht contributed no HER2-positive patients. d , Hormone receptor-positive, HER2-negative (HR ++ /HER2 −- , six cohorts), individual AUROCs 0.583–0.858 and pooled AUROC 0.762 (95% CI, 0.686–0.837); I2I^{2} = 0%, τ2\tau^{2} = 0. Łódź had no pCR events in this subtype. IMPRESS and Dordrecht contribute no patients of this subtype. Statistics: the cohort-level CIs used 2,000 bootstrap resamples. Pooled estimates used random-effects models ( Section 4.9 ). I2I^{2} and τ2\tau^{2} summarize between-cohort heterogeneity.
Figure 2: Ablation studies of NEO and benchmark biomarkers. Each panel compares the area under the receiver operating characteristic curve (AUROC) for predicting pathological complete response (pCR) between the two predictors named in the panel title. Positive differences favor the first predictor and negative differences favor the second. a – e , NEO versus computational tumor-infiltrating lymphocytes (cTILs, a ), lymphocytes-to-tumor ratio (LTR, b ), inflamed tumor ratio (ITR, c ), muTILs ( d ), and Ki-67 ( e ). f , The multi-modal transcriptome-informed pCR prediction model versus the clinical-only baseline. g , h , The transcriptome-informed versus direct pCR prediction models in the uni-modal ( g ) and multi-modal ( h ) settings. Squares show cohort-level differences, with area proportional to the random-effects weight and horizontal bars indicating 95% confidence intervals (CIs) from 2,000 paired patient-level bootstrap resamples. Diamonds show pooled differences from random-effects models across contributing cohorts, with 95% CIs ( Section A.3.2 ). Shaded bands repeat the pooled CIs, and dashed vertical lines mark zero difference. Panel headings report the pooled difference, CI, and two-sided pp value for the null hypothesis of no difference ( Section A.3.2 ). Neural-model comparisons ( f – h ) used 1,412 patients across nine cohorts. Comparisons with computational TIL biomarkers ( a – d ) used the shared-slide population of 1,386 patients across nine cohorts, with all methods scored on identical patients and slides ( Section A.4.5 ). The Ki-67 comparison ( e ) used 237 patients with recorded Ki-67 across three cohorts ( Section A.4.6 ).
Figure 3: NEO and benchmark biomarker subgroup analyses by molecular subtype. Pooled area under the receiver operating characteristic curve (AUROC) for pathological complete response (pCR), comparing NEO with computational tumor-infiltrating lymphocytes (cTILs), lymphocytes-to-tumor ratio (LTR), inflamed tumor ratio (ITR), and muTILs. All methods were evaluated on identical patients and slides within each panel. The overall shared-slide population comprised 1,386 patients ( Section A.4.5 ). Subtype panels include only cohorts meeting the minimum of 30 positive–negative patient pairs ( Section 4.9 ). a , All patients ( n=1,386n=1,386 , nine cohorts). b , Triple-negative breast cancer (TNBC; n=493n=493 , seven cohorts). c , Human epidermal growth factor receptor 2-positive (HER2 ++ ) disease ( n=425n=425 , eight cohorts). d , Hormone receptor-positive, HER2-negative (HR ++ /HER2 −- ) disease ( n=423n=423 , six cohorts). Points show pooled AUROCs from random-effects models across contributing cohorts, with 95% confidence intervals ( Section 4.9 ). The right-hand column reports two-sided pp values from pooled paired AUROC differences between each biomarker and NEO; cohort-level standard errors used 2,000 paired patient-level bootstrap resamples ( Section A.3.2 ). Arrowheads indicate confidence intervals extending beyond the plotted range; their exact bounds are reported in the value column.
Figure 4: Transfer of NEO to residual cancer burden and nodal pCR prediction. The NEO score trained to predict overall pathological complete response (pCR) was evaluated against secondary endpoints without retraining or recalibration. a , NEO score by residual cancer burden (RCB) class in 361 patients across four cohorts. Violin plots span the observed range of scores and the horizontal ticks indicate the median: 0.486, 0.312, 0.259 and 0.139 for RCB-0, RCB-I, RCB-II, and RCB-III. Patient counts are shown below each class. The one-sided pp value tests decreasing NEO scores with increasing RCB using a cohort-stratified Jonckheere–Terpstra test ( Section A.7.1 ). b , Area under the receiver operating characteristic curve (AUROC) for RCB-0/I versus RCB-II/III in the same 361 patients. Squares show cohort estimates, with area proportional to the random-effects weight. The diamond shows the pooled AUROC of 0.792 (95% CI, 0.656–0.928). The right-hand column gives the counts in the two outcome groups. c , Nodal-pCR analysis among 712 patients with clinical nodal involvement at baseline. Stacked bars show nodal pCR, residual nodal disease, and indeterminate outcomes; blue ticks mark overall pCR among patients with determinable nodal outcomes. Nexus ( n=45n=45 ) and Providence ( n=17n=17 ), marked with asterisks, had only nodal-pCR outcomes and therefore did not meet the minimum of 30 positive–negative patient pairs required for AUROC pooling ( Section 4.9 ). Among the remaining 536 patients across six cohorts, pooled AUROC was 0.736 (95% CI, 0.644–0.829) for nodal pCR and 0.775 (95% CI, 0.632–0.918) for overall pCR. Nodal pCR was defined by documented ypN0 or overall pCR ( Section A.7.2 ). In b and the right side of c , horizontal bars show cohort-level 95% CIs from 2,000 patient-level bootstrap resamples. Diamonds show pooled estimates from random-effects models across contributing cohorts, with 95% CIs ( Section 4.9 ).
Figure 5: MORPHEUS performance by cancer type. Per-gene fidelity of H&E-inferred expression, calculated separately within each cancer type. Plotted as the distribution of Pearson correlation values between inferred and measured expression across the 14,773 genes MORPHEUS predicts. Blue distributions represent the TCGA validation subset; orange distributions represent CPTAC evaluation cohort ( Section 4.12 ). Each column corresponds to one cancer type. The vertical axis shows Pearson correlation, and histogram width represents the number of genes on a single scale shared by every column (scale bar, 1,000 genes), so widths are comparable across types and between cohorts. Columns are labeled by project code, the two joined by a slash where TCGA and CPTAC code one disease differently (KIRC/CCRCC, HNSC/HNSCC, LUSC/LSCC, PAAD/PDA). Only cancer types with at least 20 evaluated cases are shown; cohort composition is reported in Tables 3 and 4 . The TCGA validation subset was used for MORPHEUS model selection, so its fidelity estimates may be optimistic. CPTAC was reserved for independent evaluation. These distributions summarize gene-level fidelity; the correlations for the TCGA validation subset underlie the pathway enrichment analysis in Figure 3 a.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Data filtering flow diagrams. Sequential inclusion and exclusion criteria applied to assemble the development set ( a , five cohorts) and the held-out evaluation set ( b , nine cohorts), following the eligibility criteria of Section 4.1 . Blue boxes indicate the number of patients retained at each step. Orange boxes indicate the number excluded and the reason for exclusion. Of 13,240 patients in the development cohorts, 1,080 were retained for model development: those who received NAT, were non-metastatic at diagnosis, were female, had an assessable pathological response to NAT, and had at least one pre-treatment diagnostic biopsy WSI. Of 6,457 patients in the evaluation cohorts, 1,412 were retained under the identical criteria. Most exclusions occurred at the first step in both sets. Several contributing institutions supplied general breast cancer cohorts, so the majority of their patients were treated in the adjuvant setting and are outside the population addressed here.
CategoryDevelopment cohorts (N=1,080)Evaluation cohorts (N=1,412)
Age median, [IQR]54.00 [45.00–63.00]52.29 [44.00–62.00]
Race
Asian7 (0.65%)10 (0.71%)
Black or African American15 (1.39%)119 (8.43%)
White315 (29.17%)315 (22.31%)
Other/Unknown743 (68.80%)968 (68.56%)
Appendix
Table 1: Characteristics of the development and evaluation cohorts. Age is reported as median [IQR] in years; categorical variables are reported as n (%). Percentages use the total number of patients in each column, except for treatment regimens, which use the number in the corresponding molecular subtype. Molecular subtypes (TNBC, HR ++ /HER2 −- , HER2 ++ ) are derived from receptor status; patients whose subtype cannot be determined are classified as unknown. Treatment regimen groups describe recorded treatment components, not specific protocols or treatment sequences.
CategoryProtegeUChicagoNexusDordrechtIMPRESSJagiellonianShaare ZedekProvidenceŁódź
Patients391289153146126116736751
Age median [IQR]56 [46–65]51 [42–61]50 [44–56]52 [42–61]54 [46–59]50 [39–57]48 [43–59]53 [43–63]63 [53–69]
Race
Asian5 (1%)0 (0%)0 (0%)0 (0%)0 (0%)0 (0%)0 (0%)5 (7%)0 (0%)
Black or African American117 (30%)0 (0%)0 (0%)0 (0%)0 (0%)0 (0%)0 (0%)2 (3%)0 (0%)
White263 (67%)0 (0%)0 (0%)0 (0%)0 (0%)0 (0%)0 (0%)52 (78%)0 (0%)
Appendix
Table 2: Characteristics of the evaluation cohorts, shown separately by cohort. Age is reported as median [IQR] in years; categorical variables are reported as n (%). Percentages use the total number of patients in each column, except for treatment regimens, which use the number in the corresponding molecular subtype. Molecular subtypes (TNBC, HR ++ /HER2 −- , HER2 ++ ) are derived from receptor status; patients whose subtype cannot be determined are classified as unknown. Treatment regimen groups describe recorded treatment components, not specific protocols or treatment sequences. Age summaries and percentages are rounded to whole numbers; abbreviated row labels are given in full in Table 1 . H, trastuzumab; P, pertuzumab; ICI, immune checkpoint inhibitor; chemo, chemotherapy; endo, endocrine therapy.
 TrainingValidation
ProjectPatientsWSIsPatientsWSIs
ACC †46183940
BLCA29333784108
BRCA835886212227
CESC2132205457
CHOL †282877
Appendix
Table 3: MORPHEUS training-validation data composition by TCGA project. Patients and distinct WSIs in the MORPHEUS training and TCGA validation subsets derived from the 80:20 patient-level split of Section 4.4 . † Fewer than 20 validation patients, not visualized in Figure 5 or Figure 3 a.
Cancer typePatientsRNA-seq samplesWSIs
BRCA106106106
CCRCC222314340
COAD104104104
GBM157157210
HNSCC110110142
LSCC108109132
Appendix
Table 4: CPTAC data composition by cancer type. Patients, primary-tumor RNA-seq samples, and WSIs, after restriction to primary tumor. Fidelity is evaluated per RNA-seq sample rather than per patient, because each sample carries its own slides ( Section 4.12 ); 41 patients contribute more than one sample and therefore more than one case. Every cohort exceeds the 20-case threshold, so all 10 appear individually in Figure 5 .
Figure 2: Components of the neural-network-based pCR prediction models and MORPHEUS. Inputs flow from left to right through the color-coded branches to the prediction heads. The pathology branch (lower, purple) begins with the whole-slide image, tiled into 224×224224\times 224 patches and encoded by the frozen Falcon encoder. The pathology branch then aggregates the resulting patch-level embeddings by gated attention ( Equation 1 ). The attention-weighted average passes through dropout and linear projections ( Section A.2.3 ) to width dintd_{\mathrm{int}} , shared between modalities, giving the slide embedding zpathology\mathbf{z}_{\mathrm{pathology}} . The tabular branch (upper lane, green) passes its inputs through a tabular ResNet and a fully-connected layer to the tabular embedding ztabular\mathbf{z}_{\mathrm{tabular}} of the same width dintd_{\mathrm{int}} . Those inputs are the baseline clinical variables, the five MORPHEUS-ICs of the H&E-inferred expression, or both, depending on the model. The fusion module (between the two lanes, orange) belongs to neither branch: it applies a second gated-attention block over the two embeddings and outputs the multi-modal embedding zmultimodal\mathbf{z}_{\mathrm{multimodal}} . A classification prediction head contains one linear layer per binary subtask. The multi-modal direct pCR prediction model uses both branches together with the fusion module, and has prediction heads on each branch and on the fused representation ( Section A.2.7 ). MORPHEUS follows the pathology branch shown here, with a linear expression-regression head in place of the classification head. NEO uses MORPHEUS-derived features together with clinical variables, calibrates each per-fold tabular ResNet and EBM, and averages the calibrated outputs ( Sections 4.8 and A.2.11 ). Hyperparameter ranges for each component, and where dropout sits within each, are given in Section A.2.10 and Table 5 .
Figure 3: Multitask learning. During training, the tabular ResNet was optimized with auxiliary tasks to provide additional supervision. In addition to pCR prediction, the model jointly predicts pathological staging (as an ordinal classification task) and T/N downstaging. The ordinal outcomes are decomposed into cumulative binary subtasks; together with pCR and the two downstaging targets, these yield ten binary subtasks, each trained using two-class softmax outputs and cross-entropy loss.
HyperparameterRange
Shared by all tuned neural networks
Elastic-net regularization weight ( α\alpha )log⁡U(10−6,10−3)\log\mathcal{U}(10^{-6},10^{-3})
Elastic-net mixing weight ( γ\gamma )U(0,1)\mathcal{U}(0,1)
Intermediate feature dimension ( dintd_{\mathrm{int}} )log⁡U(4,512)\log\mathcal{U}(4,512)
Pathology branch
Pathology-branch dropoutU(0,0.3)\mathcal{U}(0,0.3)
Appendix
Table 5: Hyperparameter search space, organized by model component. Each model’s search space is the union of the shared block and the blocks for the components it instantiates ( Section A.2.10 ). MORPHEUS and the uni-modal direct pCR prediction model use the pathology branch, the transcriptome-informed pCR prediction models use the tabular branch, and the multi-modal direct pCR prediction model uses both branches together with the fusion module. During the random-search phase, parameters were sampled independently from the distributions below; subsequent configurations were selected by Bayesian hyperparameter search. U\mathcal{U} denotes a uniform and log⁡U\log\mathcal{U} a log-uniform distribution over the stated range. Draws for integer-valued parameters (intermediate feature dimension, block size) were rounded to the nearest integer, and discrete sets were sampled uniformly. Learning rate and search budget differ between models and are given in the text.
MethodAs publishedRecomputed
TNBC N=64N=64 ( NpCR=27N_{\text{pCR}}=27 )
cTILs0.708 (0.574–0.841)0.691 (0.553–0.820)
LTR0.686 (0.550–0.821)0.583 (0.439–0.727)
ITR0.715 (0.584–0.845)0.568 (0.430–0.713)
HER2 ++ N=62N=62 ( NpCR=38N_{\text{pCR}}=38 )
cTILs0.679 (0.543–0.814)0.623 (0.481–0.760)
Appendix
Table 6: Reproduction of the re-implemented computational TIL biomarkers on IMPRESS. AUROC for pCR (95% bootstrap confidence interval) among the IMPRESS patients of each stratum. As published is the value the originating study reports on these same patients: Aswolinskiy et al. [2] for cTILs, LTR, ITR, and MTR; NEO is shown for reference. Recomputed is our implementation ( Sections A.4.1 and A.4.2 ), scored on the identical patients with the bootstrap of Section 4.9 .
Figure 4: Between-gene redundancy of the gene subsets of Figure 4 c. Redundancy is measured as the mean absolute pairwise Pearson correlation in H&E-inferred expression from the TCGA validation subset ( Section A.2.12 ). Gene lists ranked by expression fidelity (purple) or per-gene pCR AUROC (green) have higher correlations than random gene lists of the same size (gray; mean and range across 25 seeds), with differences narrowing as gene lists approach all 14,773 genes. Red diamonds indicate published signatures; the star and dashed line indicate the full gene set. The ranked subsets accumulate mutually correlated genes, which may help explain their limited gains as gene list size increases ( Figure 4 c).
SignatureSourceTotalRetained
Merck T-cell-inflamed GEPAyers et al. [3]1818
Oncotype DXPaik et al. [79]2120
HER2DXPrat et al. [86]3228
Davoli immune evasionDavoli et al. [23]4239
Danaher immune cell markersDanaher et al. [22]6047
MammaPrintVan’t Veer et al. [106]7050
Appendix
Table 7: Gene signature sources and retained gene counts. “Total” is the published gene list size, including controls where applicable; for MammaPrint it counts reporters rather than unique genes. “Retained” counts unique genes in the 14,773-gene MORPHEUS output after gene symbol mapping, deduplication, and expression filtering.
Figure 5: Pathway analysis of the two most important MORPHEUS-ICs. Gene set enrichment analysis of the hallmark pathway collection over all genes ranked by their weight in the ICA mixing-matrix column of MORPHEUS-IC0 (left) and MORPHEUS-IC2 (right) ( Sections 4.6 and 4.13 ). Bars are the normalized enrichment scores (NES) of the significantly enriched pathways: green, pathways whose genes have a high contribution to the component; red, a low contribution.
Figure 6: Biological correlates of NEO predictions. All analyses use the TCGA-BRCA cohort with MORPHEUS training patients included ( Section A.6.4 ). a , GSEA of the hallmark pathway collection over genes ranked by their association with the NEO score. Left , NES of the significantly enriched pathways; red, enriched in NEO-high tumors; green, in NEO-low tumors. Right , enrichment plots for representative pathways: proliferation (E2F targets, G2M checkpoint) and interferon response, enriched in NEO-high tumors, and early and late estrogen response, enriched in NEO-low tumors. b , Association between immune cell-type enrichment scores and the NEO score (standardized β\beta per cell type); red, positively associated cell types; green, negatively associated. c , Distribution of NEO scores by PAM50 intrinsic subtype.
More in this topic

Clinical Outcome Prediction

Clinical outcome prediction uses patient records, imaging, signals, and biomarkers to estimate risks, disease trajectories, treatment response, and survival.

See all 261 papers

Explore similar work

CardsList
Aug 18, 2026

Neoadjuvant chemotherapy response prediction using pretreatment diffusion and contrast-enhanced magnetic resonance imaging with clinical variables

Prediction of pathological complete response before neoadjuvant chemotherapy may facilitate more tailored therapeutic planning for breast cancer patients. This work proposes a deep-learning model for pretreatment data only, combining apparent diffusion coefficient maps, dynamic contrast-enhanced magnetic resonance imaging, and clinical variables. The study uses the public ACRIN 6698/I-SPY2 multicenter dataset. The architecture employs EfficientNet-B0 pretrained encoders for image feature extraction and late fusion with clinical information. Multiple clinical variables were evaluated, including age, race, histological type, HR/HER2 subtype, SBR grade, and maximum diameter. Only HR/HER2 subtype improved the average area under the receiver operating characteristic curve (AUC) and was retained in the final model. Using stratified five-fold cross-validation, standalone apparent diffusion coefficient maps achieved a mean AUC of 0.79, whereas dynamic contrast-enhanced magnetic resonance imaging achieved 0.74. Adding HR/HER2 subtype improved performance to 0.83 and 0.81, respectively. The final configuration, using both imaging modalities and HR/HER2 subtype, achieved an AUC of 0.86. These results support pretreatment multimodal learning for response prediction, although external validation is required before clinical use.

Pablo García Marcos, Paula Puerta González, Guillermo Lorenzo +6

Computer Science Department, University of Oviedo, Asturias, Spain · Biomedical Engineering Center (BME), University of Oviedo, Asturias, Spain · Electrical Engineering Department, University of Oviedo, Asturias, Spain +7

Cancer Drug Response PredictionClinical Outcome PredictionOncologyMultimodal Learning
AskPDF
Jul 1, 2026

ClinRAG-GRAPH: Clinical-prior Retrieval-Augmented Graph Model with Domain Adversarial Learning for Breast pCR Prediction

Neoadjuvant chemotherapy (NAC) response prediction is clinically important for treatment stratification in breast cancer. However, robust pre-treatment pathological complete response (pCR) prediction remains challenging due to insufficient cross-modal modeling, multicenter imaging heterogeneity, and weak evidence-grounded interpretability. We propose ClinRAG-GRAPH, a Clinically informed Retrieval-Augmented Generation Graph framework, for pre-treatment pCR prediction from DCE-MRI, structured clinical variables, and biopsy-derived pathological biomarkers. ClinRAG-GRAPH constructs an intra-patient clinical-prior graph and applies a prior-guided relation-aware graph convolutional network for structured multimodal representation learning. To improve cross-center robustness, we introduce a dual-branch domain-adversarial learning strategy to suppress protocol-related MRI bias while preserving pCR-relevant features. To enhance interpretability, we further incorporate large language model (LLM)-driven subgraph RAG module that retrieves clinically analogous historical cases and integrates retrieved evidence for pCR inference. We assemble a large-scale multicenter NAC breast cancer cohort for extensive validation, drawing from two public sources and three in-house centers.Results show that ClinRAG-GRAPH achieves AUCs of 0.815 on the internal test set and 0.774/0.712 on two external test sets, demonstrating robust pre-treatment pCR prediction across centers. The code is available at the anonymized https://github.com/miccai26-1181/ClinRAG-GRAPH.

Yaofei Duan, Yuhao Huang, Tianyu Zhang +12

Department of Medical Imaging, Radboud University Medical Center, Nijmegen, The Netherlands · The Netherlands Cancer Institute, Amsterdam, The Netherlands · Faculty of Applied Sciences, Macao Polytechnic University, Macau, China +5

Domain-Adversarial LearningRetrieval-Augmented GenerationClinical Outcome PredictionOncologyMultimodal Graph LearningCancer Drug Response Prediction
AskPDF
May 8, 2026

Multimodal Stepwise Clinically-Guided Attention Learning for Pathological Complete Response Prediction in Breast Cancer

Pathological complete response (pCR) is a key prognostic factor in breast cancer patients undergoing neoadjuvant therapy, strongly associated with long-term survival and treatment personalization. However, accurate pre-treatment pCR prediction remains challenging due to severe class imbalance and limited generalizability across diverse clinical settings. In this work, we propose a multimodal stepwise clinically-guided attention learning framework for pCR prediction from breast magnetic resonance imaging (MRI), designed to address these limitations through medically grounded spatial guidance and multimodal integration. The approach follows a stepwise training strategy inspired by physician reasoning: the model first learns global discriminative imaging patterns, then attention mechanisms are introduced to constrain the network toward tumor regions, and finally clinical variables are integrated to refine decision-making. This guidance strategy encourages prioritization of task-relevant features, improving identification of responders despite their limited representation in the dataset. Moreover, grounding attention in anatomically consistent tumor regions reduces reliance on dataset-specific patterns, thereby enhancing cross-institutional generalization. The framework is evaluated through external validation across heterogeneous MRI cohorts. Compared to non-guided single-stage baselines, the proposed approach improves sensitivity while maintaining competitive specificity, and produces anatomically coherent attention maps that support interpretation of the model's predictions. These findings highlight the potential of clinically-guided multimodal attention learning for robust and generalizable pCR prediction in breast cancer.

Alice Natalina Caragliano, Valerio Guarrasi, Michela Gravina +2

Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy · Department of Electrical Engineering and Information Technology, University of Naples Federico II, Naples, Italy · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden

Breast CancerClinical Outcome PredictionMultimodal ClassificationOncologyExplainable Medical Image Analysis
AskPDF