When Slides Met Scans: Cross-Attention for Renal Cancer Prognosis

Anusha Krishnamoorthy; Basudha Pal

eess.IVOct 26, 2026

When Slides Met Scans: Cross-Attention for Renal Cancer Prognosis

Authors: Anusha Krishnamoorthy, Basudha Pal

Organizations: Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology & Emory University · Department of Electrical & Computer Engineering, Johns Hopkins University

Abstract

Histopathology and computed tomography (CT) provide complementary views of renal cancer. We connect these modalities through bidirectional cross-attention between 16 histology prototypes and up to 32 kidney-centered CT tokens from frozen foundation models. In 210 TCGA-KIRC patients with 46 disease-specific survival (DSS) events, 5-fold evaluation yielded a C-index of $0.760\pm0.081$ and 3-year AUROC of 0.773 (95% CI 0.678--0.860), the highest averages among evaluated unimodal and fusion baselines. External testing in 180 CPTAC-CCRCC patients yielded DSS C-index of 0.629 with CT withheld. Attribution analysis linked predictions to image regions while exposing acquisition artifacts. These findings support further study of cross-scale fusion and independent multimodal validation.

Figures & tables

Figure 2: (A) Out-of-fold histology/CT risk percentiles, colored by fusion percentile; triangles denote DSS deaths. (B) In-sample IG decomposition into reference and modality contributions. (C) Prototype scores: red raises risk and blue lowers it.

Method	TCGA-KIRC	CPTAC-CCRCC
ABMIL	$0.684\pm 0.06$	$0.613\pm 0.06$
TransMIL	$0.671\pm 0.10$	$0.639\pm 0.04$
DSMIL	$0.500\pm 0.00$	$0.500\pm 0.00$
AttnMISL	$0.649\pm 0.09$	$0.608\pm 0.06$
ILRA	$0.637\pm 0.14$	$0.611\pm 0.03$
DeepSets	$0.715\pm 0.08$	$0.634\pm 0.01$

Table 1: DSS concordance comparison. Published histology-only results are reproduced from PANTHER [ 1 ] (mean

\pm

SD). Bold numbers indicate the highest reported mean in each column. Published results use UNI and different TCGA cohorts and splits, making comparisons contextual.

Model	C-index	3-year AUROC [95% CI]	Sens., 80% spec.	Sens., 90% spec.
CT tokens + positions	$0.671\pm 0.130$	0.675 [0.564, 0.782]	0.416	0.166
Histology tokens	$0.706\pm 0.111$	0.723 [0.620, 0.825]	0.570	0.443
Histology summary + MLP	$0.730\pm 0.083$	0.718 [0.622, 0.814]	0.448	0.418
Early fusion, ridge Cox	$0.688\pm 0.115$	0.711 [0.610, 0.814]	0.447	0.190
Early fusion, MLP	$0.744\pm 0.110$	0.738 [0.639, 0.836]	0.569	0.448
Late fusion, equal risks	$0.736\pm 0.116$	0.743 [0.640, 0.845]	0.534	0.440

Table 2: TCGA-KIRC DSS prediction (

n=210

). C-index is the 5-fold mean

\pm

SD. 3-year metrics use censoring-adjusted, pooled out-of-fold predictions. Brackets indicate 95% CIs and bold indicates highest means.

Figure 3: (A) CT token IG at the largest tumor cross-section, with shared scaling and kidney/tumor/cyst contours. (B) Prototype projections and patches with slide-specific scales. The blue ring is ink. Attributions are in-sample and percentiles out-of-fold. Small residuals reflect rounding and numerical integration.

Cancer Survival Prediction

Cancer survival prediction estimates patient outcomes and time to events from clinical, molecular, and medical-imaging data.

See all 92 papers

Cross-Modal Contrastive Learning from Histopathology and CT for Automated Renal Cell Carcinoma Grading
Sep 22, 2026Amit Das, Tanmay Shukla, Naofumi Tomita +9Medical Image Classification Oncology
Department of Computer Science, Dartmouth College, Hanover, NH 03755, USA · Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Hanover, NH 03755, USA · Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA +6
Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification
Oct 9, 2026Yuan Liang, Morgan May, Abraham Campbell +3Medical Image Classification Probability Calibration
Research Ireland Centre for Research Training in Machine Learning, ML-Labs, Dublin, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland · School of Computer Science, Technological University Dublin, Dublin, Ireland +2
MIST: Multimodal Survival Prediction with Genomic-Guided Histology Attention
Sep 18, 2026Muhammet Sami Yavuz, Sabri Mustafa Kahya, Richard R. Chen +2Cancer Survival Prediction Multimodal Survival Analysis
AI for Image-Guided Diagnosis and Therapy, School of Medicine and Health, Technical University of Munich (TUM), Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Technical University of Munich (TUM), Munich, Germany +2

Sep 22, 2026J/K move · Enter open · S save

Cross-Modal Contrastive Learning from Histopathology and CT for Automated Renal Cell Carcinoma Grading

Amit Das, Tanmay Shukla, Naofumi Tomita, Faraz Farhadi+8

Department of Computer Science, Dartmouth College, Hanover, NH 03755, USA · Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Hanover, NH 03755, USA · Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA+6

Medical Image Classification Oncology Multimodal Contrastive Learning Computed Tomography

Open paper AskPDF arXiv

Background: Clear cell renal cell carcinoma (ccRCC) exhibits substantial clinical heterogeneity, and accurate grade assessment is essential for risk stratification and treatment planning. However, conventional grading requires invasive tissue sampling. We developed RCC-Align, a cross-modal contrastive learning framework that leverages paired histopathology and computed tomography (CT) data during training to improve noninvasive CT-based ccRCC grade prediction. Methods: RCC-Align aligns paired whole-slide histopathology images (WSIs) and CT scans through contrastive cross-modal objectives, transferring grade-discriminative information from microscopic tissue morphology to macroscopic radiologic representations. The framework was trained and evaluated on paired TCGA and CPTAC cohorts using patient-level five-fold cross-validation. Performance for low- versus high-grade ccRCC classification was compared against CT-only baselines (DINOv2-Base and DINOv2-Finetuned) and a WSI-based reference model (GigaPath-Finetuned). Cross-modal alignment was assessed using cosine similarity analysis. Results: RCC-Align achieved an AUC of 0.601 (95% CI, 0.524-0.673) and AUPRC of 0.599 (95% CI, 0.541-0.676), outperforming DINOv2-Finetuned (AUC 0.545; AUPRC 0.543) with significantly improved low-grade prediction (p = 0.004). RCC-Align also demonstrated stronger paired WSI-CT embedding alignment compared with baselines. The WSI-based GigaPath reference achieved an AUC of 0.719. Conclusion: Pathology-guided contrastive learning improves CT-based ccRCC grading while requiring only CT at inference. This approach may complement tissue diagnosis when biopsy is unsafe, infeasible, or limited by intratumoral heterogeneity. Validation in larger, multi-institutional cohorts with external testing is needed before clinical translation.

Abstract

Cancer Survival Prediction

Explore similar work

Cross-Modal Contrastive Learning from Histopathology and CT for Automated Renal Cell Carcinoma Grading

Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification

MIST: Multimodal Survival Prediction with Genomic-Guided Histology Attention