cs.LGAug 25, 2026
SaveA Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology
Organizations: Tempus AI, Inc., Chicago, IL, USA
Abstract
Precision oncology necessitates a longitudinal model of patient state that captures cancer evolution and treatment over time, integrating multimodal observations. We introduce the oFM, a foundation model developed on a real-world oncology cohort of 1.67 million cancer patients that integrates clinical trajectories with DNA, RNA, and H&E pathology. Patient-level partitions were reserved for training, validation, and testing, with over one million patients used for training. The oFM encodes daily clinical and molecular episodes and, along with pathology images, integrates them over time to produce a patient state embedding. We evaluate frozen oFM embeddings against expert-curated clinical and molecular baseline features. In prognostic benchmarks, the oFM improved AUC for treatment response, progression-free survival, and overall survival (0.774 vs. 0.563 for overall survival). Across 11 comparative-treatment cohorts, the oFM embeddings achieved a three-fold higher pooled and scale-normalized treatment-benefit AUTOC than baseline features with improved benefit ranking in 9 of 11 cohorts, and provided stronger prognostic discrimination within both treatment arms. We also evaluated a mechanism discovery framework that interprets downstream models built on oFM embeddings by linking their predicted outcomes to clinically and biologically grounded mechanisms through an evidence-grounded temporal graph, enabling evaluation in clinical and drug-development applications.
Figures & tables
Figure 1: oFM data and architecture. (a) Longitudinal patient records integrate clinical records (EHR and concepts curated from free-text reports), molecular NGS, IHC and ISH, and H&E pathology data. (b-d) The pretraining cohort is summarized by tumor type, modality coverage, and patient-level data partition. (e-f) Daily clinical and molecular facts are encoded as episode embeddings, while pathology embeddings are projected into the shared latent space. (g) A temporal Transformer integrates date-stamped, modality-tagged tokens to produce the patient representation for downstream prognostic and treatment-benefit modeling.
Figure 2: oFM training curriculum. (a) Stage I pretrains the episode encoder by reconstructing missing tokens in per-day episodes. (b) Stage II freezes the episode and PRISM2 encoders while training the temporal encoder, prediction heads, and projectors. (c) Stage III unfreezes the episode encoder. (d) Anchor-conditioned training compares a student view to a later teacher view, with prediction conditioned on the anchor-day intervention and elapsed time. (e) The student predicts the stop-gradient representation produced by the exponential-moving-average teacher in latent space.
Figure 3: Prognostic classification (tumor therapy stratified). (a) Stratified summary (macro-mean AUC and AP) for response, PFS group, and OS group, comparing a frozen-embedding probe (oFM) to an engineered-feature baseline under an identical probe. (b) Per-stratum AUC comparison (one point per tumor therapy stratum; point area proportional to patients) showing that gains are broad-based and concentrated in larger strata.
Figure 4: Predictive evaluation. oFM frozen embeddings against the curated-feature baseline. (a) Metrics pooled across tasks by Paule-Mandel random effects: (left) is the pooled AUTOC/SD normalized by and measures how consistently nonzero the treatment benefit effect is across cohorts and (right) the pooled HRR. The yellow bands are the 5% permutation nulls, about zero (no signal) for the AUTOC/SD ranking and about one (no benefit) for the HRR; p values are computed with respect to each null. (b) Per-cohort AUTOC/SD, oFM against baseline. Area is proportional to ; points above the diagonal favor oFM and a bold outline marks .
Figure 5: Prognostic discrimination. Uno’s C-index computed within the control arm (left) and the experimental arm (right), for oFM against the curated-feature baseline. (a) Pooled across cohorts by Paule–Mandel random effects: baseline C-index (grey) and oFM (blue) with the significance of the difference via two-sided Student t-test. The yellow band shows the range indistinguishable from the baseline. (b) One point per cohort and arm for baseline or oFM, whichever scores better; area proportional to . Points above the diagonal favor oFM.
Figure 6: Mechanism-discovery pipeline. Seven steps decompose a frozen embedding and fitted head into interpretable mechanisms: (1) leave-one-out fact attribution; (2) sparse-autoencoder dictionary learning; (3) differential activation and clustering; (4) IDF-enriched concept labelling; (5) causal steering via latent-space ablation/amplification; (6) composition analysis and retrieval-grounded reasoning; (7) temporal-precedence mechanism graph with curated knowledge overlay and chain distillation. All steps run on frozen embeddings without fine-tuning.
Figure 7: Mechanism graph. Illustrative mechanism graph depicting feature interactions associated with treatment benefit in second-line or later (2L+) patients treated with trastuzumab deruxtecan (T-DXd) versus physician’s choice chemotherapy in a multi-tumor cohort. Directed chains are distilled from the temporal-precedence directed acyclic graph (DAG). Concept nodes are color-coded by category (genomic biomarkers, treatment, diagnosis, procedure, metastatic site); temporal edges are labeled with precede-fraction and median day gap (in days); annotation chips mark curated gene-drug actionability (CIViC), RNA expression correlates derived via probing analysis, and LLM-hypothesized relations with their retrieved citations listed in Table 1 .
| # | Reference |
| 1 | CIViC evidence EID993: KRAS Exon 2 Mutation in Colorectal Cancer (Predictive, level A). |
| 2 | CIViC evidence EID2998: KRAS Mutation in Lung Non-small Cell Carcinoma (Predictive, level A). |
| 3 | CIViC evidence EID5345: KRAS Mutation in Colorectal Cancer (Predictive, level A). |
| 4 | CIViC evidence EID704: CD274 Expression in Melanoma (Predictive, level B). |
| 5 | CIViC evidence EID1167: CD274 Expression in Lung Non-small Cell Carcinoma (Predictive, level B). |
| 6 | CIViC evidence EID7313: PIK3CA Mutation in Breast Cancer (Predictive, level A). |
Table 1: Numbered references for the mechanism-chain (Figure 7 ). Sources: PubMed entries in author/journal form; CIViC, PharmGKB and trial records in accession form. CIViC tiering includes cross-indication evidence.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Tumor type | Therapy group | OS (pos) | PFS (pos) | Response (pos) |
|---|---|---|---|---|
| Colorectal cancer | Biologic + chemotherapy | 383 (285) | 355 (280) | 451 (196) |
| NSCLC | Chemotherapy + IO/ICI | 325 (207) | 327 (214) | 450 (181) |
| Breast cancer | Hormone + targeted (SM) | 239 (203) | 279 (214) | 277 (94) |
| Colorectal cancer | Chemotherapy | 282 (226) | 248 (199) | 240 (100) |
| Pancreatic cancer | Chemotherapy | 274 (148) | 205 (122) | 281 (62) |
| Prostate cancer | Hormone | 299 (290) | 326 (287) | 100 (44) |
Table A1: Per-stratum composition by tumor type and therapy group, for the strata entering the prognostic-classification analysis (minimum 20 patients per stratum, as in the figure). Values are patients, (positive cases). An em dash indicates the stratum did not enter that task: fewer than 20 patients, a single outcome class, or no available embeddings.
| Cohort | Selected features |
| Breast HR+/HER2 CDK4/6 and aromatase inhibitor (AI) comb. vs. AI alone, 1L | 104 |
| CRC RAS WT anti-EGFR vs. anti-VEGF, 1L | 197 |
| NSCLC pembro–chemo combo vs. chemo, 1L | 94 |
| NSCLC pembro–chemo combo vs. pembro alone, 1L | 90 |
| NSCLC dual IO vs. IO–chemo combo, 1L | 81 |
| mRCC dual IO vs. IO–TKI, 1L | 160 |
Table A2: Number of selected baseline features per predictive evaluation cohort.
| Cohort | Experimental / Control arm | Clinical question | Train+val (exp / ctrl) | Test (exp / ctrl) |
|---|---|---|---|---|
| Breast HR+/HER2- CDK4/6 and Aromatase inhibitors (AI) comb vs AI alone 1L | Exp: CDK4/6 inhibitor plus aromatase inhibitor (AI) Ctrl: aromatase inhibitor alone | Does adding a CDK4/6 inhibitor (ribociclib, palbociclib, or abemaciclib) to first-line aromatase inhibitor therapy improve progression-free and overall survival in HR+/HER2- metastatic breast cancer? | 814 / 146 | 291 / 63 |
| CRC RAS WT anti-EGFR vs anti-VEGF 1L | Exp: anti-EGFR antibody (cetuximab or panitumumab) combined with FOLFOX or FOLFIRI chemotherapy Ctrl: bevacizumab combined with FOLFOX | In RAS wild-type metastatic colorectal cancer, does first-line anti-EGFR antibody plus chemotherapy improve overall survival versus bevacizumab plus chemotherapy? | 204 / 427 | 72 / 144 |
| NSCLC Pembro Chemo combo vs Chemo 1L | Exp: pembrolizumab plus platinum (carboplatin or cisplatin) plus pemetrexed Ctrl: platinum plus pemetrexed (chemotherapy only) | In driver-negative metastatic NSCLC, does adding pembrolizumab to platinum plus pemetrexed improve overall survival versus chemotherapy alone, and across PD-L1 subgroups ( 1%, 1–49%, 50%), which patients derive the greatest incremental benefit? | 1034 / 113 | 323 / 30 |
| NSCLC Pembro Chemo combo vs Pembro alone 1L | Exp: pembrolizumab plus platinum-based chemotherapy Ctrl: pembrolizumab monotherapy | In PD-L1 50% driver-negative metastatic NSCLC, does adding chemotherapy to first-line pembrolizumab improve outcomes versus pembrolizumab monotherapy? | 259 / 311 | 87 / 101 |
| NSCLC dual IO vs IO Chemo combo 1L | Exp: chemo-immunotherapy with pembrolizumab plus platinum plus pemetrexed or paclitaxel (KEYNOTE-189 style) Ctrl: dual checkpoint immunotherapy with ipilimumab plus nivolumab, with or without chemotherapy (CheckMate-227 style) | In driver-negative metastatic NSCLC, does first-line chemo-immunotherapy (pembrolizumab-based) outperform dual checkpoint blockade (ipilimumab plus nivolumab)? | 1231 / 111 | 398 / 54 |
| mRCC dual IO vs IO-TKI 1L | Exp: IO-TKI combinations (pembrolizumab plus axitinib, lenvatinib plus pembrolizumab, cabozantinib plus nivolumab, or avelumab plus axitinib) Ctrl: dual checkpoint immunotherapy (ipilimumab plus nivolumab) | In metastatic renal cell carcinoma, does first-line IO-TKI combination therapy improve outcomes versus dual checkpoint immunotherapy (ipilimumab plus nivolumab)? | 269 / 191 | 81 / 74 |
Table A3: Survival-benchmark cohorts. Two-arm comparative cohorts evaluated for differential treatment benefit on overall survival. Each row gives the cohort (with line-of-therapy and clinical-use-case tags), the experimental vs. control regimens, the clinical question (condensed to its primary clause), and the number of experimental / control patients in the combined train+validation (model-development) and held-out test splits. Patient counts are for the genomic (DNA+RNA) subset.
| Component | Parameters |
| Trained parameters: oFM | |
| Episode encoder (GatorTron-base-2k + FoNE) | 358M |
| transformer layers | 303M |
| token-embedding matrix ( ) | 51.3M |
| position/type embeddings + LayerNorm | 2.1M |
| FoNE numeric embedder | 1.1M |
Table A4: Parameter counts for different components in the oFM model.
More in this topic
Multimodal Foundation Models
Multimodal foundation models learn shared capabilities across text, images, audio, video, and other signals for understanding, generation, and action.
See all 354 papers