GeneICL: A Tabular Foundation Model for Bulk Transcriptomics

Michael Bohl; Alexander Theus; David Wissel; Valentina Boeva

cs.LGOct 6, 2026

GeneICL: A Tabular Foundation Model for Bulk Transcriptomics

Authors: Michael Bohl, Alexander Theus, David Wissel, Valentina Boeva

Organizations: ETH Zurich · Max Planck Institute for Intelligent Systems · Swiss Institute of Bioinformatics · Université Paris Cité

Abstract

Gene expression is widely measured in biomedicine, yet clinical outcome prediction remains challenging due to high dimensionality, strong feature correlations, and limited labeled data. Large self-supervised transcriptomic foundation models often fail to outperform simple supervised baselines. Tabular foundation models offer an alternative through in-context learning, but are typically pretrained on generic synthetic data rather than transcriptomic structure. We ask whether transcriptomics-aware pretraining, rather than scale, is the missing ingredient. Towards this end, we introduce GeneICL, a 4.2M-parameter tabular foundation model combining a semi-synthetic pretraining prior built from measured bulk expression profiles with a parameter-efficient recurrent architecture. We further enable right-censored survival prediction via a training-free reduction to regression using Cox partial-likelihood residuals. We evaluate GeneICL on 80 clinical outcome-prediction tasks spanning classification, regression, and survival. Tabular foundation models consistently outperform self-supervised transcriptomic models, while GeneICL achieves the best overall rank among evaluated foundation models and tuned baselines. GeneICL does so with up to 387 $\times$ fewer parameters, no gradient updates at inference, and predictions within seconds on a laptop CPU.

Figures & tables

Figure 1: Parameter–performance trade-off across foundation models. Performance is the task-macro average rank across classification and regression tasks. The dashed line marks the Pareto frontier. Lower is better on both axes.

Figure 2: Overview of GeneICL. (a) Transcriptomics-grounded pretraining prior. A variational autoencoder (VAE) is trained on unlabeled real bulk transcriptomes. The decoder is used to generate synthetic expression datasets. A randomly sampled nonlinear function maps the individual profiles to a classification or regression label for every sample. Generated datasets are split into a labeled support set and a query set, forming one synthetic pretraining task. (b) GeneICL architecture. Transcriptomic profiles are projected with the support-fitted PCA and fed into an encoder. A label embedding of the support labels is added to the support tokens in the encoder and in each iteration of the weight-tied looped in-context-learning (ICL) transformer block. Classification and regression heads read out predictions from the query tokens. (c) Benchmark. GeneICL is compared against tabular and transcriptomic foundation models and conventional baselines (performance shown schematically, see Table 1 ).

Figure 3: Benchmark overview. The benchmark comprises 27 classification, 25 regression, and 28 survival tasks. Bars summarize endpoint categories or cohort sources. Sample-size distributions show individual tasks (dots), interquartile ranges (shading), and medians (ticks) on a shared logarithmic scale.

Tabular foundation models
Model	Params $\downarrow$	Avg. rank $\downarrow$	Classification $\uparrow$			Regression $\uparrow$
			AUROC	AUPRC	Bal. Acc.	Pearson	Spearman	$\mathbf{R^{2}}$
GeneICL †	4.24M	4.64	0.742 (2)	0.695 (4)	0.592 (6)	0.528 (1)	0.537 (1)	0.274 (5)
GeneICL	4.24M	4.98	0.741 (4)	0.689(2)	0.588(3)	0.521 (1)	0.533(1)	0.271 (4)
TabFM †	1.65B	5.65	0.721	0.686	0.577	0.515	0.535	0.268
TabICLv2 †	28.0M	7.24	0.712	0.671	0.584	0.495	0.519	0.243

Table 1: Performance across 27 classification and 25 regression tasks. Best and second-best results per metric are bold and underlined , respectively. Avg. rank is the mean rank across all six metrics and tasks (lower is better). † denotes ensembled predictions. GeneICL † averages predictions across PCA thresholds and support-set subsets (Appendix H.3 ). Parentheses for GeneICL denote the standard deviation over three seeds in units of the last decimal place.

Figure 4: Survival prediction across 28 SurvBoard datasets. We compare GeneICL with Cox residual fitting against alternative tabular survival adaptations, SurvPFN, and tuned baselines. Panels show Antolini’s C-index ( a ), integrated Brier score ( b ), and D-calibration

\chi^{2}

( c ). Small points denote datasets; larger markers show study-weighted aggregates across TARGET, TCGA, METABRIC, and ICGC. Agg. reports this aggregate and Rank the mean within-dataset rank; best and second-best values are bold and underlined . Dashed lines mark chance-level C-index (

0.5

) and the D-calibration threshold (

\chi^{2}=16.9

p=0.05

Variant	Params $\downarrow$	Attn.	Avg. rank $\downarrow$	Classification $\uparrow$			Regression $\uparrow$
				AUROC	AUPRC	Bal. Acc.	Pearson	Spearman	$\mathbf{R^{2}}$
GeneICL	4.24M	$\mathcal{O}(n)$	2.56	0.741 (4)	0.689 (2)	0.588(3)	0.521 (1)	0.533 (1)	0.271 (4)
Untied (8 blocks)	19.25M	$\mathcal{O}(n)$	2.70	0.740 (2)	0.689 (3)	0.589 (1)	0.517 (1)	0.531 (2)	0.263(6)
Full attention	4.05M	$\mathcal{O}(n^{2})$	2.85	0.739(0)	0.688(2)	0.570(6)	0.515(3)	0.530(1)	0.273 (3)
TabICLv2 prior pretraining	4.24M	$\mathcal{O}(n)$	3.55	0.737(1)	0.690 (1)	0.576(3)	0.496(6)	0.514(5)	0.238(7)
Real expression data pretraining	4.24M	$\mathcal{O}(n)$	3.33	0.735(3)	0.682(3)	0.593 (11)	0.505(3)	0.517(4)	0.248(4)

Table 2: Ablation of the main GeneICL design choices . Attn. denotes the asymptotic cost of the feature-axis attention. Parentheses report the standard deviation over three seeds in units of the last decimal place. Best and second-best results are bold and underlined, respectively.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$n$	Type	Description
GSE107422	80	cls	Systemic recurrence from colorectal cancer resection tissue
GSE116324	44	cls	Bortezomib-induction response (response sustained $\geq$ 1 yr) in multiple myeloma patients
GSE120622	80	cls	Non-Small cell lung cancer (surgical tumor) relapse
GSE154261	73	cls	24-month recurrence of early stage non-muscle-invasive bladder cancer
GSE156699	88	cls	platinum chemotherapy response (PFS $\geq$ 6 mo) of high-grade serous ovarian cancer patients
GSE157657	59	cls	Anti-tuberculosis therapy response (4-class) from whole blood

Appendix

Table 3: Datasets used in the benchmark.

n

is the number of usable (non-NaN) rows (samples) for the associated task. For datasets with several tasks of the same type (i.e. BeatAML, CMP), the range over tasks is given. Type: cls = classification, reg = regression.

Variant	Params	Attn.	Classification			Regression
			AUROC	AUPRC	Bal. Acc.	Pearson	Spearman	$\mathbf{R^{2}}$
GeneICL	4.24M	$\mathcal{O}(n)$	0.846 (2)	0.812 (1)	0.787 (10)	0.806 (7)	0.753 (8)	0.629 (4)
Untied (8 blocks)	19.25M	$\mathcal{O}(n)$	0.845(1)	0.812 (3)	0.784 (6)	0.805 (5)	0.758 (4)	0.622 (8)
Full attention	4.05M	$\mathcal{O}(n^{2})$	0.843(1)	0.808(2)	0.778(11)	0.784(13)	0.737(20)	0.581(28)
TabICLv2 prior pretraining	4.24M	$\mathcal{O}(n)$	0.836(4)	0.801(5)	0.769(18)	0.744(11)	0.688(5)	0.518(20)
Real expression data pretraining	4.24M	$\mathcal{O}(n)$	0.843(3)	0.800(14)	0.731(59)	0.788(9)	0.736(8)	0.457(35)

Appendix

Table 4: Experiments evaluated on our holdout datasets (three classification and two regression) . In the last experiment, GeneICL was given all principal components at inference instead of those explaining 90% of the support-set variance. Attn. denotes the asymptotic cost of the feature-axis attention. Parentheses report the standard deviation over three seeds in units of the last decimal place. Best and second-best results are bold and underlined, respectively.

Parameter	Value
Architecture
Embedding dim $D$	128
Attention heads	8
Encoder depth	3
CLS columns	4
Inducing rows (col. attn. bottleneck)	4

Appendix

Table 5: Hyperparameter selection for GeneICL . Bolded values indicate identical configurations to TabICLv2.

Figure 5: Runtime vs. support set size. Wall-clock runtime per cross-validation fold (log scale) against the number of support (training) samples ( $\log_{2}$ scale), for GeneICL (blue), the other tabular foundation models (red to purple) and the tuned baselines (green, dashed). Each faint marker is one of the 52 benchmark tasks. Lines connect the medians over tasks within support-size bins (edges 32, 64, 128, 256, 512, 1024, 4096). Classification and regression tasks are pooled. Runtime covers fit and predict: for the in-context models that is the forward pass. All models ran on an NVIDIA GeForce RTX 4090 GPU, except CatBoost and the linear model, which ran on AMD EPYC 7763 CPUs.

Setting	Value
Input dimension	20,021 genes
Hidden width (encoder / decoder)	5,000 / 5,000
Latent dimension	256
Activation	GELU
Parameters	204M
KL weight	1, linear warm-up over 1,500 steps

Appendix

Table 6: VAE configuration and hyperparameters.

Quantity	Distribution
Task and label space
Support fraction (shared)	$\mathcal{U}[0.5,0.9]$
Samples per task	log $\mathcal{U}[32,1024]$
Number of selected PCs $k$	log $\mathcal{U}\{2,\ldots,\min(64,k_{\mathrm{tot}})\}$
Labels
Number of classes $K$	log $\mathcal{U}\{2,\ldots,10\}$

Appendix

Table 7: Sampling distributions of the pretraining prior. Quantities are resampled for every synthetic dataset unless marked as shared across the batch.

Panel of Figure 6 — Figure 6: Depth extrapolation of the recurrent in-context learner. GeneICL and a plain recurrent baseline are trained with $T=8$ recurrence steps and evaluated with up to 128 steps at inference time. The plain baseline repeatedly applies a post-norm Transformer block, whereas GeneICL uses the fixed-evidence, evolving-hidden-state recurrence described in Section 3.2 . Panels report (a) AUROC, (b) AUPRC, (c) balanced accuracy, (d) Pearson correlation, (e) Spearman correlation, and (f) $R^{2}$ . Lines and markers show the mean over three independently trained seeds and shaded regions indicate $\pm 1$ standard deviation. GeneICL remains stable far beyond the training depth, whereas plain recurrence degrades under continued iteration.

Figure 7: Inference-time ensembling strategies for GeneICL. Each ensembling method is applied independently to a frozen GeneICL checkpoint and compared against a single-forward-pass baseline. We report AUROC, AUPRC, and balanced accuracy for classification, and Pearson correlation, Spearman correlation, and

R^{2}

for regression. Bars show the mean performance across three independently trained seeds, with error bars indicating one standard deviation. The dashed vertical line marks the single-forward baseline. Generic ensembling strategies are compared with transcriptomic-specific strategies based on highly variable genes and Hallmark pathways. Higher is better for all metrics.

Explore similar work

Tabular Foundation Models for Clinical Survival Analysis via Survival-Aware Adaptation
Jun 10, 2026Minh-Khoi Pham, Luca Cotugno, Alina Sirbu +3Survival Predictions Survival Analysis
ADAPT Centre, Dublin City University, Dublin, Ireland · School of Computing, Dublin City University, Dublin, Ireland · Department of Computer Science and Engineering, University of Bologna, Bologna, Italy
TabICLv2: A better, faster, scalable, and open tabular foundation model
Feb 11, 2026Jingang Qu, David Holzmüller, Gaël Varoquaux +1Tabular Foundation Models Pretraining
SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France
SurvPFN: Towards Foundation Models for Survival Predictions
Jun 3, 2026Samuel Böhm, Lennart Purucker, Frank Hutter +1Survival Predictions Survival Analysis
Institute of Epidemiology and Prevention, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany · Institute of Epidemiology and Prevention, Medical Center – University of Freiburg, Freiburg, Germany · Department of Computer Science, University of Freiburg, Freiburg, Germany +7

Jun 10, 2026, cs.LGJ/K move · Enter open · S save

Tabular Foundation Models for Clinical Survival Analysis via Survival-Aware Adaptation

Minh-Khoi Pham, Luca Cotugno, Alina Sirbu, Tai Tan Mai+2

ADAPT Centre, Dublin City University, Dublin, Ireland · School of Computing, Dublin City University, Dublin, Ireland · Department of Computer Science and Engineering, University of Bologna, Bologna, Italy

Survival Predictions Survival Analysis Prognostics Tabular Foundation Models Intensive Care Unit Parameter-Efficient Adaptation

Open paper AskPDF arXiv

Predicting time-to-event outcomes such as mortality is a fundamental task in clinical decision-making, commonly addressed through survival analysis. While classical statistical and deep learning approaches have been widely studied, they typically require task-specific training and sufficient labeled data. Recent advances in tabular foundation models offer a new paradigm by learning general-purpose representations for structured data. However, their applicability to censored time-to-event prediction in clinical settings remains underexplored, as typical applications are restricted to discrete classification rather than survival analysis tasks. In this work, we propose a lightweight adaptation approach for applying tabular foundation models to clinical survival analysis by directly training a survival-aware head on top of the pretrained representations. We study representative architectures, including TabPFN, TabDPT, and TabICL, and adapt them using a multi-task logistic regression (MTLR) head to model right-censored time-to-event outcomes. We evaluate this approach on a diverse set of public survival benchmarks and two large-scale ICU cohorts, MIMIC-IV and eICU. Our results show that this transfer learning approach achieves competitive or superior performance compared to strong baselines. On MIMIC-IV, TabDPT-FT-MTLR reaches a C-index of 0.856, corresponding to a relative improvement of +1.4% over the best non-FM baseline (DeepSurv, 0.844) and +6.7% over the best zero-shot model (0.802). On eICU, TabICL-FT-MTLR achieves 0.797, yielding gains of +1.7% (DeepSurv, 0.784) and +6.4% (0.749), respectively. These findings highlight the importance of combining pretrained tabular representations with survival-aware objectives and suggest that tabular foundation models provide a practical and effective alternative for clinical survival prediction.

Abstract

Explore similar work

Tabular Foundation Models for Clinical Survival Analysis via Survival-Aware Adaptation

TabICLv2: A better, faster, scalable, and open tabular foundation model

SurvPFN: Towards Foundation Models for Survival Predictions