Speedrunning Relational Foundation Model Pretraining

Gabriel Wedson Mendonça de Souza Veloso; Luan Borges Teodoro Reis Sena; Denis Oliveira Correa; Francisco Galuppo Azevedo

cs.AIOct 20, 2026

Speedrunning Relational Foundation Model Pretraining

Authors: Gabriel Wedson Mendonça de Souza Veloso, Luan Borges Teodoro Reis Sena, Denis Oliveira Correa, Francisco Galuppo Azevedo

Organizations: Kunumi Institute, Belo Horizonte, Minas Gerais, Brazil · Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

Abstract

Progress on foundation model pretraining is gated by who can afford a trial, and relational pretraining is no exception. Recipes are developed inside individual labs, reported under private protocols, and do not accumulate. Speedrun leaderboards remove that barrier by decentralizing the search rather than the training run. The problem is fixed, the stack is open, entries are ranked by wall-clock time to a quality target, and one GPU is enough to hold the record. We introduce modded-nanoRT, the first such leaderboard for Relational Transformers. Which database to hold out decides whether the benchmark can resolve improvements at all, so we fix it by calibration, ranking six candidates by how sharply a 45-minute learning curve separates improvement from seed noise. We open the leaderboard with typed trilinear attention, a relational operator that conditions attention scores and values on the foreign key traversed, together with Muon and SIGReg. The recipe reaches the target in 305.63 seconds on one NVIDIA H200, 8.11x faster than the reference implementation, and wins 13 of 15 pairs outside the development database. We will release the benchmark, its frozen artifacts, and the record runs. Our code is available at https://github.com/kunumi/nanoRT.

Figures & tables

Pretraining tasks	RelBench v1 binary classification tasks, minus rel-Avito and rel-Event
Mixture	temperature sampling, $\alpha=0.5$
Held-out database	rel-Avito
Metric	pooled AUROC on rel-Avito
Target	$0.81621$
Reference model	RT, 12 blocks, width 256, 8 heads
Trainable parameters	19.3M

Table 1: The modded-nanoRT protocol.

Panel of Figure 1 — Figure 1: The two architectural changes in the record recipe, marked in burgundy. A frozen text embedding of each foreign-key description enters the trilinear operator, which replaces RT’s neighbor attention in all twelve blocks. SIGReg is applied during training, at the first block only.

Panel of Figure 2 — Figure 2: Cumulative record lineage, from the reference implementation’s $41$ : $20$ to $305.63$ seconds. Entries are chronological records, not controlled comparisons.

Component	Paired contrast	$\Delta$ AUROC (pp)	Wins
Muon	on AdamW RT	$+2.56\pm 3.13$	3/3
Muon	with typed attention	$+6.60\pm 3.53$	3/3
Typed attention	on Muon $+$ SIGReg	$+3.56\pm 4.79$	2/3
Typed attention	on AdamW	$-1.21\pm 5.79$	1/3
SIGReg	before typed attention	$-0.47\pm 0.66$	0/3
SIGReg	with typed attention	$+0.26\pm 0.42$	3/3

Table 2: Paired change in best rel-Avito pooled AUROC (row minus control, percentage points). Mean

\pm

standard deviation across three seeds; Wins counts seeds with a positive difference.

Held-out DB	AdamW RT	Final	$\Delta$ (pp)	Wins
rel-Amazon	58.45	58.17	$-0.29\pm 12.86$	2/3
rel-Avito	80.21	83.15	$+2.94\pm\phantom{0}6.24$	2/3
rel-F1	79.55	84.58	$+5.04\pm\phantom{0}1.69$	3/3
rel-HM	62.77	65.14	$+2.37\pm\phantom{0}0.51$	3/3
rel-Stack	75.07	76.96	$+1.88\pm\phantom{0}1.85$	3/3
rel-Trial	51.50	52.38	$+0.88\pm\phantom{0}2.91$	2/3

Table 3: Best pooled AUROC across leave-one-database-out holdouts.

\Delta

is Final minus AdamW RT in percentage points, mean

\pm

standard deviation across three seeds.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Relation vector	Best AUROC	Steps 100–900	Reached target
Semantic FK (record recipe)	$0.8335\pm 0.0330$	$0.7656\pm 0.0327$	5/7
Shuffled FK	$0.8371\pm 0.0381$	$0.7794\pm 0.0395$	5/7
Random per type/layer	$\mathbf{0.8397}\pm 0.0156$	$\mathbf{0.7830}\pm 0.0254$	7/7
Random per edge	$0.8204\pm 0.0281$	$0.7739\pm 0.0328$	3/7

Appendix

Table 4: Relation-vector ablation on rel-Avito over seven seeds (mean

\pm

standard deviation). Reached target counts seeds whose best pooled AUROC exceeds

0.81621

Panel of Figure 3 — Figure 3: Mean AUROC over steps 100–900. The bar denotes the nonsignificance clique at $\alpha=0.05$ .

Panel of Figure 4 — Figure 4: Reference evaluation trajectory over the 45-minute calibration budget ( $11{,}621$ updates, evaluated every $500$ ). Its best value, $0.81621$ at $41$ : $20$ , defines the target; the record recipe exceeds it at $5$ : $05.6$ , reaching $0.8714$ . Times are upper bounds determined by the evaluation grids.

Relational Foundation Models

Relational foundation models learn reusable representations and predictors for tasks spanning linked tables and databases.

See all 17 papers

Speedrunning Tabular Foundation Model Pretraining
Jun 2, 2026Salih Bora Ozturk, Alexander Pfefferle, Frank HutterTabular Foundation Models Tabular Prediction
University of Freiburg, Freiburg im Breisgau, Germany · ELLIS Institute Tübingen, Tübingen, Germany · Prior Labs, Freiburg im Breisgau, Germany
KumoRFM-2: Scaling Foundation Models for Relational Learning
Apr 14, 2026Valter Hudovernik, Federico López, Vid Kocijan +4Relational ICL Relational Foundation Models
Relational In-Context Learning via Synthetic Pre-training with Structural Prior
Mar 4, 2026Yanbo Wang, Jiaxuan You, Chuan Shi +1Relational ICL Relational Foundation Models
Institute for Artificial Intelligence, Peking University · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Beijing University of Post and Telecommunication +1

Jun 2, 2026J/K move · Enter open · S save

Speedrunning Tabular Foundation Model Pretraining

Salih Bora Ozturk, Alexander Pfefferle, Frank Hutter

University of Freiburg, Freiburg im Breisgau, Germany · ELLIS Institute Tübingen, Tübingen, Germany · Prior Labs, Freiburg im Breisgau, Germany

Tabular Foundation Models Tabular Prediction

Open paper AskPDF arXiv

Pretraining cost is a major bottleneck for research on tabular foundation models, slowing the iteration cycle for new architectures, priors, and optimization ideas. Yet the community lacks a simple way to compare and accumulate pretraining speedups. We introduce a community speedrun for nanoTabPFN: contributors modify a single-file training script and compete to reach a fixed downstream ROC AUC target on subsampled TabArena using one NVIDIA L40S GPU. The current best record reaches the target in 0.92 minutes, an 81x speedup over the 74.32 minute baseline while using 22x fewer synthetic datasets. The speedrun format provides a simple protocol for the community to add, verify, and stack pretraining improvements, with the leaderboard open to contributions. Code and records are available at https://github.com/borawhocodess/modded-nanotabpfn.

Abstract

Relational Foundation Models

Explore similar work

Speedrunning Tabular Foundation Model Pretraining

KumoRFM-2: Scaling Foundation Models for Relational Learning

Relational In-Context Learning via Synthetic Pre-training with Structural Prior