RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

Rishabh Ranjan; Vignesh Kothapalli; Harshvardhan Agarwal; Charilaos Kanatsoulis; Roshan Upendra; Tom Palczewski; Carlos Guestrin; Jure Leskovec

cs.LGOct 13, 2026

RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

Authors: Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal, Charilaos Kanatsoulis, Roshan Upendra, Tom Palczewski, Carlos Guestrin, Jure Leskovec

Organizations: Stanford University · SAP

Abstract

Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639 web-crawled databases with 6,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.

Figures & tables

Figure 1: RT-J pretraining recipe. (1) We sample a database and prediction task from the Join . The task specifies the target table and the target column . (2) We sample a target row from the target table. Rows after the target row’s timestamp are dropped; kept target-table rows form the candidate in-context examples . (3) From the target row, many parallel random walks rank candidate examples by visit count, on the fly. (4) We process candidate examples in descending visit-count order, using each as the root of a local bounded-width breadth-first-search (BFS) over primary–foreign key links; the context size and BFS hyperparameters are sampled randomly. (5) The collected rows form the context window for the relational transformer (RT), which treats cells as tokens . The target table rows in the context enable in-context learning . The target cell is always masked; non-target cells may be masked optionally. (6) RT-J predicts all masked cells and is trained against their ground-truth values.

Figure 2: In-context regression. Examples are matched. RT-J at train context size of

8192

cells fits around

225

examples on average. At all example counts, RT-J is

>2

nMAE points better than the best baseline; at all performance levels, RT-J needs

\gtrsim 10\times

fewer examples.

Figure 3: RT-J vs baselines. On regression (left), RT-J dominates baselines at every example count, by up to

5

nMAE points over the strongest one. On classification (right), RT-J leads by up to

3

AUROC points, especially when examples are scarce. Per-task plots are in Appendix J .

		Fine-tuned		In-context
Model	Pretraining data	nMAE % $\downarrow$	AUROC % $\uparrow$	nMAE % $\downarrow$	AUROC % $\uparrow$
RT	none (random init)	$29.3$	$77.8$	–	–
RT-PluRel	PluRel	$27.6$	$78.5$	$32.9$	$73.0$
RT-J	PluRel $\to$ the Join	$\mathbf{27.1}$	$\mathbf{79.0}$	$\mathbf{32.4}$	$\mathbf{74.5}$

Table 1: Regression nMAE and classification AUROC averaged over all RelBench tasks for RT models with no pretraining (RT), synthetic-only pretraining (RT-PluRel) and synthetic + real pretraining ( RT-J ), for both fine-tuning and in-context learning settings. Per-task results are in Appendix A .

Figure 6: Context-retrieval ablation with RT-J at test-time. Left sub-panels plot the metric against context size

L

and right sub-panels plot it against the average labels delivered at

L=8192

(non-label cells provide relational context). Random-walk (ours) is competitive with the best, despite being cheaper and more scalable. RT-J generalizes to other retrievers too. Retriever quality is not strongly correlated with the label count; relational context is important too. Per-task plots are in Appendix J .

Comparison	Reg.	Clf.	Comparison	Reg.	Clf.
Pretraining data			Context retrieval
RT-J vs. RT-PluRel	$+0.6$	$+1.5$	random walk vs. width-256 BFS	$+1.6$	$+3.9$
the Join -only vs. PluRel-only	$+1.8$	$+2.8$	random walk vs. uniform random	$+4.1$	$+6.1$
Test-time compute
tuned vs. default context	$+0.8$	$-0.5$	16 context ensemble vs. 1 context	$+0.2$	$+1.1$

Table 2: Summary of gains, in nMAE (regression) and AUROC (classification) points.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Combined model + system leaderboard of RelArena-

\alpha

, reproduced from Hayler et al. [27] : Elo over the

21

RelBench v1 forecasting tasks (anchored at

1000

for the global constant predictor). RT-PluRel (hatched) is a system submission with its own tuning protocol; TabPFN-Rel variants in pink, other baselines in blue, constant predictors in grey.

Regression (nMAE %, lower is better)
		Fine-tuned			In-context
DB	Task	RT (rand init)	RT-PluRel	RT-J	RT-PluRel	RT-J
rel-amazon	item-ltv	$7.50$	$7.26$	$\mathbf{7.19}$	$8.65$	$\mathbf{7.81}$
rel-amazon	user-ltv	$\mathbf{24.15}$	$24.58$	$24.29$	$29.60$	$\mathbf{29.14}$
rel-avito	ad-ctr	$38.53$	$37.86$	$\mathbf{36.03}$	$\mathbf{41.71}$	$42.01$
rel-event	user-attendance	$32.52$	$31.64$	$\mathbf{31.63}$	$43.78$	$\mathbf{42.69}$

Appendix

Table 3: RT models on the RelBench leaderboard. Per-task test metrics of RT [ 23 ] , RT-PluRel and RT-J , fine-tuned on each task or used in-context, as listed on the RelBench leaderboard. All metrics in percent. Best per row within each group in bold; lower is better for nMAE, higher is better for AUROC.

Domain	Source	DBs	License(s)	Ref.
Commerce & Online Platforms
tech community	Stack Exchange / GH Archive / HN	75	CC BY 4.0 Open public API CC BY-SA 4.0	gharchive.org hn.algolia.com archive.org
recommendations	Amazon / MovieLens / IMDb / UCI	34	MIT (Amazon ToS) Research-use Research-use Research-use Research-use Public Domain CC BY 4.0	github.com data.imdb.com cseweb.ucsd.edu grouplens.org grouplens.org research.gov archive.ics.uci.edu
prediction market	Manifold / Polymarket	2	ToS-restricted ToS-restricted	manifold.markets polymarket.com
ecommerce	RecSys Challenge 2015	1	CC BY-NC-ND 4.0	kaggle.com
consumer product	Open Food Facts	1	ODbL	world.openfoodfacts.org

Appendix

Table 4: Per-domain corpus breakdown across all collected domains, grouped by italic subject-area headers and sorted by descending DB count. DBs : raw count. The License(s) and Ref. cells list one (license, reference) pair per line with rows aligning across the two columns.

Template	Type	Prediction target (example)
Aggregation—volume / intensity of activity
aggregate_numeric	r	mean / sum of a numeric column over the next window (e.g. a player’s home runs next season)
count_events	r	number of events for the entity in the next window (e.g. orders per customer next month)
distinct_count	r	distinct values of a column the entity touches (e.g. distinct co-authors next year)
concentration	r	entity’s share of total activity (e.g. seller’s share of category sales)
Threshold / existence—continuous signal at a decision boundary

Appendix

Table 5: The 25 task-template families instantiated by the corpus, organized under italic signal-type headers. Each row gives the prediction target as the model sees it, paired with one concrete domain example. r : regression; c : binary classification.

Panel of Figure 10 — Figure 10: Schema of join-berka : every table, its row count, and the foreign key links. Arrows point from a foreign key to the table it references; $\dagger$ marks a table carrying a time column. The dashed box is the task table derived from the database.

Panel of Figure 11 — Figure 11: Schema of join-brightkite . Here friendships and users tables are joined by more than one foreign key, so the edges are drawn separately and labelled with the referencing column.

Panel of Figure 12 — Figure 12: Schema of join-oulad .

Dataset	Task	Entity	Prediction target
rel-amazon	user-churn	User	Whether the user does not review any product in the next 3 months.
rel-amazon	item-churn	Item	Whether the item receives no reviews in the next 3 months.
rel-avito	user-visits	User	Whether the user visits more than one ad in the next 4 days.
rel-avito	user-clicks	User	Whether the user clicks more than one ad in the next 4 days.
rel-f1	driver-dnf	Driver	Whether the driver does not finish a race in the next month.
rel-f1	driver-top3	Driver	Whether the driver qualifies in the top 3 for a race in the next month.

Appendix

Table 6: Binary classification forecasting tasks in RelBench.

Dataset	Task	Entity	Prediction target
rel-amazon	user-ltv	User	Value of the products the user buys and reviews in the next 3 months.
rel-amazon	item-ltv	Item	Value of the purchases and reviews the item receives in the next 3 months.
rel-avito	ad-ctr	Ad	Click-through rate for the ad over the next 4 days.
rel-f1	driver-position	Driver	Average finishing position of the driver across races in the next 2 months.
rel-hm	item-sales	Item	Total sales for the article in the next week.
rel-stack	post-votes	Post	Number of votes the post receives in the next 3 months.

Appendix

Table 7: Regression forecasting tasks in RelBench.

Regime	RelBench tasks	Count
rare event	rel-avito/user-clicks , rel-stack/user-badge , rel-stack/user-engagement	3
cold start	rel-trial/study-outcome , rel-trial/study-adverse , rel-avito/user-clicks	3
heavy-tailed target	rel-amazon/item-ltv , rel-avito/ad-ctr	2
long horizon	rel-amazon/user-churn , rel-amazon/item-churn , rel-event/user-repeat , rel-hm/user-churn	4

Appendix

Table 8: RelBench tasks in each hard regime. A task can belong to several regimes.

Panel of Figure 13 — Figure 13: Pretraining ablations: multi-cell masking rate and task mix, as validation nMAE / AUROC over pretraining steps. Each arm differs from RT-J ’s own pretraining run in one knob and is trained to the same $32{,}769$ -step budget. (a) Masking non-target cells at any rate $p_{\max}>0$ costs both metrics; RT-J masks only the target cell. (b) Forecasting tasks alone nearly match the mixture on classification but lose on regression, and autocompletion alone trails on both; the mixture is competitive on both.

Figure 14: Validation nMAE and AUROC (SWA,

8192

-cell context) over pretraining steps. The two runs from scratch are shown from step

2

K; the PluRel-only run stops early at

18

K steps.

Stage	$(10\mathrm{K},20)$	$(10\mathrm{K},10)$	$(1\mathrm{K},20)$	$(1\mathrm{K},10)$
Context sampler (CPU)	$28.06\pm 10.39$	$17.87\pm 8.14$	$14.31\pm 7.99$	$16.00\pm 8.49$
Forward pass (GPU)	$10.03\pm 0.38$	$9.85\pm 0.42$	$9.81\pm 0.43$	$9.78\pm 0.45$
Total	$38.09\pm 10.31$	$27.72\pm 8.12$	$24.11\pm 7.99$	$25.78\pm 8.52$

Appendix

Table 9: Per-example inference cost in ms at batch size

1

and

L=8192

, on one B200 GPU with one AMD EPYC 9565 CPU: per-task means over

100

timed steps after

5

warmup steps, then mean

\pm

std. over the

21

RelBench tasks.

Task	$(10\mathrm{K},20)$	$(10\mathrm{K},10)$	$(1\mathrm{K},20)$	$(1\mathrm{K},10)$
rel-amazon/item-churn	$76.67\pm 0.05$	$76.69\pm 0.06$	$\mathbf{76.79}\pm 0.12$	$\mathbf{76.91}\pm 0.14$
rel-amazon/user-churn	$66.88\pm 0.10$	$\mathbf{67.05}\pm 0.11$	$66.69\pm 0.14$	$66.50\pm 0.12$
rel-avito/user-clicks	$\mathbf{52.31}\pm 0.98$	$\mathbf{52.83}\pm 2.65$	$\mathbf{53.14}\pm 3.41$	$\mathbf{53.87}\pm 2.69$
rel-avito/user-visits	$61.70\pm 0.22$	$\mathbf{62.68}\pm 0.62$	$\mathbf{62.50}\pm 0.62$	$\mathbf{62.42}\pm 0.59$
rel-event/user-ignore	$\mathbf{79.91}\pm 1.95$	$\mathbf{79.55}\pm 2.20$	$\mathbf{78.94}\pm 2.16$	$77.71\pm 0.84$
rel-event/user-repeat	$\mathbf{77.91}\pm 1.13$	$\mathbf{77.48}\pm 1.24$	$\mathbf{77.07}\pm 1.23$	$75.70\pm 1.73$

Appendix

Table 10: AUROC (%, higher is better) of RT-J for

(W,K)

on the

8192

-row test subsample at

L=8192

, mean

\pm

std. over

3

context seeds; the mean row averages the tasks per seed. Bold: within one standard deviation of the best per row.

Task	$(10\mathrm{K},20)$	$(10\mathrm{K},10)$	$(1\mathrm{K},20)$	$(1\mathrm{K},10)$
rel-amazon/item-ltv	$8.49\pm 0.01$	$8.25\pm 0.09$	$8.35\pm 0.03$	$\mathbf{8.07}\pm 0.03$
rel-amazon/user-ltv	$\mathbf{30.21}\pm 0.10$	$30.44\pm 0.07$	$30.45\pm 0.07$	$\mathbf{30.17}\pm 0.11$
rel-avito/ad-ctr	$\mathbf{42.17}\pm 0.55$	$\mathbf{42.22}\pm 0.45$	$43.07\pm 0.64$	$43.11\pm 0.59$
rel-event/user-attendance	$\mathbf{43.46}\pm 1.66$	$\mathbf{43.38}\pm 1.71$	$\mathbf{43.45}\pm 1.70$	$\mathbf{43.78}\pm 1.85$
rel-f1/driver-position	$\mathbf{38.53}\pm 0.03$	$39.53\pm 0.10$	$39.97\pm 0.18$	$39.75\pm 0.07$
rel-hm/item-sales	$14.07\pm 0.20$	$\mathbf{11.61}\pm 0.09$	$12.56\pm 0.07$	$13.38\pm 0.06$

Appendix

Table 11: nMAE (%, lower is better) of RT-J for

(W,K)

on the

8192

-row test subsample at

L=8192

, mean

\pm

std. over

3

context seeds; the mean row averages the tasks per seed. Bold: within one standard deviation of the best per row.

Figure 15: Few-shot relational prediction vs. in-context-learning baselines on a fixed

8192

-row-per-task subsample of the RelBench test set, with the baselines extended to

131\mathrm{K}

-cell contexts. As in Figure 3 , the

x

-axis is the mean number of in-context labeled task rows and the markers on the RT-J curve are annotated with the underlying context size ( RT-J caps at its

8\mathrm{K}

-cell training context). With

20\times

more labels than RT-J ever sees, the TabICLv2 pipelines lead on classification, and LLM Agent + TabICLv2 passes RT-J ’s regression error only from

65\mathrm{K}

cells on.

Figure 16: Per-task test nMAE versus the average number of in-context examples on RelBench regression tasks; per-task counterpart of Figure 3 (left).

Figure 17: Per-task test AUROC versus the average number of in-context examples on RelBench classification tasks; per-task counterpart of Figure 3 (right).

Setting	Total passes	Fully parallel	Extra labels
Default context	$1$ test	$1$ test	none
Tuning ( $120$ configurations, $4$ seeds)	$144$ val. $+$ $1$ test	$1$ val. $+$ $1$ test	$\leq 4096$ (val.)
Ensembling ( $16$ seeds)	$16$ test	$1$ test	none

Appendix

Table 12: Passes over an evaluation split, in total and on the critical path when fully parallelized, and labels used beyond the training split.

Figure 18: Per-task test nMAE for context-retrieval ablations on RelBench regression tasks; per-task counterpart of Figure 6 (left).

Figure 19: Per-task test AUROC for context-retrieval ablations on RelBench classification tasks; per-task counterpart of Figure 6 (right).

Figure 20: Per-task test nMAE for the schema-semantics ablation on RelBench regression tasks; per-task counterpart of Figure 5 (left).

Figure 21: Per-task test AUROC for the schema-semantics ablation on RelBench classification tasks; per-task counterpart of Figure 5 (right).

DB	Task	$L$	$\ell$	$b$	$\rho$	Default	Tuned
Regression (nMAE %, lower is better)
rel-amazon	item-ltv	$4\mathrm{K}$	$2048$	$8$	$0$	$8.50$	$\mathbf{7.88}$
rel-amazon	user-ltv	$4\mathrm{K}$	$2048$	$8$	$0$	$30.02$	$\mathbf{29.45}$
rel-avito	ad-ctr	$8\mathrm{K}$	$1024$	$8$	$0$	$42.69$	$\mathbf{42.47}$
rel-event	user-attendance	$8\mathrm{K}$	$256$	$8$	$1$	$43.59$	$\mathbf{42.54}$
rel-f1	driver-position	$8\mathrm{K}$	$256$	$8$	$1$	$\mathbf{38.60}$	$38.60$

Appendix

Table 13: Per-task test metrics on the 8192-subsampled test set: shared default context (

L=8\mathrm{K}

\ell=256

b=32

\rho=1

) vs. the per-task tuned context. Columns

L

\ell

b

\rho

report the configuration selected on the validation split. Single context seed (no ensembling). All metrics shown in percent. Best per row in bold; lower is better for nMAE, higher for AUROC.

Figure 22: Per-task test nMAE for context ensembling, default vs. per-task-tuned context, on RelBench regression tasks; per-task counterpart of Figure 5 (left).

Figure 23: Per-task test AUROC for context ensembling, default vs. per-task-tuned context, on RelBench classification tasks; per-task counterpart of Figure 5 (right).

Explore similar work

Relational In-Context Learning via Synthetic Pre-training with Structural Prior
Mar 4, 2026Yanbo Wang, Jiaxuan You, Chuan Shi +1Relational Foundation Models Relational Databases
Institute for Artificial Intelligence, Peking University · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Beijing University of Post and Telecommunication +1
OpenRFM: Dissecting Relational In-Context Learning
Jun 3, 2026Zhikai Chen, Junyu Yin, Jialiang Gu +5Relational Foundation Models Synthetic Data Pretraining
Michigan State University · George Mason University · Georgia Institute of Technology +1
KumoRFM-2: Scaling Foundation Models for Relational Learning
Apr 14, 2026Valter Hudovernik, Federico López, Vid Kocijan +4Relational Foundation Models Relational Deep Learning

Mar 4, 2026J/K move · Enter open · S save

Relational In-Context Learning via Synthetic Pre-training with Structural Prior

Yanbo Wang, Jiaxuan You, Chuan Shi, Muhan Zhang

Institute for Artificial Intelligence, Peking University · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Beijing University of Post and Telecommunication+1

Relational Foundation Models Relational Databases In-Context Learning Synthetic Data Pretraining Synthetic Data Generation

Open paper AskPDF arXiv

Relational Databases (RDBs) are the backbone of modern business, yet they lack foundation models comparable to those in text or vision. A key obstacle is that high-quality RDBs are private, scarce, and structurally heterogeneous, making internet-scale pre-training infeasible. To overcome this data scarcity, we introduce RDB-PFN, the first relational foundation model trained purely via synthetic data. Inspired by Prior-Data Fitted Networks (PFNs), where synthetic data generated from Structural Causal Models (SCMs) enables reasoning on single tables, we design a Relational Prior Generator to create an infinite stream of diverse RDBs from scratch. Pre-training on over 2 million synthetic single-table and relational tasks, RDB-PFN learns to adapt to any new database instantly via genuine in-context learning. Experiments show that RDB-PFN achieves strong few-shot performance on 19 real-world relational prediction tasks, outperforming state-of-the-art tabular foundation models evaluated on the same DFS-linearized inputs, while using a lightweight architecture and fast inference. The code is available at https://github.com/MuLabPKU/RDBPFN.

Abstract

Explore similar work

Relational In-Context Learning via Synthetic Pre-training with Structural Prior

OpenRFM: Dissecting Relational In-Context Learning

KumoRFM-2: Scaling Foundation Models for Relational Learning