When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior

Shivam Dubey; Mohamed Bouadi; Nassim Bouarour; Varun Kulkarni; Aditya Tanna; Vinay Kumar Sankarapu

cs.LGOct 2, 2026

When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior

Authors: Shivam Dubey, Mohamed Bouadi, Nassim Bouarour, Varun Kulkarni, Aditya Tanna, Vinay Kumar Sankarapu

Organizations: Lexsi Labs

Abstract

Relational foundation models are increasingly pretrained on synthetic databases, yet downstream benchmarks reveal little about why one synthetic corpus produces a better model than another. In particular, strong performance may arise from realistic row-level statistics without the model ever learning to use relational structure. We study this as a data-attribution problem: which property of synthetic pretraining data induces relational computation? Using four Relational Transformer checkpoints trained with the same architecture, initialization, objective, and compute budget on corpora produced by four relational data generators, we trace a measurable property of the data to learned computation and downstream behavior. We hypothesize that relational mechanisms emerge when cross-table information is predictively necessary for the masked-cell pretraining objective. RelDiff exhibits by far the largest predictive gain from foreign-key-linked parents, and its corresponding model is uniquely sensitive to foreign-key interventions on unseen databases. This dependence survives a random-initialization control, grows monotonically with the fraction of corrupted links, and localizes to a serial cross-table pathway. Finally, disrupting the same mechanism during downstream inference removes RelDiff's advantage on relational tasks while leaving structure-insensitive models nearly unchanged. These results connect a property of synthetic training data to a learned mechanism and, through intervention, to downstream behavior.

Figures & tables

Corpus	Targets	$N_{\mathrm{p}}/N_{\mathrm{c}}$	Row-only $R^{2}$	$\Delta_{\mathrm{parent}}$	$\Delta_{\mathrm{child}}$
RelDiff	48	30/22	0.022	+0.072 $[+0.018,+0.091]$	$-0.001$
GRDM	120	65/78	$-0.011$	$-0.057$ $[-0.076,-0.044]$	$-0.105$
PluRel	113	84/71	0.166	+0.009 $[+0.001,+0.022]$	$-0.000$
RDB-PFN	120	72/68	0.436	+0.001 $[-0.004,+0.009]$	$-0.005$

Table 1: Predictive structure in the synthetic pretraining corpora. Row-only

R^{2}

measures target predictability without cross-table information.

\Delta_{\mathrm{parent}}

and

\Delta_{\mathrm{child}}

measure the additional predictive value of FK-linked parent attributes and simple child aggregates, respectively.

N_{\mathrm{p}}/N_{\mathrm{c}}

gives the number of target-column problems for which each relational feature set is available.

Model	$\mathcal{L}_{0}$	$\Delta\mathcal{L}_{\mathrm{nbr}}$	$\Delta\mathcal{L}_{\mathrm{shuffle}}$	$\Delta\mathcal{L}_{\mathrm{remove}}$	Rel. FK change
RelDiff	0.241	+0.127	+0.202	+0.197	+83.6%
GRDM	0.342	$-0.002$	+0.018	+0.034	+5.3%
PluRel	0.161	$-0.000$	+0.010	+0.019	+6.1%
RDB-PFN	0.208	+0.003	+0.007	+0.015	+3.4%
Random init	0.556	$-0.025$	+0.027	+0.057	+4.9%

Table 2: Functional reliance on relational structure on rel-stack . Entries are original masked-cell loss and absolute changes under three interventions. The final column reports the relative loss increase under full FK shuffling.

Figure 1: From data-side relational signal to model-side structural reliance. Left: median gain in held-out

R^{2}

obtained by adding FK-linked parent attributes to a row-only predictor; error bars are 95% bootstrap intervals. Right: increase in masked-cell reconstruction loss as an increasing fraction of rel-stack FK assignments is corrupted. RelDiff exhibits both the strongest parent-linked predictive signal and the steepest functional response to corruption.

Model	Original	$\Delta\mathcal{L}_{\mathrm{nbr}}$	$\Delta\mathcal{L}_{\mathrm{shuffle}}$	$\Delta\mathcal{L}_{\mathrm{remove}}$
RelDiff	0.339	+0.029	+0.069	+0.065
GRDM	0.350	+0.013	$-0.005$	$-0.007$
PluRel	0.339	+0.007	+0.009	+0.014
RDB-PFN	0.351	$-0.002$	+0.003	+0.007

Table 3: Structural reliance averaged over three real RelBench databases. Entries are mean original masked-cell loss and absolute changes under each intervention. The experiment uses a broader 27-batch probe suite than Table 2 .

Panel of Figure 2 — Figure 2: Pathway-specific intervention localizes a serial cross-table computation. Neighbor attention aggregates linked children into a parent representation, which subsequently reaches the target through parent access in feat . Blocking parent access makes additional nbr ablation behaviorally redundant.

Figure 3: Behavioral importance of individual nbr attention heads. Individual head ablations reveal concentrated functional specialization in RelDiff. Its most important head accounts for approximately

39.5\%

of the positive head-attributable loss increase, compared with

12.5\%

under uniform contribution from eight heads.

Figure 4: Mean classification AUROC as a function of in-context length. Each point averages the six classification tasks. The 30k points incorporate the

n=2048

re-evaluation of the two rare-label rel-stack tasks. Under the corrected evaluation all four checkpoints degrade from context 1,024 to 30k, with RelDiff showing the smallest decline.

Context	Model	Original	FK shuffled	Nbr ablated	$\Delta_{\mathrm{FK}}$	$\Delta_{\mathrm{nbr}}$
30k	RelDiff	0.865	0.579	0.624	$-0.286$	$-0.241$
30k	GRDM	0.520	0.622	0.537	+0.102	+0.018
30k	PluRel	0.652	0.650	0.636	$-0.003$	$-0.017$
30k	RDB-PFN	0.642	0.643	0.634	+0.001	$-0.008$
1,024	RelDiff	0.847	0.718	0.728	$-0.129$	$-0.119$
1,024	GRDM	0.609	0.618	0.622	+0.009	+0.013

Table 4: Downstream intervention on rel-stack user-engagement. Entries are AUROC on identical examples under standard inference, FK shuffling, and internal nbr ablation. The 30k evaluation uses

n=2048

examples (60 positives).

\Delta_{\mathrm{FK}}

and

\Delta_{\mathrm{nbr}}

are changes relative to original inference.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Generator	Reference data	Relational structure	Value generation
RelDiff	Yes	fitted to reference FK graph	joint graph-conditioned diffusion
GRDM	Yes	reference structure / degree statistics	graph-conditional diffusion
PluRel	No	procedural relational sampler	conditional causal mechanisms
RDB-PFN	No	hand-designed relational prior	SCM-style synthetic prior

Appendix

Table 5: Summary of the four synthetic pretraining generators. “Reference-conditioned” denotes generation tied to the schema or relational structure of an existing database; “reference-free” denotes generation from a synthetic prior.

Corpus	Numerical	Text / categorical	Boolean
RelDiff	512	234	0
GRDM	518	0	0
PluRel	1132	0	518
RDB-PFN	1081	0	0

Appendix

Table 6: Column-type census in the available synthetic corpora. Counts are over the sampled databases used in the audit. The available GRDM export contains no preserved text/categorical columns: string and datetime fields are represented numerically.

Figure 5: Task-level relational predictive gains. Each point corresponds to one target-column prediction problem. RelDiff exhibits substantially larger gains from linked-parent attributes, whereas the simple child aggregates considered here provide little additional predictive signal.

Corpus	Feature	Table	Cross-table
RelDiff	1.10	0.57	1.04
GRDM	0.69	0.43	0.71
PluRel	0.79	0.34	0.85
RDB-PFN	0.82	0.16	0.81

Appendix

Table 7: Diversity of the generated corpora. Values are mean pairwise embedding distances along feature-, table-, and cross-table axes. RelDiff has the largest measured diversity, but the ordering of the remaining corpora does not reproduce the ordering of functional relational reliance.

Figure 6: Output diversity measured directly in the generated databases. RelDiff has the largest diversity on all three axes, but diversity alone does not reproduce the ordering of functional relational reliance.

Model	Encoders	Col	Feat	Nbr	FFN
GRDM	4.386	3.440	4.047	3.129	3.525
PluRel	2.345	2.893	2.636	2.198	2.504
RDB-PFN	2.311	3.142	2.455	2.142	2.445
RelDiff	3.690	3.025	3.315	3.622	3.089

Appendix

Table 8: Per-parameter checkpoint deviation by RT component. Entries are RMS deviations from the four-checkpoint mean, multiplied by

10^{3}

. Among the attention streams, RelDiff has its largest deviation in nbr , whereas GRDM has its largest deviation in feat .

Figure 7: Checkpoint deviation by attention stream and depth. Weight-space differences identify components that were affected differently during pretraining. Functional importance is established separately by intervention.

Checkpoint pair	Early CKA	Late CKA
GRDM / RelDiff	0.944	0.824
PluRel / RelDiff	0.944	0.810
GRDM / PluRel	0.938	0.718
PluRel / RDB-PFN	0.909	0.707
RDB-PFN / RelDiff	0.870	0.587
GRDM / RDB-PFN	0.808	0.406

Appendix

Table 9: Pairwise linear CKA on identical real-data inputs. Early and late values average blocks 0–5 and 6–11, respectively.

Model	Normalized entropy	Top-1 mass
RelDiff	0.843	0.396
GRDM	0.858	0.394
RDB-PFN	0.890	0.361
PluRel	0.898	0.361

Appendix

Table 10: Within-neighborhood attention statistics. Lower normalized entropy indicates more selective attention among the FK-linked cells permitted by the architectural mask.

Relational Foundation Models

Relational foundation models learn reusable representations and predictors for tasks spanning linked tables and databases.

See all 17 papers

Relational In-Context Learning via Synthetic Pre-training with Structural Prior
Mar 4, 2026Yanbo Wang, Jiaxuan You, Chuan Shi +1Relational ICL Relational Foundation Models
Institute for Artificial Intelligence, Peking University · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Beijing University of Post and Telecommunication +1
Curriculum Matters: Data-Efficient Relational PFN Pretraining with Synthetic Data
Jul 31, 2026Mohammad Sadeq Abolhasani, Viswanath GanapathySynthetic Data Pretraining Synthetic Data Generation
SAP Labs, LLC. Palo Alto, CA, United States
PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models
Feb 3, 2026Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik +4Relational Foundation Models Synthetic Data Pretraining
1Stanford University · 2Kumo AI · 3SAP Labs LLC

Mar 4, 2026J/K move · Enter open · S save

Relational In-Context Learning via Synthetic Pre-training with Structural Prior

Yanbo Wang, Jiaxuan You, Chuan Shi, Muhan Zhang

Institute for Artificial Intelligence, Peking University · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Beijing University of Post and Telecommunication+1

Relational ICL Relational Foundation Models Relational Databases In-Context Learning Synthetic Data Pretraining Synthetic Data Generation

Open paper AskPDF arXiv

Relational Databases (RDBs) are the backbone of modern business, yet they lack foundation models comparable to those in text or vision. A key obstacle is that high-quality RDBs are private, scarce, and structurally heterogeneous, making internet-scale pre-training infeasible. To overcome this data scarcity, we introduce RDB-PFN, the first relational foundation model trained purely via synthetic data. Inspired by Prior-Data Fitted Networks (PFNs), where synthetic data generated from Structural Causal Models (SCMs) enables reasoning on single tables, we design a Relational Prior Generator to create an infinite stream of diverse RDBs from scratch. Pre-training on over 2 million synthetic single-table and relational tasks, RDB-PFN learns to adapt to any new database instantly via genuine in-context learning. Experiments show that RDB-PFN achieves strong few-shot performance on 19 real-world relational prediction tasks, outperforming state-of-the-art tabular foundation models evaluated on the same DFS-linearized inputs, while using a lightweight architecture and fast inference. The code is available at https://github.com/MuLabPKU/RDBPFN.

Abstract

Relational Foundation Models

Explore similar work

Relational In-Context Learning via Synthetic Pre-training with Structural Prior

Curriculum Matters: Data-Efficient Relational PFN Pretraining with Synthetic Data

PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models