Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

Yixin Peng; Er Jin; Diego Collarana; Stefan Decker

cs.LGSep 28, 2026

Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

Authors: Yixin Peng, Er Jin, Diego Collarana, Stefan Decker

Organizations: RWTH Aachen University, Aachen, Germany · Fraunhofer FIT, Sankt Augustin, Germany

Abstract

Relational deep learning models database rows and foreign-key links as a heterogeneous graph for prediction from record attributes and relational context. These graphs contain two distinct temporal signals: record age changes with the prediction cutoff, while intervals between observed records remain fixed. Prior work often treats time as a single signal or studies temporal representation and pretraining separately. We investigate how explicitly encoding both signals affects temporal pretraining for downstream tasks. Our framework combines Multi-scale Time Encoding, which captures record age using learnable time scales and type-specific projections, with Rotary Time Encoding, which represents signed inter-record intervals through rotary transformations during graph propagation. We pair these encodings with three self-supervised objectives: historical relation recovery, horizon-aware future relation activity prediction, and temporal subgraph contrast. All inputs respect their observation cutoffs. Pretraining proceeds in two stages: subgraph contrast first learns neighborhood representations, followed by refinement through either relation recovery or future activity prediction. We evaluate on five RelBench datasets across 11 classification and regression tasks using heterogeneous GNN and graph Transformer backbones. With both encodings, the best evaluated staged schedules improve over supervised training with the same encodings by 3.02% and 1.06% on the two backbones, respectively, and over controls without pretraining or either encoding by 3.24% and 2.37%.

Figures & tables

Figure 1: From relational tables to historical subgraphs for RDL. (a) Customer, purchase, and product tables: id denotes primary keys, c_id and p_id denote foreign keys, and time records purchase observation times. (b) Rows become nodes whose types follow their source tables; foreign-key references become typed edges with corresponding reverse relations for bidirectional message passing. Purchase nodes carry timestamps

t_{v}

. (c) An illustrative two-hop sample around seed C1, using only information available by cutoff

\tau=\text{day }5

. S4 is excluded because its timestamp exceeds

\tau

; S3 and C2 lie outside the two-hop neighborhood.

Figure 2: Comparison of representative graph pretraining strategies. In each of the first three columns, paired symbols indicate coverage of the supervision granularity and explicit use of node or relation types at that granularity, respectively. Filled and half-filled circles denote direct and indirect or limited coverage; B denotes backbone-dependent encoding, and a dash denotes no explicit coverage.

Figure 3: Relative test-performance gains (%) across 11 tasks, computed from task metrics averaged over four random seeds. All panels use the same-backbone supervised control without our time encodings (only with DayPE) or pretraining as reference (A0). Mean type gain averages gains within each task type and then equally across types; average rank weights all tasks equally (lower is better). Colors saturate at $\pm 10\%$ ; white dots mark values beyond this range. TS denotes Temp-Sub ; outlined rows identify our two-stage schedules, and bold summary values mark the best result in each panel.

Figure 4: Contributions of temporal encoding and staged pretraining. (a) Gain differences from ours ( TimeMix + TimeRoPE ), without pretraining. DayPE is the single-scale baseline for TimeMix . (b) Pretraining gains over matched supervised controls (A0: DayPE ; B0: TimeMix + TimeRoPE ). Arrows connect single- and two-stage results with equal total pretraining steps.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Database	Domain	Tables	Rows	FK relations	Tasks
Arxiv	Scholarly publications	6	2,733,846	6	2
Avito	Online advertising	8	24,653,915	11	3
F1	Motor racing	9	97,606	13	2
H&M	Fashion retail	3	16,931,173	2	2
Event	Social events	5	44,822,992	7	2

Appendix

Table 1: Stored database statistics. FK denotes foreign-key columns; Tasks counts the tasks evaluated here.

Database	Task	Type	Train	Validation	Test	Entities
Arxiv	author-category	Multiclass classification	210,769	39,015	39,655	126,219
	paper-citation	Binary classification	534,233	155,845	193,696	193,696
Avito	user-clicks	Binary classification	59,454	21,183	47,996	66,449
	user-visits	Binary classification	86,619	29,979	36,129	63,405
	ad-ctr	Regression	5,100	1,766	1,816	4,997
F1	driver-dnf	Binary classification	11,411	566	702	821

Appendix

Table 2: Task sizes in the stored splits. Entities are counted once across training, validation, and test.

		GNN		GT
Task	Metric	HeteroGNN	RelGNN	HGT	RelGT	THGFM
arxiv/author-category	Acc. $\uparrow$	$0.1311{\pm}0.0020$	$0.1298{\pm}0.0010$	$\mathbf{0.1331{\pm}0.0017}$	$0.1037{\pm}0.0050$	$0.1324{\pm}0.0027$
arxiv/paper-citation	AUC $\uparrow$	$0.6495{\pm}0.0069$	$0.6216{\pm}0.0095$	$0.6583{\pm}0.0074$	$0.6550{\pm}0.0027$	$\mathbf{0.6652{\pm}0.0019}$
avito/user-clicks	AUC $\uparrow$	$0.6435{\pm}0.0021$	$\mathbf{0.6496{\pm}0.0336}$	$0.6369{\pm}0.0032$	$0.6298{\pm}0.0185$	$0.6405{\pm}0.0008$
avito/user-visits	AUC $\uparrow$	$\mathbf{0.6938{\pm}0.0012}$	$0.6735{\pm}0.0014$	$0.6763{\pm}0.0095$	$0.6899{\pm}0.0186$	$0.6830{\pm}0.0075$
f1/driver-dnf	AUC $\uparrow$	$0.7442{\pm}0.0042$	$\mathbf{0.8020{\pm}0.0042}$	$0.7978{\pm}0.0064$	$0.7830{\pm}0.0117$	$0.7999{\pm}0.0068$

Appendix

Table 3: Backbone validation performance (mean

\pm

standard deviation) used to select one representative per family. Ranks are computed across all five models for each task and averaged equally over all 11 tasks. Bold marks the best task mean, the highest task-win count, and the lowest mean rank within each family; arrows indicate the preferred direction.

Panel A: HeteroGNN ( GNN backbone)
	Multiclass classification		Binary classification							Regression
Encoding	Author	Mean task gain	Citation	Ignore	DNF	Churn	Clicks	Visits	Mean task gain	Attend.	Position	Sales	CTR	Mean task gain	Mean type gain
	Acc. $\uparrow$	$\Delta\%\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	$\Delta\%\uparrow$	MAE $\downarrow$	MAE $\downarrow$	MAE $\downarrow$	MAE $\downarrow$	$\Delta\%\uparrow$	$\Delta\%\uparrow$
DayPE (reference)	$0.1236{\pm}0.0005$	$+0.00\%$	$0.6450{\pm}0.0047$	$0.8031{\pm}0.0157$	$0.7101{\pm}0.0105$	$\mathbf{0.6961{\pm}0.0019}$	$\mathbf{0.6474{\pm}0.0164}$	$\mathbf{0.6627{\pm}0.0008}$	$\mathbf{+0.00\%}$	$0.2620{\pm}0.0013$	$4.3596{\pm}0.0168$	$\mathbf{0.0604{\pm}0.0001}$	$\mathbf{0.0403{\pm}0.0003}$	$\mathbf{+0.00\%}$	$+0.00\%$
TimeMix	$\mathbf{0.1265{\pm}0.0028}$	$\mathbf{+2.35\%}$	$0.6430{\pm}0.0043$	$0.8083{\pm}0.0095$	$\mathbf{0.7316{\pm}0.0043}$	$0.6868{\pm}0.0028$	$0.6282{\pm}0.0042$	$0.6355{\pm}0.0090$	$-0.84\%$	$0.2577{\pm}0.0083$	$4.3178{\pm}0.0382$	$0.0638{\pm}0.0015$	$0.0408{\pm}0.0006$	$-0.98\%$	$+0.18\%$
DayPE + TimeRoPE	$0.1239{\pm}0.0017$	$+0.24\%$	$\mathbf{0.6527{\pm}0.0036}$	$0.8202{\pm}0.0051$	$0.7227{\pm}0.0099$	$0.6896{\pm}0.0052$	$0.6318{\pm}0.0022$	$0.6454{\pm}0.0050$	$-0.14\%$	$0.2616{\pm}0.0021$	$4.1803{\pm}0.1225$	$0.0617{\pm}0.0006$	$0.0421{\pm}0.0001$	$-0.49\%$	$-0.13\%$

Appendix

Table 4: Temporal encoding ablation (mean

\pm

standard deviation over four runs). Gains are relative to the same backbone with DayPE ( Equation 10 ) and without TimeRoPE or pretraining in each panel. TimeMix replaces DayPE ; TimeRoPE adds rotary interval encoding to the specified record-age encoder. Mean type gain weights the three task types equally. Bold marks the best entry per panel and column.

Figure 5: Record-age distributions by task: (a) all nine intervals and (b) a magnified view of ages up to one day, on the original percentage scale.

Figure 6: Support for Rel-Future horizons: (a) the relation-balanced percentage of eligible events supporting each horizon and (b) the resulting horizon sampling distribution.

Panel A: DayPE without TimeRoPE . All $\Delta$ columns use Ref. A0 below.
	Pretraining strategy		Multiclass classification		Binary classification							Regression
Temporal	Graph-level	Local stage	Author	Task gain	Citation	Ignore	DNF	Churn	Clicks	Visits	Mean task gain	Attend.	Position	Sales	CTR	Mean task gain	Mean type gain
			Acc. $\uparrow$	$\Delta\%\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	$\Delta\%\uparrow$	MAE $\downarrow$	MAE $\downarrow$	MAE $\downarrow$	MAE $\downarrow$	$\Delta\%\uparrow$	$\Delta\%\uparrow$
w/o	No pretraining ( Ref. A0 )		$0.1236{\pm}0.0005$	$+0.00\%$	$0.6450{\pm}0.0047$	$0.8031{\pm}0.0157$	$0.7101{\pm}0.0105$	$0.6961{\pm}0.0019$	$0.6474{\pm}0.0164$	$0.6627{\pm}0.0008$	$+0.00\%$	$0.2620{\pm}0.0013$	$4.3596{\pm}0.0168$	$0.0604{\pm}0.0001$	$0.0403{\pm}0.0003$	$+0.00\%$	$+0.00\%$
Baseline pretraining strategies
w/o	–	Un-SAGE	$0.1209{\pm}0.0039$	$-2.18\%$	$0.5023{\pm}0.0053$	$0.8206{\pm}0.0286$	$0.7038{\pm}0.0284$	$0.6902{\pm}0.0011$	$0.6482{\pm}0.0096$	$0.6570{\pm}0.0032$	$-3.74\%$	$0.2632{\pm}0.0059$	$4.3955{\pm}0.0202$	$0.0565{\pm}0.0008$	$\mathbf{0.0362{\pm}0.0002}$	$+4.24\%$	$-0.56\%$

Appendix

Table 5: Pretraining transfer on HeteroGNN (mean

\pm

standard deviation over four runs). Panel A uses DayPE without TimeRoPE ; Panel B uses TimeMix with TimeRoPE . Gains are relative to the marked reference within each panel. Bold marks the best value per panel and column.

Panel A: DayPE without TimeRoPE . All $\Delta$ columns use Ref. A0 below.
	Pretraining strategy		Multiclass classification		Binary classification							Regression
Temporal	Graph-level	Local stage	Author	Task gain	Citation	Ignore	DNF	Churn	Clicks	Visits	Mean task gain	Attend.	Position	Sales	CTR	Mean task gain	Mean type gain
			Acc. $\uparrow$	$\Delta\%\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	$\Delta\%\uparrow$	MAE $\downarrow$	MAE $\downarrow$	MAE $\downarrow$	MAE $\downarrow$	$\Delta\%\uparrow$	$\Delta\%\uparrow$
w/o	No pretraining ( Ref. A0 )		$\mathbf{0.1235{\pm}0.0020}$	$\mathbf{+0.00\%}$	$0.6603{\pm}0.0012$	$0.8194{\pm}0.0326$	$0.7396{\pm}0.0068$	$0.6969{\pm}0.0041$	$0.6346{\pm}0.0023$	$0.6369{\pm}0.0033$	$+0.00\%$	$0.2439{\pm}0.0009$	$4.3693{\pm}0.0132$	$0.0613{\pm}0.0006$	$0.0431{\pm}0.0001$	$+0.00\%$	$+0.00\%$
Baseline pretraining strategies
w/o	–	Un-SAGE	$0.1194{\pm}0.0009$	$-3.32\%$	$\mathbf{0.6644{\pm}0.0054}$	$0.8631{\pm}0.0352$	$0.7052{\pm}0.0348$	$0.6944{\pm}0.0014$	$0.6297{\pm}0.0087$	$0.6228{\pm}0.0046$	$-0.34\%$	$0.2635{\pm}0.0032$	$4.6955{\pm}0.0240$	$0.0581{\pm}0.0005$	$0.0429{\pm}0.0002$	$-2.10\%$	$-1.92\%$

Appendix

Table 6: Pretraining transfer on THGFM (mean

\pm

Attribute type	PyTorch Frame encoder	Operation
Categorical	EmbeddingEncoder	Learned category lookup.
Numerical	LinearEncoder	Standardize using column statistics, then apply a learned affine map. Missing values are imputed with the column mean.
Multicategorical	MultiCategoricalEmbeddingEncoder	Average learned embeddings of the categories in a cell.
Vector / text embedding	LinearEmbeddingEncoder	Apply a learned affine map to the input vector.
Timestamp	TimestampEncoder	Encode the year positionally and other calendar fields cyclically, then apply a learned linear map with bias.

Appendix

Table 7: Column encoders used by both backbones. Only types present in a table are instantiated; all outputs have dimension 128.

Setting	GNN ( HeteroGNN )	GT ( THGFM )
Graph layers / hidden width	3 / 128	3 / 128
Table feature encoder	4-layer ResNet, width 128	4-layer ResNet, width 128
Table-encoder dropout	0.2	0.2
Text embedding / learned mapping	Fixed GloVe / $300\rightarrow 128$	Fixed GloVe / $300\rightarrow 128$
Heads per attention branch	Not applicable	8 (16 dimensions each)
Graph-operator dropout	None	0.05

Appendix

Table 8: Architecture settings for the selected GNN and GT . Temporal settings apply when the corresponding proposed encoding is enabled.

Setting	GNN and GT
Optimizer / learning rate / weight decay	AdamW / $2\times 10^{-3}$ / $10^{-5}$
Learning-rate schedule	Constant; no warmup or decay
Single-stage pretraining epochs / steps per epoch	10 / 10,000
Two-stage pretraining epochs / steps per epoch (each stage)	10 / 5,000
Total pretraining steps (either schedule)	100,000
Pretraining batch size	256; 512 for the H&M curricula

Appendix

Table 9: Training settings shared by the selected GNN and GT . Single- and two-stage pretraining each receive 100,000 steps in total.

Figure 7: Pretraining metrics for HeteroGNN with DayPE and without TimeRoPE : single objectives.

Figure 8: Pretraining metrics for HeteroGNN with DayPE and without TimeRoPE : single objectives and stage 2 after Temp-Sub .

Figure 9: Fine-tuning validation metrics for HeteroGNN with DayPE and without TimeRoPE : single-stage baselines.

Figure 10: Fine-tuning validation metrics for HeteroGNN with DayPE and without TimeRoPE : single and staged objectives.

Figure 11: Pretraining losses for HeteroGNN with DayPE and without TimeRoPE : CTRL, TGPM, TVE, and the shared Temp-Sub first stage.

Figure 12: Fine-tuning validation metrics for HeteroGNN with DayPE and without TimeRoPE : CTRL, TGPM, TVE, and their TS-initialized variants.

Figure 13: Pretraining metrics for HeteroGNN with TimeMix and TimeRoPE : single objectives.

Figure 14: Pretraining metrics for HeteroGNN with TimeMix and TimeRoPE : single objectives and stage 2 after Temp-Sub .

Figure 15: Fine-tuning validation metrics for HeteroGNN with TimeMix and TimeRoPE : single-stage baselines.

Figure 16: Fine-tuning validation metrics for HeteroGNN with TimeMix and TimeRoPE : single and staged objectives.

Figure 17: Pretraining losses for HeteroGNN with TimeMix and TimeRoPE : CTRL, TGPM, TVE, and the shared Temp-Sub first stage.

Figure 18: Fine-tuning validation metrics for HeteroGNN with TimeMix and TimeRoPE : CTRL, TGPM, TVE, and their TS-initialized variants.

Figure 19: Pretraining metrics for THGFM with DayPE and without TimeRoPE : single objectives.

Figure 20: Pretraining metrics for THGFM with DayPE and without TimeRoPE : single objectives and stage 2 after Temp-Sub .

Figure 21: Fine-tuning validation metrics for THGFM with DayPE and without TimeRoPE : single-stage baselines.

Figure 22: Fine-tuning validation metrics for THGFM with DayPE and without TimeRoPE : single and staged objectives.

Figure 23: Pretraining losses for THGFM with DayPE and without TimeRoPE : CTRL, TGPM, TVE, and the shared Temp-Sub first stage.

Figure 24: Fine-tuning validation metrics for THGFM with DayPE and without TimeRoPE : CTRL, TGPM, TVE, and their TS-initialized variants.

Figure 25: Pretraining metrics for THGFM with TimeMix and TimeRoPE : single objectives.

Figure 26: Pretraining metrics for THGFM with TimeMix and TimeRoPE : single objectives and stage 2 after Temp-Sub .

Figure 27: Fine-tuning validation metrics for THGFM with TimeMix and TimeRoPE : single-stage baselines.

Figure 28: Fine-tuning validation metrics for THGFM with TimeMix and TimeRoPE : single and staged objectives.

Figure 29: Pretraining losses for THGFM with TimeMix and TimeRoPE : CTRL, TGPM, TVE, and the shared Temp-Sub first stage.

Figure 30: Fine-tuning validation metrics for THGFM with TimeMix and TimeRoPE : CTRL, TGPM, TVE, and their TS-initialized variants.

Explore similar work

May 15, 2026cs.LG

Gaussian Relational Graph Transformer

Relational graph learning enables predictive modeling over relational databases by directly capturing dependencies across interconnected tables. While relational graph Transformers extend the receptive field beyond local message passing, a larger receptive field does not necessarily provide more useful information: sampling must preserve relational structure without introducing excessive semantically irrelevant nodes, while attention must account for the temporal relevance of the retained information. We propose GelGT, a Gaussian relational graph transformer that explicitly addresses these challenges. GelGT introduces a structure-semantic collaborative sampling strategy to preserve structural connectivity while filtering irrelevant semantic information, and incorporates a Gaussian graph attention mechanism with a learnable Gaussian bias on the sampled subgraphs to dynamically encode temporal dependencies. We provide theoretical analysis characterizing the structural preservation, semantic refinement, and temporal discrimination of these mechanisms. Experiments on \textbf{7} real-world relational datasets covering \textbf{21} prediction tasks show that GelGT consistently outperforms existing relational graph learning methods, with improvements of up to \textbf{13.8%}.

Zezhong Ding, Jin Li, Xugang Wang +1

School of Artificial Intelligence and Data Science, University of Science and Technology of China (USTC) · Data Darkness Lab, Suzhou Institute for Advanced Research, USTC · School of Biomedical Engineering, USTC +1

Multiplex Graph Transformers Relational Deep Learning Interleaved Graph Attention Transformer Architectures

Ask PDF

Jul 29, 2026cs.LG

THGFM: Dual-Branch Temporal Heterogeneous Graph Fusion Model

Temporal heterogeneous graphs offer a natural abstraction for dynamic relational systems in which diverse node and relation types co-exist and evolve over time. Learning on such graphs requires jointly modeling cross-type structural heterogeneity and the temporal dynamics of interactions, yet existing methods still struggle to reconcile parameter-efficient cross-type transfer with relation-aware specialization, and typically inject time only as additive features outside the attention kernel. We propose \textbf{THGFM}, a web-scale temporal heterogeneous graph fusion model that addresses both limitations within a unified dual-path architecture. THGFM couples a \textit{Shared-Space Temporal Attention} branch for parameter-efficient cross-type transfer with a \textit{Relational Type-Partitioned Temporal Attention} branch for relation-aware specialization, and integrates them through \textit{Dual-Path Relational--Shared Fusion}, instantiated with \textit{Type-Conditioned Non-Competitive Gated Sum Fusion}: a adaptive mechanism that assigns independent, type-conditioned feature-wise gates to the shared and specialized branches, allowing both to be amplified or suppressed without zero-sum competition. To directly incorporate relative time into the attention score, THGFM further introduces \textit{Rotary Temporal Attention}, which rotates queries and keys by half-phases of relative time before matching. THGFM consistently outperforms baseline graph transformer models on academic graphs benchmarks, delivering a

+3.25\%

six-task mean gain, with peak relative gains of

+12.37\%

on OAG-CS PV,

+4.87\%

on PF-

L_2

, and

+1.18\%

on PF-

L_1

, and

+4.24\%

+3.73\%

, and

+4.61\%

on OGBN-MAG, HTAG-ArXiv, and HTAG-DBLP, respectively.

Yixin Peng, Diego Collarana, Er Jin +1

RWTH Aachen University, Aachen, Germany · Fraunhofer FIT, Sankt Augustin, Germany

Temporal Graph Neural Networks Multi-Stream Temporal Fusion Heterogeneity Entities

Ask PDF

May 15, 2026cs.DB

Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks

Relational databases store much of the world's structured information, and they are essential for driving complex predictive applications. However, deep learning progress on relational data remains limited, as conventional approaches flatten databases into single tables via manual feature engineering, discarding relational context. Relational deep learning (RDL) addresses this by modeling databases as relational entity graphs (REGs) for graph neural networks (GNNs), but remains task- and database-specific. To combine the strengths of both paradigms, we propose a hybrid architecture combining a fine-tuned BART encoder to capture intra-row semantics with a GraphSAGE-based GNN over REGs to inject relational context. Experiments on RelBench show that the GNN substantially enriches BART's row embeddings, achieving a ROC-AUC of 67.40 on the driver-dnf task from the rel-f1 dataset. This performance is competitive with supervised baselines such as LightGBM (68.86) and narrows the gap to RDL (72.62) to within 5.22 points, though a substantial gap remains to state-of-the-art foundation models such as KumoRFM (82.63). These results suggest that lightweight hybrid LM-GNN architectures offer a promising and resource-efficient path towards foundation models for relational databases.

Jingcheng Wu, Ratan Bahadur Thapa, Mojtaba Nayyeri +4

University of Stuttgart, Stuttgart, Germany · Web and Internet Science Research Group, University of Southampton, Southampton, United Kingdom

Relational Deep Learning Relational Databases Graph Neural Networks Wireless Foundation Models Deep Learning Text Embeddings

Ask PDF