FTD-GNO: Memory-Efficient Graph Neural Operators through Functional Tensor Decomposition of the Kernel

Xiaomin Zhang; Boyue Wang; Junbin Gao; Yongli Hu anbd Baocai Yin

cs.LGOct 3, 2026

FTD-GNO: Memory-Efficient Graph Neural Operators through Functional Tensor Decomposition of the Kernel

Authors: Xiaomin Zhang, Boyue Wang, Junbin Gao, Yongli Hu anbd Baocai Yin

Organizations: Beijing Key Laboratory of Multimedia and Intelligent Software Technology Beijing Artificial Intelligence Institute School of Information Science and Technology Beijing University of Technology, Beijing 100124, China · Discipline of Business Analytics The University of Sydney Business School The University of Sydney, Camperdown NSW 2006, Australia

Abstract

Graph Neural Operators (GNOs) provide flexible surrogate models for learning solution operators of partial differential equations (PDEs). However, standard GNOs typically parameterize the integral kernel with a monolithic neural network and evaluate kernel interactions over graph edges, leading to substantial computational and memory overhead at high resolutions or with large neighborhoods. To address these limitations, we propose Functional Tensor Decomposition Graph Neural Operator (FTD-GNO), a memory-efficient GNO framework that decouples the high-dimensional continuous integral kernel into low-dimensional mode-wise functions. By instantiating the kernel with classical tensor decomposition formats, including CP, Tensor-Train, and Tucker decompositions, FTD-GNO enables algebraic reconstruction of the integral operator without explicitly materializing full edge-wise kernel tensors. This factorized formulation reduces the memory footprint of kernel evaluation and aggregation while retaining the continuous operator-learning structure of GNOs. Theoretical complexity analysis shows that FTD-GNO substantially lowers parameter and activation-memory costs associated with high-dimensional kernel construction. Experiments show lower peak memory than the corresponding unfactorized graph-integral baselines, with shorter recorded training times. Fourier-graph experiments further demonstrate that FTD can improve the efficiency of a graph-integral layer within a hybrid operator and has good scalability.

Figures & tables

Figure 1: Efficient GNO via Function Tensor Decomposition: GNO calculates the integral kernel through a single large neural network and explicitly constructs the complete integral kernel tensor on each edge. The proposed methodology involves the decomposition of the integral kernel into multiple single-variable neural networks, in conjunction with the utilisation of configurable tensor decomposition modes. This approach facilitates the reconstruction of the integral operator, thereby circumventing the occurrence of memory overflow.

Figure 2: (a) The full architecture of proposed neural operator; (b) example of integral kernel parameterization with Functional Tensor Decompositions on a integral type of four variables.

Method	Single Kernel Evaluation	Network Parameters	Memory Occupancy
GNO	$O(W^{2}L)$	$O(W^{2}L)$	$O(I^{d})$
CP-GNO	$O(dw^{2}lR)$	$O(dw^{2}lR)$	$O(Id)$
TT-GNO	$O(d(w^{2}l+R^{2}))$	$O(dw^{2}lR^{2})$	$O(Id)$
Tucker-GNO	$O(dw^{2}l+R^{d})$	$O(R^{d}+dw^{2}lR)$	$O(Id)$

Table 1: Complexity analysis of different methods.

Model	Darcy $16\times 16$		NS $16\times 16$		NS $128\times 128$
Model	Error	Peak memory	Error	Peak memory	Error	Peak memory
GNO	0.1928	435.43	0.5482	420.61	OOM	OOM
CP-GNO	0.2114	197.02	0.5538	181.38	0.5510	6002.06
TT-GNO	0.2114	197.12	0.5538	181.38	0.5510	6009.94
Tucker-GNO	0.2104	201.68	0.5493	183.32	0.5489	8963.39
FNO	0.1042	93.58	0.0789	92.92	0.0783	1934.74

Table 2: Comprehensive benchmark of relative

L_{2}

errors. Peak memory is reported in MB.

Model	$16\times 16$			$32\times 32$			Parameters	Parameters
Model	Error	Time (s)	Speedup	Error	Time (s)	Speedup	Parameters	Reduction
GNO	0.1872	6542.28	–	0.1723	22602.04	–	95265	–
CP-GNO	0.1975	647.00	10.11 $\times$	0.1815	545.27	41.45 $\times$	74725	21.56%
TT-GNO	0.1975	534.03	12.25 $\times$	0.1815	654.29	34.54 $\times$	74725	21.56%
Tucker-GNO	0.2157	665.17	9.84 $\times$	0.2214	635.18	35.58 $\times$	72709	23.68%

Table 3: Experiment results of relative

L_{2}

errors, training time, and parameters on Darcy flow.

	$16\times 16$			$32\times 32$			$64\times 64$
Model	$r{=}0.08$	$r{=}0.12$	$r{=}0.16$	$r{=}0.08$	$r{=}0.12$	$r{=}0.16$	$r{=}0.08$	$r{=}0.12$	$r{=}0.16$
GNO	420.6	696.3	1511.0	6426.6	13208.9	19733.3	OOM	OOM	OOM
CP-GNO ( $R{=}8$ )	182.7	182.7	182.9	680.3	682.0	683.6	2688.8	2715.9	2753.5
CP-GNO ( $R{=}16$ )	274.7	274.8	275.0	1048.4	1050.0	1051.6	4160.8	4187.4	4225.5
CP-GNO ( $R{=}32$ )	458.8	458.9	459.1	1784.5	1786.1	1787.7	7104.9	7132.0	7168.9
TT-GNO ( $R{=}8$ )	182.7	182.7	182.9	680.3	682.0	683.6	2688.9	2715.9	2753.5

Table 4: Peak GPU memory footprint (MB) of different models under varying resolutions and radii.

Metric	GNO	CP-8	CP-16	CP-32	TT-8	TT-16	TT-32	Tcr-8	Tcr-16	Tcr-32
$16{\times}16\,{\rightarrow}\,32{\times}32$ , $r{=}0.12$	19.0 $\times$	3.7 $\times$	3.8 $\times$	3.9 $\times$	3.7 $\times$	3.8 $\times$	3.9 $\times$	3.7 $\times$	3.8 $\times$	3.9 $\times$
$r{:}\,\,0.08\,{\rightarrow}\,0.16$ , $32{\times}32$	3.1 $\times$	1.0 $\times$	1.0 $\times$	1.0 $\times$	1.0 $\times$	1.0 $\times$	1.0 $\times$	1.0 $\times$	1.0 $\times$	1.0 $\times$
Reduction vs. GNO ( $32\!\times\!32$ , $r{=}0.16$ )	1.0 $\times$	28.9 $\times$	18.8 $\times$	11.0 $\times$	28.9 $\times$	18.8 $\times$	11.0 $\times$	28.7 $\times$	18.6 $\times$	10.9 $\times$

Table 5: Derived memory scaling and efficiency analysis from Table 4 .

Model	$16\times 16$	$32\times 32$
GNO	0.1872	0.1838
CP-GNO	0.1975	0.1956
TT-GNO	0.1975	0.1956
Tucker-GNO	0.2157	0.2214

Table 6:

l_{2}

relative test errors.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Standard GNO Formulation	FTD-GNO Decomposition
CP	${\kappa\left({x,y;\phi}\right)}$	${\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right){A_{r}^{(2)}}\left(y\right)}}$
	${\kappa\left({x,y;\phi}\right){v^{\left(l\right)}}\left(y\right)}$	${\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right){A_{r}^{(2)}}\left(y\right){v^{\left(l\right)}}\left(y\right)}}$
	${\kappa\left({x,y,a\left(y\right);\phi}\right)}$	${\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right){A_{r}^{(2)}}\left(y\right){A_{r}^{(3)}}\left({a\left(y\right)}\right)}}$
	${\kappa\left({x,y,a\left(y\right);\phi}\right){v^{\left(l\right)}}\left(y\right)}$	${\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right){A_{r}^{(2)}}\left(y\right){A_{r}^{(3)}}\left({a\left(y\right)}\right){v^{\left(l\right)}}\left(y\right)}}$
	${\kappa\left({x,y,a\left(x\right),a\left(y\right);\phi}\right)}$	${\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right){A_{r}^{(2)}}\left(y\right){A_{r}^{(3)}}\left({a\left(x\right)}\right){A_{r}^{(4)}}\left({a\left(y\right)}\right)}}$
	${\kappa\left({x,y,a\left(x\right),a\left(y\right);\phi}\right){v^{\left(l\right)}}\left(y\right)}$	${\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right){A_{r}^{(2)}}\left(y\right){A_{r}^{(3)}}\left({a\left(x\right)}\right){A_{r}^{(4)}}\left({a\left(y\right)}\right){v^{\left(l\right)}}\left(y\right)}}$

Appendix

Table 7: Functional tensor decomposition formulas of the integral kernel under CP, TT, and Tucker formats for multi-variable inputs.

Type	Algebraic Reconstruction Formulation of FTD-GNO
CP	$\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right)\bigg({\sum\limits_{y\in N\left(x\right)}{{A_{r}^{(2)}}\left(y\right)}}\bigg)}$
	$\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right)\bigg({\sum\limits_{y\in N\left(x\right)}{{A_{r}^{(2)}}\left(y\right){v^{\left(l\right)}}\left(y\right)}}\bigg)}$
	$\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right)\bigg({\sum\limits_{y\in N\left(x\right)}{{A_{r}^{(2)}}\left(y\right){A_{r}^{(3)}}\left({a\left(y\right)}\right)}}\bigg)}$
	$\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right)\bigg(\sum\limits_{y\in N\left(x\right)}{{A_{r}^{(2)}}\left(y\right){A_{r}^{(3)}}\left({a\left(y\right)}\right){v^{\left(l\right)}}\left(y\right)}\bigg)}$
	$\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right){A_{r}^{(3)}}\left({a\left(x\right)}\right)\bigg({\sum\limits_{y\in N\left(x\right)}{{A_{r}^{(2)}}\left(y\right){A_{r}^{(4)}}\left({a\left(y\right)}\right)}}\bigg)}$
	$\sum\limits_{r=1}^{R}{{A_{r}^{(1)}}\left(x\right){A_{r}^{(3)}}\left({a\left(x\right)}\right)\bigg(\sum\limits_{y\in N\left(x\right)}{{A_{r}^{(2)}}\left(y\right){A_{r}^{(4)}}\left({a\left(y\right)}\right){v^{\left(l\right)}}\left(y\right)}\bigg)}$

Appendix

Table 8: Algebraic reconstruction formulas of the integral algorithm under CP, TT, and Tucker decomposition formats.

N(x)

means the neighhourhood of

x

Model	Parameters	Error	Peak memory
GNO	95265	0.1872	420.6
LR-GNO	74785	0.1946	479.8
CP-GNO	74725	0.1975	182.7
TT-GNO	74725	0.1975	182.7
Tucker-GNO	72709	0.2157	183.7

Appendix

Table 9: Parameter matched comparison on Darcy flow at

16\times 16

and 300 epochs. Peak memory is reported in MB.

Model	Discretization	Epochs	Error	Peak memory	Time (s)
GNO	972 irregular nodes	100	0.2800	263.79	17640.04
CP-GNO	972 irregular nodes	100	0.3021	61.74	517.41
TT-GNO	972 irregular nodes	100	0.3021	61.74	521.04
Tucker-GNO	972 irregular nodes	100	0.3154	63.23	502.32
CP-GNO	972 irregular nodes	300	0.2623	61.74	1542.86
TT-GNO	972 irregular nodes	300	0.2623	61.74	1513.48

Appendix

Table 10: Elasticity on sample-dependent irregular meshes with 972 nodes.

Model	Epochs	Error	Time (s)
GNO	300	0.1872	6542.28
CP-GNO	300	0.1975	647.00
CP-GNO	500	0.1971	1081.49
CP-GNO	700	0.1931	1603.67
CP-GNO	1000	0.1926	2120.37
CP-GNO	2000	0.1902	4521.40

Appendix

Table 11: Extended training study on Darcy flow at

16\times 16

Ablation	$w$	$l$	$R$	Error	Parameters	Peak memory	Time (s)
$w$	16	2	8	0.2269	24293	152.82	205.09
$w$	32	2	8	0.2250	48101	189.09	214.35
$w$	64	2	8	0.2259	108005	269.38	259.47
$l$	64	1	8	0.2114	74725	197.02	232.84
$l$	64	2	8	0.2259	108005	269.38	259.47
$l$	64	3	8	0.2308	141285	334.16	283.20

Appendix

Table 12: CP-GNO sensitivity to factor-network width, depth and rank on Darcy flow.

$R$	CP $L_{2}$ error	TT $L_{2}$ error	Tucker $L_{2}$ error
2	0.2161	0.2161	0.2177
4	0.2171	0.2171	0.2104
8	0.2114	0.2114	0.2257
16	0.2276	0.2276	0.2134

Appendix

Table 13: Rank sensitivity of different decomposition methods.

Model	$(x,y)$	$(x,y,a(y))$	$(x,y,a(x),a(y))$
GNO	420.6	432.1	426.7
CP-GNO	182.7	203.1	223.2
TT-GNO	182.7	217.3	251.6
Tucker-GNO	183.7	211.2	861.5

Appendix

Table 14: Peak GPU memory footprint (MB) versus the number of input variables, tested at grid resolution

16\times 16

, radius

r=0.08

, and decomposition rank

R=8

	$16\times 16$			$32\times 32$			$64\times 64$
Model	$r{=}0.08$	$r{=}0.12$	$r{=}0.16$	$r{=}0.08$	$r{=}0.12$	$r{=}0.16$	$r{=}0.08$	$r{=}0.12$	$r{=}0.16$
GNO	421.13 $\pm$ 0.68	696.80 $\pm$ 0.62	1511.53 $\pm$ 0.68	6427.10 $\pm$ 0.62	13209.40 $\pm$ 0.62	19733.80 $\pm$ 0.62	OOM	OOM	OOM
CP-GNO ( $R{=}8$ )	183.20 $\pm$ 0.62	183.23 $\pm$ 0.68	183.43 $\pm$ 0.68	680.87 $\pm$ 0.67	682.50 $\pm$ 0.62	684.10 $\pm$ 0.62	2689.77 $\pm$ 0.95	2715.33 $\pm$ 0.67	2753.67 $\pm$ 0.47
CP-GNO ( $R{=}16$ )	275.20 $\pm$ 0.62	275.30 $\pm$ 0.62	275.50 $\pm$ 0.62	1048.90 $\pm$ 0.62	1050.53 $\pm$ 0.68	1052.13 $\pm$ 0.68	4161.77 $\pm$ 0.95	4187.80 $\pm$ 0.61	4225.70 $\pm$ 0.53
CP-GNO ( $R{=}32$ )	459.30 $\pm$ 0.62	459.40 $\pm$ 0.62	459.60 $\pm$ 0.62	1785.00 $\pm$ 0.62	1786.63 $\pm$ 0.68	1788.20 $\pm$ 0.62	7104.93 $\pm$ 0.45	7132.07 $\pm$ 0.50	7169.57 $\pm$ 0.76
TT-GNO ( $R{=}8$ )	183.20 $\pm$ 0.62	183.23 $\pm$ 0.68	183.43 $\pm$ 0.68	680.87 $\pm$ 0.67	682.50 $\pm$ 0.62	684.10 $\pm$ 0.62	2689.77 $\pm$ 0.95	2715.33 $\pm$ 0.67	2753.67 $\pm$ 0.47

Appendix

Table 15: Peak GPU memory footprint (MB) of different models under varying resolutions and radii. Results are consistently reported as mean

\pm

standard deviation across independent runs to ensure statistical significance.

Graph Neural Operators

See all 10 papers

May 29, 2026

On Efficient Scaling of GNNs via IO-Aware Layers Implementations

Graph Neural Networks (GNNs) are bottlenecked by sparse, irregular memory access. Popular frameworks such as DGL and PyTorch Geometric support general message passing, but complex layers often materialize edge-wise intermediates, increasing memory traffic and limiting scalability on large graphs. We take an I/O- and arithmetic-intensity--centric view and show that widely used layers fall into three kernel families: SpMM-based convolutions, reduction-based aggregations, and attention-based layers (GATv2/Graph Transformer). For each family, we develop GPU kernels that reduce data movement, improve locality, and remain robust across realistic graphs. We also study graph reordering and find that its impact depends on the kernel mapping: it benefits neighbor-parallel (gather-dominated) kernels more consistently than feature-parallel designs. Empirically, our fused attention kernels reach up to

\textbf{3.9}\times

speedup for Graph Transformer (median

\textbf{1.6}\times

), with Tensor Core (block-sparse) variants up to

\textbf{7.3}\times

on locally dense graphs; for GATv2 we reach up to

\textbf{8.5}\times

speedup (median

\textbf{2.0}\times

) while reducing peak memory by up to

\textbf{76}\times

(median

\textbf{6}\times

). Our degree-aware reduction kernels achieve up to

\textbf{10}\times

speedup (median

\textbf{2.6}\times

). For SpMM-based layers, properly cached cuSPARSE achieves up to

\textbf{8}\times

speedup over DGL and outperforms evaluated custom baselines in the majority of evaluations. We release our implementations as drop-in replacements to support reproducible, hardware-aware GNN acceleration.

Daria Fomina, Daniil Krasylnikov, Alexey Boykov +3

1Yandex · 2HSE University · 3ITMO University

Graph Attention Networks GPU Kernel Optimization GPU Acceleration High-Performance Computing GPU Kernel Generation Graph Transformer

Ask PDF

Aug 30, 2026

Joint Spatiotemporal Spectral Neural Operators for Learning PDEs on Irregular Domains

Learning solution operators for partial differential equations (PDEs) on irregular and geometry-dependent domains remains a central challenge in scientific machine learning. While spectral methods provide strong inductive biases for modeling global interactions, they are typically limited to regular domains, and existing neural approaches often require domain warping, interpolation, or costly geometric embeddings. We introduce the Graph Spectral Neural Operator (GSNO), a neural operator that combines spatial graph spectral decompositions with temporal Fourier transforms through a unified space--time spectral kernel. This formulation enables globally coherent operator learning on non-Cartesian discretizations without domain warping or autoregressive rollouts. By replacing learned geometric embeddings with a graph Laplacian spectral basis, GSNO provides geometry-aware spectral learning with low parameter complexity. Across steady and unsteady PDE benchmarks on irregular and geometry-dependent domains, GSNO achieves strong accuracy with reduced runtime and parameter counts, while demonstrating robust zero-shot generalization across mesh resolutions and geometry families.

Abdolmehdi Behroozi, Chaopeng Shen

Department of Civil and Environmental Engineering, Penn State University, University Park, PA 16802, USA

PDE Surrogate Modeling PDE Operator Learning Spectral Methods Neural Operators

Ask PDF

Jul 6, 2026

FAST: A Holistic Framework for Optimizing Memory-I/O, Computation, and Sampling in Temporal GNN Training

Temporal Graph Neural Networks (TGNNs) are widely used for learning from dynamic graphs in applications such as recommendation, social network analysis, and traffic forecasting. However, scaling TGNN training to large dynamic graphs remains challenging due to three intertwined bottlenecks: memory I/O, irregular computation, and temporal neighbor sampling. Existing systems often optimize these stages in isolation, leaving substantial performance headroom on the table. We present FAST, a holistic framework that accelerates end-to-end TGNN training by jointly optimizing sampling, memory I/O, and computation. FAST introduces SlimCache, which exploits within-batch compression and cross-batch caching to reduce host-device data movement under limited GPU memory budgets. It further designs thread-efficient graph operators tailored to sparse temporal subgraphs, improving GPU cache locality and reducing the latency of aggregation and edge softmax. In addition, FAST employs a topology-aware sampling strategy that improves CPU cache locality and accelerates temporal neighbor sampling. Extensive experiments on real-world large dynamic graphs show that FAST achieves an average of 2.1x (up to 4.7x) speedup over state-of-the-art systems without sacrificing model accuracy.

Yushu Cai, Qingrui Zhu, Lei Liu +3

Xidian University Guangzhou, Guangdong, China · Hunan University Changsha, Hunan, China

Temporal GNNs Graph Neural Networks GPU Kernel Optimization

Ask PDF

Abstract

Graph Neural Operators

Explore similar work

On Efficient Scaling of GNNs via IO-Aware Layers Implementations

Joint Spatiotemporal Spectral Neural Operators for Learning PDEs on Irregular Domains

FAST: A Holistic Framework for Optimizing Memory-I/O, Computation, and Sampling in Temporal GNN Training