Smoothness Errors in Dynamics Models and How to Avoid Them

Edward Berman; Luisa Li; Jung Yeon Park; Robin Walters

cs.LGFeb 5, 2026

Smoothness Errors in Dynamics Models and How to Avoid Them

Authors: Edward Berman, Luisa Li, Jung Yeon Park, Robin Walters

Organizations: Geometric Learning Lab, Northeastern University.

Abstract

Modern neural networks have shown promise for solving partial differential equations over surfaces, often by discretizing the surface as a mesh and learning with a mesh-aware graph neural network. However, graph neural networks suffer from oversmoothing, where a node's features become increasingly similar to those of its neighbors. Unitary graph convolutions, which are mathematically constrained to preserve smoothness, have been proposed to address this issue. Despite this, in many physical systems, such as diffusion processes, smoothness naturally increases and unitarity may be overconstraining. In this paper, we systematically study the smoothing effects of different GNNs for dynamics modeling and prove that unitary convolutions hurt performance for such tasks. We propose relaxed unitary convolutions that balance smoothness preservation with the natural smoothing required for physical systems. We also generalize unitary and relaxed unitary convolutions from graphs to meshes. In experiments on PDEs such as the heat and wave equations over complex meshes and on weather forecasting, we find that our method outperforms several strong baselines, including mesh-aware transformers and equivariant neural networks.

Figures & tables

Figure 1: Qualitative comparison of autoregressive model predictions for the heat equation on the armadillo mesh at timestep $T=190$ . Our R-UniMesh model remains faithful to the ground truth during each step of the rollout, whereas the EMAN model over smooths and the Hermes model under smooths. A more complete comparison over several timesteps is in Section C.4 , Table 3 .

Figure 2: Top: After zero padding, individual unitary blocks are stacked and the output is fed into an unconstrained smoothness-breaking component. Bottom: Each block uses Taylor truncated unitary convolution.

Model	MSE $(\downarrow)$	MRE $(\downarrow)$
GCN	$1.08\cdot 10^{-2}$	$5.99\cdot 10^{-2}$
Lie Uni	$0.14\cdot 10^{-2}$	$8.86\cdot 10^{-2}$
R-UniGraph (Ours)	$\mathbf{0.11\cdot 10^{-2}}$	$\mathbf{2.07\cdot 10^{-2}}$

Table 1: MSE and mean Rayleigh quotient error (MRE) of GCN, a Lie unitary convolution network, and R-UniGraph . The best run for each method out of

5

runs is shown. The best performance is bold.

Figure 3: Left: The dotted lines indicate the mean Rayleigh quotient for input heat graphs at $T=3$ and target graphs at $T=4$ . We also show the mean Rayleigh quotient for the best performing GCN, R-UniGraph , and Lie unitary models. R-UniGraph is best at capturing the true smoothness. Right: Validation MSE of the same three models. R-UniGraph has the best performance. Results for the full set of runs are provided in Section B.3 .

Heat $(\alpha=1)$
	Convolutional			Attentional		Message Passing
Metric	GCN	GemCNN	R-UniMesh (Ours)	EMAN	Transformer	MPNN	EGNN	Hermes
NRMSE $(\downarrow)$	–	–	$\mathbf{51.9\pm 3.6}$	$73.50\pm 3.8$	$92.5\pm 5.6$	$99.45\pm 4.8$	$344.25\pm 110.5$	$\underline{73.02\pm 4.7}$
SMAPE $(\downarrow)$	–	$375.4\pm 0.53$	$\mathbf{79.7\pm 5.6}$	$110.9\pm 13.3$	$213.9\pm 2.7$	$223.6\pm 1.5$	$319.33\pm 7.59$	$\underline{107.6\pm 7.4}$
RE $(\downarrow)$	–	$52.21\pm 9.4$	$\mathbf{9.1\pm 7.4}$	$\underline{14.2\pm 1.4}$	$46.0\pm 3.7$	$76.06\pm 3.6$	$81.5\pm 8.77$	$39.76\pm 4.7$
Wave $(c=1)$

Table 2: NRMSE, SMAPE, and RE averaged over all rollouts on all test meshes for the heat, wave, and Cahn-Hilliard equations. The best values are in bold and second best are underlined. Errors and standard deviations are reported over all test meshes and initializations. Cells with a dash (–) indicate models that do not converge for a given metric. Errors are scaled by

\times 196

, the number of rollout timesteps. R-UniMesh is competitive across all tasks and excels at solving the heat equation on unseen meshes. Models are grouped together by flavor: convolutional, attention, or message passing ( Bronstein et al., 2021 ) .

Figure 4: The Rayleigh quotient for each timestep on an unseen mesh for Hermes, EMAN, and R-UniMesh models. The R-UniMesh is the best at capturing the true smoothness for heat.

Figure 5: RMSE and ACC as a function of lead time for all models temperature prediction. R-UniMesh has a competitive RMSE, especially at early lead time. R-UniMesh also maintains viability for lead times of roughly $2$ days according to the ECMWF baseline. Exact values recorded in Table 13 ( Section D.4 ).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Illustration of an intrinsic Delaunay edge flip performed by the Robust Laplacian edge rewiring algorithm. This figure is a reproduced version of Figure 7, Sharp and Crane (2020) .

Panel of Figure 7 — Figure 7: Sample heat diffusion process on a grid discretized as a graph. Node neighbors are the nodes that sit adjacent in the grid.

Figure 8: KL divergence between distribution of Rayleigh quotients before and after applying the model. Results are averaged over 10 runs.

Panel of Figure 9 — Figure 9: Validation MSE and mean Rayleigh quotient for $5$ different Lie Unitary Convolution runs. MSE tends to increase as $\mathbf{T}_{\rm max}$ increases. $\mathbf{T}_{\rm max}=3$ offers the best balance between accuracy under MSE loss and smoothness errors.

Panel of Figure 10 — Figure 10: Top: Validation MSE for an ensemble of $5$ runs for a GCN (left), R-UniGraph (middle), and a Lie unitary convolution network (right) at timestep $t=3$ . R-UniGraph significantly outperforms the GCN and also outperforms the Lie unitary network. Bottom: The average Rayleigh quotient over all graphs for an ensemble of $5$ runs for the same models at timestep $t=3$ . The GCN is under constrained and biased towards oversmoothing at initialization. R-UniGraph is able to roughly match the true smoothness of the labeled graphs. The Lie unitary network is overconstrained and can not model the Rayleigh quotient of the labels because it is forced to preserve the Rayleigh quotient of the input graphs.

Panel of Table 3 — Table 3: Qualitative comparison of model performance for the heat equation on the armadillo mesh. Our R-UniMesh model remains faithful to the ground truth during each step of the rollout, whereas the EMAN model over smooths and the Hermes model under smooths.

Figure 11: Smoothness for a Hermes model as measured by the $2-$ point correlation function. The plot indicates undersmoothing in each radial bin.

Heat ( $\alpha=1$ )
Model	$\text{err}_{\rm smooth}$ $(\downarrow)$
GemCNN	–
EMAN	$\mathbf{4.04\times 10^{-3}}$
Hermes	$9.71\times 10^{-3}$
Wave ( $c=1$ )
Model	$\text{err}_{\rm smooth}$ $(\downarrow)$

Appendix

Table 4:

\text{err}_{\rm smooth}

for Gauge Equivariant models on the MeshPDE datasets. Dashes (–) indicate non-convergence. Best performing model is indicated with bold text .

Heat $(\alpha=1)$
Metric	R-UniMesh	MeshGraphNet
NRMSE $(\downarrow)$	$\mathbf{51.9\pm 3.6}$	$108.1\pm 3.2$
SMAPE $(\downarrow)$	$\mathbf{79.7\pm 5.6}$	$144.1\pm 6.3$
RE $(\downarrow)$	$\mathbf{9.1\pm 7.4}$	$53.4\pm 1.6$
Wave $(c=1)$
NRMSE $(\downarrow)$	$\mathbf{236.5\pm 6.4}$	$253.6\pm 28.2$

Appendix

Table 5: NRMSE, SMAPE, and RE averaged over all rollouts on all test meshes for the heat, wave, and Cahn-Hilliard equations. The best values are in bold. R-UniMesh outperforms MeshGraphNet on heat and is competitive on the other PDEs.

Heat
Metric	Unitary GCN ( $T=10$ )	Unitary GCN ( $T=3$ )	Unconstrained GCN
NRMSE $(\downarrow)$	$\mathbf{51.9\pm 3.6}$	$70.1\pm 6.3$	$416.9\pm 14.3$
SMAPE $(\downarrow)$	$79.7\pm 5.6$	$\mathbf{76.7\pm 10.4}$	$248.8\pm 6.6$
RE $(\downarrow)$	$9.1\pm 7.4$	$\mathbf{9.0\pm 0.6}$	$31.5\pm 8.6$
Wave
NRMSE $(\downarrow)$	$236.5\pm 6.4$	$242.2\pm 4.1$	$\mathbf{196.0\pm 0.0}$

Appendix

Table 6: Comparison for the smoothness-preserving component type on NRMSE, SMAPE, and RE averaged over all rollouts on all test meshes for the heat, wave, and Cahn-Hilliard equations. The best values are in bold.

Heat
Metric	Zero Pad	GCN
NRMSE $(\downarrow)$	$\mathbf{51.9\pm 3.6}$	$4{,}689.6\pm 232.0$
SMAPE $(\downarrow)$	$\mathbf{79.7\pm 5.6}$	$367.7\pm 0.6$
RE $(\downarrow)$	$\mathbf{9.1\pm 7.4}$	$153.9\pm 13.4$
Wave
NRMSE $(\downarrow)$	$\mathbf{236.5\pm 6.4}$	$12{,}532.0\pm 242.2$

Appendix

Table 7: Comparison of Zero Pad and GCN mapping strategies for increasing the latent dimension on NRMSE, SMAPE, and RE averaged over all rollouts on all test meshes for the heat, wave, and Cahn-Hilliard equations. The best values are in bold.

Heat
Metric	GroupSort	ReLU
NRMSE $(\downarrow)$	$\mathbf{51.9\pm 3.6}$	$87.7\pm 10.5$
SMAPE $(\downarrow)$	$\mathbf{79.7\pm 5.6}$	$82.9\pm 3.6$
RE $(\downarrow)$	$9.1\pm 7.4$	$\mathbf{9.0\pm 10.5}$
Wave
NRMSE $(\downarrow)$	$\mathbf{236.5\pm 6.4}$	$825.7\pm 12.3$

Appendix

Table 8: Comparison of GroupSort and ReLU activation functions on NRMSE, SMAPE, and RE averaged over all rollouts on all test meshes for the heat, wave, and Cahn-Hilliard equations. The best values are in bold.

Heat
Metric	MLP (Depth = 1)	MLP (Depth = 3)
NRMSE $(\downarrow)$	$99.9\pm 13.1$	$\mathbf{51.9\pm 3.6}$
SMAPE $(\downarrow)$	$181.6\pm 4.0$	$\mathbf{79.7\pm 5.6}$
RE $(\downarrow)$	$21.2\pm 7.1$	$\mathbf{9.1\pm 7.4}$
Wave
NRMSE $(\downarrow)$	$-$	$236.5\pm 6.4$

Appendix

Table 9: Effect of MLP depth on NRMSE, SMAPE, and RE averaged over all rollouts on all test meshes for the heat, wave, and Cahn-Hilliard equations. The best values are in bold.

Heat
Metric	MLP (Width = 64)	MLP (Width = 128)
NRMSE $(\downarrow)$	$\mathbf{51.9\pm 3.6}$	$88.2\pm 6.8$
SMAPE $(\downarrow)$	$\mathbf{79.7\pm 5.6}$	$136.3\pm 4.7$
RE $(\downarrow)$	$\mathbf{9.1\pm 7.4}$	$10.7\pm 2.3$
Wave
NRMSE $(\downarrow)$	$\mathbf{236.5\pm 6.4}$	$1{,}433.0\pm 35.0$

Appendix

Table 10: Effect of MLP width on NRMSE, SMAPE, and RE averaged over all rollouts on all test meshes for the heat, wave, and Cahn-Hilliard equations. The best values are in bold.

Panel of Table 11 — Table 11: Sample initializations for the heat and wave equations on the PyVista meshes.

Panel of Table 12 — Table 12: Sample initializations for the Cahn-Hilliard equation on the PyVista meshes.

Figure 12: RMSE and ACC as a function of lead time for all models geopotential prediction. R-UniMesh has a competitive RMSE, especially at early lead time. R-UniMesh also maintains viability for lead times of roughly $2$ days according to the ECMWF baseline.

Model	Metric	6h	12h	18h	24h	30h	36h	42h	48h	240h
EGNN	Z500 ACC	0.909	0.946	0.948	0.912	0.835	0.747	0.639	0.538	-0.134
	Z500 RMSE	347.83	266.96	261.09	340.43	467.68	581.20	706.01	812.70	3769.31
	T850 ACC	0.840	0.923	0.894	0.817	0.725	0.637	0.541	0.456	0.093
	T850 RMSE	1.94	1.32	1.54	2.02	2.46	2.81	3.17	3.47	6.71
EMAN	Z500 ACC	0.114	0.125	0.121	0.123	0.120	0.122	0.123	0.123	0.127
	Z500 RMSE	39530.01	52028.53	65037.68	89210.70	103179.47	119529.81	139314.85	156786.23	4898180.46

Appendix

Table 13: Weather Forecasting Results, ACC and RMSE at Different Lead Times for Temperature and Geopotential.

Model	RE Temperature $(\downarrow)$	RE Geopotential $(\downarrow)$
GCN	$6.8\cdot 10^{-2}\pm 1.7\cdot 10^{-2}$	$1.2\cdot 10^{-3}\pm 1.0\cdot 10^{-3}$
MPNN	$3.0\cdot 10^{-3}\pm 2.7\cdot 10^{-3}$	$1.3\cdot 10^{-3}\pm 8.5\cdot 10^{-4}$
Hermes	$3.1\cdot 10^{-1}\pm 7.5\cdot 10^{-1}$	$1.0\cdot 10^{-1}\pm 3.6\cdot 10^{-1}$
GemCNN	$\mathbf{8.9\cdot 10^{-4}\pm 7.0\cdot 10^{-4}}$	$9.4\cdot 10^{-4}\pm 6.1\cdot 10^{-4}$
EMAN	$7.7\cdot 10^{0}\pm 5.2\cdot 10^{0}$	$4.1\pm 2.6\cdot 10^{-2}$
EGNN	$1.1\cdot 10^{-3}\pm 7.3\cdot 10^{-4}$	$\mathbf{8.3\cdot 10^{-4}\pm 1.0\cdot 10^{-3}}$

Appendix

Table 14: Rayleigh error for all models for all initializations on WB

2

temperature and geopotential. Best performing model is indicated with bold text . Errors scaled up by

\times 40

Graph Neural Networks

Graph Neural Networks learn from graph-structured data by combining information across connected nodes, edges, and larger structures.

See all 735 papers

Graph Unitary Message Passing
Mar 17, 2024Haiquan Qiu, Quanming YaoGraph Neural Networks Message Passing Neural Networks
Department of Electrical Engineering, Tsinghua University · 2Beijing National Research Center for Information Science and Technology · 3State Key laboratory of Space Network and Communications
Persistent Gaussian Perturbations Prevent Oversmoothing in Recurrent Graph Neural Networks
Jul 30, 2026Mostafa Haghir ChehreghaniGraph Neural Networks Stochastic Linear Dynamical Systems
Department of Computer Engineering Amirkabir University of Technology (Tehran Polytechnic) Tehran, Iran
Graph Navier Stokes Networks
May 20, 2026Zexing Zhao, Guangsi Shi, Yu Gong +4Graph Neural Networks GNN Oversmoothing
Northwest A&F University Yangling, Shaanxi, China · Corporate Research Center, Midea Group Shanghai, China · Bosch Shanghai, China

Mar 17, 2024J/K move · Enter open · S save

Graph Unitary Message Passing

Haiquan Qiu, Quanming Yao

Department of Electrical Engineering, Tsinghua University · 2Beijing National Research Center for Information Science and Technology · 3State Key laboratory of Space Network and Communications

Graph Neural Networks Message Passing Neural Networks Over-Smoothing in GNNs

Open paper AskPDF arXiv

Unitarity is a useful principle for stabilizing deep neural networks, but in graph neural networks (GNNs) instability is induced not only by learnable parameters but also by the graph propagation operator. Motivated by this distinction, we propose Graph Unitary Message Passing (GUMP), a message-passing framework that uses a unitary propagation operator on a transformed graph to avoid graph-induced exponential decay under repeated propagation. GUMP combines (i) a graph transformation that maps an input graph to an Eulerian line-graph construction admitting unitary adjacency matrices, and (ii) a practical unitary projection procedure based on Newton-Schulz iteration. Theoretical analysis clarifies that, under standard analysis assumptions, unitary propagation keeps the graph-propagation term depth-stable, while vanilla normalized propagation exhibits exponential decay in its non-trivial spectral components. Across synthetic long-range tasks, TUDataset benchmarks, and LRGB datasets, GUMP improves over vanilla message passing and achieves competitive or superior performance against strong baselines. Code is available at https://github.com/ucker/gump_code.

Abstract

Graph Neural Networks

Explore similar work

Graph Unitary Message Passing

Persistent Gaussian Perturbations Prevent Oversmoothing in Recurrent Graph Neural Networks

Graph Navier Stokes Networks