cs.CEOct 8, 2026

Mesh-Native Tokenisation: Retrofitting Foundation Models to Unstructured Data

Authors: Georg Maierhofer, Miles Cranmer, Payel Mukhopadhyay

Organizations: University of Cambridge

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Physics foundation models are an increasingly popular approach for building versatile emulators for spatiotemporal systems. They require tokenisation layers that lift physical field data into a latent representation suitable for dynamic processing. A typical choice for this layer is a convolutional encoder, which lifts field information in a locality-aware way and can be trained very efficiently. However, this choice cannot be applied to a large class of scientific computing problems where data are represented on unstructured meshes, for example simulations on domains with curved boundaries. Such data typically require graph neural network architectures, which are more expensive to train and, in the form currently used, encode information differently from their CNN counterparts, thus limiting the portability of physics emulators between these data regimes. In this work we introduce Mesh-Native Convolution (MNC), a locality-aware tokenisation layer that encodes unstructured mesh-based data in a way that is analogous to CNNs. This approach leads to models that can be trained once on mesh data and then deployed to arbitrary unstructured meshes, even if these are irregular, with minimal retraining. We show that this tokenisation approach can be used to retrofit grid-pretrained foundation models efficiently on unstructured meshes and provide theoretical results to confirm the consistency of this approach with classical CNN encoders. We demonstrate the favourable performance of this retrofitting approach, which in many cases reaches acceptable accuracy even in zero-shot deployment, and outperforms existing baselines after only a few epochs of fine-tuning, thus providing an efficient bridge between the two data regimes.

Figures & tables

Figure 1: (a) MNC loads grid-pretrained weights directly onto a mesh-compatible GNN architecture, for consistent retrofitting of pretrained physics emulators. (b) Zero-shot shear-flow single-step prediction after grid pretraining, including MNC (ours), GINO ( Li et al., 2023c ) and LNO ( Wang and Wang, 2024 ) . Values are normalised MSE for the displayed trajectories.
Figure 2: Schematic overview of the MNC layer. MNC loads a pretrained CNN’s weights into a continuous spline and evaluates the same kernel at physical offsets on an unstructured mesh.
Figure 4: Discretisation invariance of MNC. (a) Refining kernel spacing and mesh together (Proposition 2 ). In the limit MNC corresponds to the continuous kernel κ\kappa and the mesh enters only through its sampling density ρ\rho . (b) Two triangulations of the same vertices. The GCN neighbourhood of the origin is shown in blue. On the constant field the degree-normalised GCN returns 11 and 1.1521.152 at the origin ( 9 ), whereas the MNC window (dashed) yields identical outputs.
 Overtone+MNCGINOLNOTransolver++MeshGraphNets
Next-step VRMSE     
TRL2D26.25±0.4526.25\pm 0.4529.55±0.1029.55\pm 0.1048.78±3.6748.78\pm 3.6740.46±0.2840.46\pm 0.2845.07±0.9945.07\pm 0.99
Active matter4.88±0.124.88\pm 0.1211.54±0.2011.54\pm 0.208.00±0.458.00\pm 0.4512.44±0.0712.44\pm 0.0719.58±0.1819.58\pm 0.18
Shear flow1.42±0.031.42\pm 0.031.82±0.001.82\pm 0.003.85±0.043.85\pm 0.041.04±0.221.04\pm 0.222.55±0.022.55\pm 0.02
10-step rollout VRMSE     
TRL2D52.27±0.9852.27\pm 0.9863.64±0.8563.64\pm 0.8585.03±7.4885.03\pm 7.4879.93±1.9079.93\pm 1.9085.04±2.6285.04\pm 2.62
Table 1: Zero-shot transfer onto the irregular mesh (Experiment 1): next-step and ten-step rollout VRMSE, in units of 10−210^{-2} . Mean ±\pm standard deviation over three pretraining seeds.
Figure 5: Zero-shot one-step density error on the irregular TRL2D mesh. Numbers are the window’s density VRMSE; predicting no change (persistence) scores 0.308.
Figure 6: Test next-step VRMSE (mean) on the irregular TRL2D mesh against fine-tuning epoch. Grid-pretrained models are marked in solid lines and randomly initialised ones in dashed lines.
  Overtone+MNCGINOLNOTransolver++MeshGraphNets
Next-step VRMSE     
TRL2Dpretrained22.10±0.0422.10\pm 0.0429.40±0.0829.40\pm 0.0840.89±0.1240.89\pm 0.1240.76±0.0740.76\pm 0.0733.91±0.4333.91\pm 0.43
 scratch22.77±0.2022.77\pm 0.2029.78±0.2729.78\pm 0.2753.39±0.1153.39\pm 0.1139.78±0.1339.78\pm 0.1336.84±0.4636.84\pm 0.46
Active matterpretrained3.30±0.063.30\pm 0.0610.71±0.0410.71\pm 0.048.12±0.058.12\pm 0.0512.81±0.0412.81\pm 0.0411.20±0.2611.20\pm 0.26
 scratch4.36±0.084.36\pm 0.0812.84±0.2012.84\pm 0.2015.86±1.2715.86\pm 1.2714.09±0.0814.09\pm 0.0813.35±0.0313.35\pm 0.03
Shear flowpretrained0.43±0.010.43\pm 0.011.73±0.021.73\pm 0.023.56±0.123.56\pm 0.120.98±0.050.98\pm 0.051.78±0.051.78\pm 0.05
Table 2: Fine-tuning on the irregular mesh: next-step and ten-step rollout VRMSE, in units of 10−210^{-2} .
 Next-step VRMSE10-step rollout VRMSE
Walrus+MNC, pre-trained, fine-tuned0.15±0.010.15\pm 0.011.02±0.001.02\pm 0.00
Walrus+MNC, pre-trained, zero-shot0.410.413.303.30
Walrus+MNC, random init, fine-tuned0.44±0.010.44\pm 0.012.47±0.042.47\pm 0.04
MeshGraphNets, trained on cylinder flow0.80±0.040.80\pm 0.048.26±0.038.26\pm 0.03
Transolver++, trained on cylinder flow0.18±0.010.18\pm 0.012.05±0.162.05\pm 0.16
GINO, trained on cylinder flow0.70±0.010.70\pm 0.013.80±0.213.80\pm 0.21
Table 3: Cylinder flow dataset: next-step and ten-step rollout VRMSE, in units of 10−210^{-2} . All training and fine-tuning runs were given the same compute budget of 7.5h on a single NVIDIA GH200 GPU.
Figure 7: Error in next-step xx -velocity prediction on cylinder flow. Numbers are the window’s xx -velocity VRMSE; predicting no change (persistence) scores 0.0314.
Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Coordinate rotary position embedding for mesh tokens: only the source of pos\mathrm{pos} changes. (a) On the grid, pos=linspace(−1,1,⋅)\mathrm{pos}=\mathrm{linspace}(-1,1,\cdot) along each axis. (b) After retrofitting, pos\mathrm{pos} is the anchor coordinate, rescaled by the outermost anchors ( 6 ). The window covered by the hole emits no token. On a lattice this recovers the angles of (a) exactly. (c) The stock axial embedding would place the flat list of anchors along one axis, separating vertical neighbours (ringed) and collapsing the second axis.
Figure 9: Meshes AA and BB near the origin. Each vertex in the origin’s GCN neighbourhood (blue) is labelled with its contribution to GΓ(1)0G_{\Gamma}(\mathbf{1})_{0} , the origin’s own label being its self-loop.
Figure 10: TRL2D: the regular mesh twin with the zoom region marked in red and the zoomed mesh below it (left), and log⁡10\log_{10} density at two times along a tcool=0.18t_{\mathrm{cool}}=0.18 trajectory. The Well mesh twins are triangulations of the original grids, so grid and mesh tokenisers see exactly the same samples.
Figure 11: TRL2D on the Experiment 1 irregular mesh: the Gmsh Delaunay triangulation ( 49,00849{,}008 nodes, 146,135146{,}135 edges, node count matched to the grid) with the zoom region marked in red (left), and the same tcool=0.18t_{\mathrm{cool}}=0.18 density snapshots as Figure 10 , bilinearly resampled onto the mesh nodes which are exactly the samples the zero-shot-evaluated models see.
Figure 12: Shear flow: the mesh twin with zoom (left), and the tracer at two times along a Re=104Re=10^{4} , Sc=1Sc=1 trajectory that develops Kelvin–Helmholtz billows.
Figure 13: Shear flow on the Experiment 1 irregular mesh ( 131,733131{,}733 nodes, 393,936393{,}936 edges, node count matched to the grid): the same Re=104Re=10^{4} , Sc=1Sc=1 tracer snapshots as Figure 12 , bilinearly resampled onto the mesh nodes (trajectory from the official validation file; Experiment 1 itself uses the trajectory-level resplit of the training files).
Figure 14: Active matter on the Experiment 1 irregular mesh ( 65,63265{,}632 nodes, 196,005196{,}005 edges, node count matched to the grid): the mesh with the zoom region marked in red (left), and the orientation-tensor component DxxD_{xx} at two times along a ζ=9\zeta=9 , α=−3\alpha=-3 test trajectory, bilinearly resampled onto the mesh nodes. The concentration is nearly uniform ( 1±5⋅10−41\pm 5\cdot 10^{-4} ), so a tensor channel is shown.
Figure 15: Cylinder flow: one of the per-trajectory meshes, with inflow, outflow and wall nodes marked in the zoom (left), and velocity magnitude at two times.
DatasetDomain / resolutionFieldsTrajectoriesStepsNodesEdges
TRL2D128×384128\times 384 grid, [−0.5,0.5]×[−1,2][-0.5,0.5]\times[-1,2]ρ, p, vx, vy\rho,\,p,\,v_{x},\,v_{y}72/9/910149,15249{,}152146,433146{,}433
Shear flow256×512256\times 512 grid, [0,1]2[0,1]^{2} periodictracer, p, vx, vyp,\,v_{x},\,v_{y}896/112/112200131,072131{,}072391,681391{,}681
Active matter256×256256\times 256 grid, [0,10]2[0,10]^{2} periodicc, vx, vy, D, Ec,\,v_{x},\,v_{y},\,D,\,E (11 ch.)175/24/268165,63265{,}632196,005196{,}005
Cylinder flowunstructured 1.6×0.411.6\times 0.41 channelp, vx, vyp,\,v_{x},\,v_{y}1000/100/1006001,7321{,}732 - 2,0592{,}059≈5,400\approx 5{,}400
Appendix
Table 4: Datasets summary. Trajectories are given as train/validation/test of the official splits; nodes and edges refer to the mesh representation (undirected edges): the lattice twin for TRL2D and shear flow, and the Experiment 1 irregular mesh for active matter, which has no lattice twin.
ModelImplementationConfigurationParameters
MNC retrofit ( Mukhopadhyay et al., 2026 )Overtone-family patch modulator12 space-time blocks, width 384, 6 heads; patch 16 (two 4×44\times 4 , stride-4 stages)26,301,955 (same)
GINO ( Li et al., 2023c )neuraloperator 2.0.0, GINO_Poisson2d referenceFNO width 64, 4 layers, 20×2020\times 20 modes; latent lattice of 64 cells on the short axis; GNO radius 1.5 cells5,099,036 (5,122,381)
LNO ( Wang and Wang, 2024 )official codewidth 640, 8 blocks, 512 latent modes, 8 heads29,901,316 (29,932,683)
Transolver++ ( Luo et al., 2025 )official code8 layers, width 256, 8 heads, 32 slices, MLP ratio 12,303,308 (2,326,611)
MeshGraphNets ( Pfaff et al., 2021 )PyTorch re-implementation15 message-passing steps, latent 128; 3-hidden-layer MLPs with LayerNorm2,879,748 (2,886,027)
Appendix
Table 5: Architectures of Experiments 1 and 2. Parameter counts are for the four-channel datasets (TRL2D, shear flow) and, in brackets, for the eleven-channel active-matter dataset. MNC and the grid source have identical counts, since the transplant replaces operations, not parameters.
SettingValue
OptimiserAdamW, β=(0.9,0.95)\beta=(0.9,0.95) , weight decay 10−410^{-4} , gradient-norm clip 5
Learning rate10−410^{-4} for every architecture and stage
Grid pretraining50,000 updates at batch 16; 500 warm-up updates, cosine decay to 5%5\%
Mesh training (retrofit and scratch)20,000 updates at batch 8; 100 warm-up updates, constant rate
Seedsthree per mesh-training arm; retrofits start from the seed-0 source
Context / targetsix frames / field increment
Appendix
Table 6: Training settings of Experiments 1 and 2, shared by all architectures. One “epoch” in Figure 6 is 100 mesh-training updates.
Figure 16: One-step density error (prediction −- truth) on the irregular TRL2D mesh, test trajectory 7, frame 79: grid-pretrained models zero-shot and after fine-tuning, and the same architectures trained from scratch, at initialisation and after 20k updates. Numbers are the density VRMSE of this window. The fine-tuned MeshGraphNets panel was not exported for this window.
TokeniserTRL2DShear flowCylinder flow
CNN (grid)(1.62±0.16)×10−3(1.62\pm 0.16)\times 10^{-3}(1.18±0.03)×10−3(1.18\pm 0.03)\times 10^{-3}—
MNC (ours)(1.68±0.06)×10−3(1.68\pm 0.06)\times 10^{-3}(1.15±0.08)×10−3(1.15\pm 0.08)\times 10^{-3}(2.58±0.01)×10−2(2.58\pm 0.01)\times 10^{-2}
GINO(1.79±0.01)×10−2(1.79\pm 0.01)\times 10^{-2}(5.25±0.01)×10−2(5.25\pm 0.01)\times 10^{-2}(2.29±0.10)×10−1(2.29\pm 0.10)\times 10^{-1}
LNO (32 modes)(9.93±0.02)×10−2(9.93\pm 0.02)\times 10^{-2}(7.37±0.74)×10−2(7.37\pm 0.74)\times 10^{-2}(9.12±0.64)×10−2(9.12\pm 0.64)\times 10^{-2}
Appendix
Table 7: Tokeniser reconstruction from scratch after 100 epochs: test normalised MSE, mean ±\pm standard deviation over three seeds. The Well grid twins use identical samples; cylinder flow uses native meshes. LNO uses 32 modes; latent-parity results are in Appendix F.2 .
Figure 17: Shear-flow autoencoding after a change of discretisation. Solid curves: CNN →\to MNC, GINO →\to GINO and LNO →\to LNO transfer. Dashed curves: the corresponding models trained from scratch. Bands show variation across three seeds. Epoch zero is before any mesh training; source pretraining is additional to the horizontal-axis budget.
TokeniserTRL2DShear flow
LNO (32 modes)9.93×10−29.93\times 10^{-2}7.37×10−27.37\times 10^{-2}
LNO (latent parity)9.84×10−29.84\times 10^{-2}4.17×10−24.17\times 10^{-2}
MNC (ours)1.68×10−31.68\times 10^{-3}1.15×10−31.15\times 10^{-3}
Appendix
Table 8: LNO mode-count ablation: test normalised MSE after 100 epochs of direct training (run means). The 32-mode and MNC rows repeat Table 7 for reference.
TokeniserTRL2DShear flowCylinder flow
CNN (grid)0.55 ±\pm 0.040.61 ±\pm 0.03—
MNC (ours)3.28 ±\pm 0.193.46 ±\pm 0.105.80 ±\pm 0.18
GINO1.67 ±\pm 0.023.89 ±\pm 0.091.32 ±\pm 0.05
LNO (32 modes)1.85 ±\pm 0.004.45 ±\pm 0.010.81 ±\pm 0.02
Appendix
Table 9: Mean training time per epoch in seconds for the reconstruction study (each model’s summed per-step time on the same single NVIDIA GH200; mean ±\pm standard deviation over three seeds).
Figure 18: The shear-flow target meshes used for tokeniser transfer. Source-grid fields are resampled onto their nodes.
Figure 19: Companion to Figure 17 on the uniform triangular lattice. Solid curves use grid-pretrained weights; dashed curves start from random initialisation. Bands show variation across three seeds. Pretraining is not included in the horizontal-axis mesh-training budget.
TokenisereescratchTRL2DShear flowTRL2D ++ Shear
MNC (ours)01.116 ±\pm 0.0831.086 ±\pm 0.1610.999 ±\pm 0.4130.556 ±\pm 0.034
 250.042 ±\pm 0.0010.039 ±\pm 0.0000.039 ±\pm 0.0020.037 ±\pm 0.002
 500.035 ±\pm 0.0010.031 ±\pm 0.0010.031 ±\pm 0.0010.030 ±\pm 0.001
 1000.026 ±\pm 0.0000.024 ±\pm 0.0010.025 ±\pm 0.0010.023 ±\pm 0.000
GINO00.993 ±\pm 0.0340.576 ±\pm 0.0130.528 ±\pm 0.0080.547 ±\pm 0.008
 250.373 ±\pm 0.0240.312 ±\pm 0.0140.362 ±\pm 0.0500.318 ±\pm 0.017
Appendix
Table 10: Cross-domain reconstruction transfer to native cylinder-flow meshes: normalised MSE after ee mesh-training epochs (mean ±\pm standard deviation, three seeds). e=0e=0 is zero-shot; “scratch” has no pretraining. The source is CNN for MNC, and each baseline’s own tokeniser for GINO and LNO. These are not forecasting results.

Explore similar work

CardsList
Mar 5, 2026

On the Value of Tokeniser Pretraining in Physics Foundation Models

We investigate the impact of tokeniser pretraining on the accuracy and efficiency of physics emulation. Modern high-resolution simulations produce vast volumes of data spanning diverse physical regimes and scales. Training foundation models to learn the dynamics underlying such data enables the modelling of complex multiphysics phenomena, especially in data-limited settings. The emerging class of physics foundation models typically aims to learn two tasks jointly: (i) extracting compact representations of high-resolution spatiotemporal data, and (ii) capturing governing physical dynamics. However, learning both tasks from scratch simultaneously can impede the effectiveness of either process. We show that pretraining the tokeniser with an autoencoding objective prior to training the dynamics model enhances computational efficiency for physics emulation. Notably, the magnitude of this benefit depends on domain alignment: pretraining on the same physical system as the emulation task yields the largest improvements, while pretraining on other systems provides moderate gains. In-domain pretraining reduces VRMSE by 64% after 10,500 training steps compared to training from scratch. To our knowledge, this is the first systematic investigation of tokeniser pretraining for physics foundation models. We further introduce flexible spatiotemporal compression operations that extend causal convolutions to support runtime-adjustable compression ratios, enabling efficient adaptation to diverse downstream tasks. Our findings provide practical guidance for training efficient physics emulators and highlight the importance of strategic pretraining data selection.

Hadi Sotoudeh, Payel Mukhopadhyay, Ruben Ohana +4

University of Cambridge · Flatiron Institute · New York University +1

Physics-Based SimulationDomain-Adaptive Pretraining
AskPDF
May 1, 2026

Mesh Field Theory: Port-Hamiltonian Formulation of Mesh-Based Physics

We present Mesh Field Theory (MeshFT) and its neural realization, MeshFT-Net: a structure-preserving framework for mesh-based continuum physics that cleanly separates the physics' topological structure from its metric structure. Imposing minimal physical principles (locality, permutation equivariance, orientation covariance, and energy balance/dissipation inequality), we prove a reduction theorem for mesh-based physics. Under these conditions, the physical dynamics admit a local factorization into a port-Hamiltonian form: the conservative interconnection is fixed uniquely by mesh topology, whereas metric effects enter only through constitutive relations and dissipation. This reduction clarifies what must be fixed and what should be learned, directly informing MeshFT-Net's design. Across evaluations on analytic and realistic datasets, physics-consistency tests, and out-of-distribution validation, MeshFT-Net achieves near-zero energy drift and strong physical fidelity (correct dispersion and momentum conservation) along with robust extrapolation and high data efficiency. By eliminating non-physical degrees of freedom and learning only metric-dependent structure, MeshFT provides a principled inductive bias for stable, faithful, and data-efficient learning-based physical simulation.

Satoshi Noguchi, Yoshinobu Kawahara

Center for Mathematical Science and Advanced Technology, JAMSTEC · Center for Advanced Intelligence Project, RIKEN · Graduate School of Information Science and Technology, The University of Osaka.

Dynamical SystemsScientific MLPhysics-Informed MLPort-Hamiltonian SystemsPhysics-Based Simulation
AskPDF
Jun 3, 2026

MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers

Conventional patchified Transformers operate on uniform spatial partitions, distributing computational effort evenly across the domain irrespective of local features. This inflexible tokenization scheme is inherently limited in its ability to efficiently represent and process solutions to complex PDEs. To address this, we propose MeshTok, an adaptive mesh refinement (AMR)-inspired tokenization and sequence modeling framework. This method selectively refines spatial regions exhibiting sharp gradients, transient features, or multiscale structures, generating a heterogeneous set of multiscale tokens defined on a fixed simulation grid. These tokens are processed within a unified Transformer sequence, enabling the model to simultaneously capture coarse-grained global context and fine-grained local details without requiring specialized architectural components. Although adaptive refinement moderately increases token count, it promotes a more targeted allocation of computational resources to physically informative regions, which we view as a practical inductive bias rather than a formal optimality guarantee. Experimental evaluations across multiple PDE families and benchmark datasets demonstrate that MeshTok consistently improves the efficiency-accuracy trade-off compared to uniform-grid baselines. This suggests adaptive multiscale tokenization as a scalable and generalizable design principle for neural PDE modeling. Code is available at https://github.com/SCAILab-USTC/MeshTok.

Yanshun Zhao, Xiaoyu Peng, Jiamin Jiang +2

School of Mathematical Sciences, University of Science and Technology of China, Hefei 230026, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou 215123, China.

Neural PDE SolversPDE Surrogate ModelingAdaptive Mesh Refinement
AskPDF