Does Transolver really need a Transformer?

Shizheng Wen; Siddhartha Mishra

cs.LGSep 26, 2026

Does Transolver really need a Transformer?

Authors: Shizheng Wen, Siddhartha Mishra

Organizations: Seminar for Applied Mathematics, ETH Zurich, Switzerland

Abstract

The widely used Transolver family of neural operators is based on physics-attention, which softly assigns the points of an unstructured mesh to a small number of slices, applies self-attention among the resulting tokens, and broadcasts the result back to the points. We provide a comprehensive empirical and theoretical analysis to elucidate the mechanisms which are responsible for model performance. To this end, we perform careful ablations on a challenging suite of nine 3D fluid dynamics benchmarks to find that replacing token attention with a constant linear map does not affect the accuracy. Thus, Transolver does not need a Transformer at all. However, removing the global mixing (slicing/deslicing) or doing it only once leads to performance collapse. We leverage the theory of averaging neural operators to explain and corroborate our findings by showing that just slicing/deslicing, in conjunction with pointwise MLPs, already suffices for universal approximation of continuous operators and attention is redundant in this context. Finally, we provide a novel FlashAttention-style efficient implementation of the key slicing/deslicing module of Transolver. This flashslice kernel streams slice and deslice over the points without materializing the heavy slice-weight tensor, while reproducing the best available implementation to floating point error. At the same time, it leads to very significant memory and compute savings, particularly at large slice counts.

Figures & tables

Figure 1: What Transolver’s accuracy does, and does not, need. (a) A Transolver layer: a persistent full-resolution point stream, coupled through slice and deslice with shared per-point slice weights $w$ to $H{\times}G$ tokens that are re-formed at every layer. Each ablation is drawn on the element it changes; $w^{(1)}$ denotes the layer- $1$ weights reused by FrozenSlice and $w^{\prime}$ the separate deslice weights of Untied . (b) Validation-error ratio to the full model for the five ablations on nine benchmarks, markers as in (a). The variants that keep the coupling ( NoTokenAttn , Untied , FrozenSlice ) sit at $\approx 1\times$ ; removing it ( MlpOnly ) costs $1.04$ – $5.6\times$ and collapsing the point stream ( SliceOnce ) up to $6.8\times$ . (c) One reference solution per benchmark family, each beside the rows of (b) it belongs to: Taylor–Green vorticity, and surface pressure with streamlines on SHIFT-Wing, DrivAerNet++, SHIFT-SUV and DrivAerML.

SHIFT-Wing
Dataset	Baseline	NoTokenAttn	MlpOnly	Untied	FrozenSlice	SliceOnce
Taylor–Green	$0.0756{\scriptstyle\pm.0001}$	$\mathbf{0.0745}{\scriptstyle\pm.0001}$	$0.0786$	$0.0755$	$0.0777$	$0.188$
surface	$0.0580{\scriptstyle\pm.0005}$	$0.0579{\scriptstyle\pm.0005}$	$0.139$	$\mathbf{0.0576}$	$0.0577$	$0.155$
volume	$\mathbf{0.127}{\scriptstyle\pm.0007}$	$\mathbf{0.127}{\scriptstyle\pm.0008}$	$0.408$	$0.128$	$0.135$	$0.793$
DrivAerNet++
surface	$0.193{\scriptstyle\pm.005}$	$\mathbf{0.188}{\scriptstyle\pm.002}$	$0.294$	$0.190$	$0.192$	$0.435$

Table 1: Ablating physics-attention across nine benchmarks. Best validation relative

L^{1}

(lower is better), averaged over the predicted quantities in normalized units; per-quantity errors in physical units are in Tables D.2 and D.3 . Mean

\pm

std over three seeds for Baseline and NoTokenAttn , single seed otherwise; best per row in bold . Three significant figures; standard deviations to one.

Figure 2: Four controls. (a) Token attention

\times

projection tying on SHIFT-SUV surface, change against the tied Baseline ; Untied-LN is the released LinearNO layer ( Hu et al., 2026 ) in our scaffold. (b) NoTokenAttn against Baseline across the slice count

G

, same benchmark. Grey band in (a, b): the baseline’s three-seed spread. (c) Capacity control on DrivAerNet++; bars labelled by parameter count relative to Baseline . (d) Test error of MlpOnly over Baseline per field, pressure against velocity or wall shear (Table D.2 ).

Figure 3: FlashSlice turns the reduced layer into a systems object. (a) The point tensors and the

32

KB token block live in HBM; the slice weights exist only as one

B_{N}{\times}G

tile in registers, formed on the fly in each of two passes over the points and recomputed in the backward, so the

N{\times}H{\times}G

tensor is never written. (b, c) Peak memory of a training step against slice count and depth (

N{=}262

k, eager PyTorch, no compilation or checkpointing);

\times

marks the first configuration that does not fit in

95

GB. Step times of the same sweeps are in Figure F.1 (Appendix F ).

			time (ms)			peak memory (GB)
		$N$	eager	ours	speedup	eager	ours	saved
full model, $G{=}32$ , single-tile kernels
training	fp32	$262$ k	$171$	$126$	$1.35\times$	$28.0$	$24.0$	$14\%$
training	bf16	$262$ k	$143$	$93.1$	$1.53\times$	$20.4$	$15.4$	$24\%$
training	bf16	$1$ M	$585$	$345$	$1.70\times$	$81.3$	$61.3$	$25\%$
inference	fp32	$8.4$ M	$1944$	$1160$	$1.68\times$	$80.5$	$64.5$	$20\%$

Table 2: FlashSlice against eager PyTorch on one GH200 (PyTorch 2.5, Triton 3.0; medians of

\geq 10

iterations measured in one job; no compilation or checkpointing; “OOM”

=

does not fit in

95

GB). Top: the model of our experiments (

L{=}8

C{=}256

H{=}8

G{=}32

), single-tile kernels, exact fp32 dots in fp32 and bf16 tensor-core dots in bf16. Bottom: one layer (

H{=}8

D{=}32

) at large slice counts,

G

-blocked kernels, bf16 inputs and dots.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Learned slice functions with and without attention. Last-layer slice functions $s_{g}(x)$ for Baseline (top) and NoTokenAttn (bottom), shown on the DrivAerML test car from Figure D.1 . The first three columns show the closest pairs under Hungarian matching by cosine similarity over points ( $0.82$ , $0.72$ , and $0.69$ ). Each slice is normalized by its own maximum. The rightmost column shows per-point assignment entropy on a shared color scale.

variant	nonlocal budget	hard ceiling?	measured cost
Baseline / Untied	$L\,G\,c$	no (Prop. 1 )	reference
NoTokenAttn	$L\,G\,c$	no (Prop. 1 )	$-2.6\%$ to $+7.9\%$
FrozenSlice	$L\,G\,c$	no (§ A.4 )	$-0.5\%$ to $+17\%$
SliceOnce	$G\,c$	yes, rank (Prop. 2 )	$2.2$ – $6.8\times$
MlpOnly	$0$	yes, locality (Cor. 1 )	$1.04$ – $5.6\times$

Appendix

Table A.1: What the theory settles, and what it leaves to experiment. Nonlocal budget and hard ceiling per variant, against the measured cost relative to Baseline in Table 1 .

Benchmark	steps	train/val/test	points/sample	subsample	in $\to$ out ch.	inputs	outputs
Taylor–Green	250k	1.55M/19.3k/38.6k	$32^{3}$	$1.0$	$5{\to}5$	state at $t$ (+ lead time)	state at $t{+}\Delta t$
SHIFT-Wing surface	250k	8377/1000/1000	1.6M	$0.1$	$8{\to}4$	normals, freestream (varying)	$p$ , $\tau$
SHIFT-Wing volume	140k ∗	8377/1000/1000	5.85M	$0.046$	$6{\to}4$	SDF, freestream (varying)	$p$ , $u,v,w$
DrivAerNet++ surface	184k	5825/1148/1154	459k	$0.5$	$8{\to}4$	normals, freestream (const.)	$p$ , $\tau$
DrivAerNet++ volume	180k	5825/1148/1154	11.0M	$0.02$	$6{\to}4$	SDF, freestream (const.)	$p$ , $u,v,w$
SHIFT-SUV surface	50k	798/100/100	2.57M	$0.092$	$3{\to}4$	normals only	$p$ , $\tau$

Appendix

Table B.1: Benchmark configurations. Training steps, split sizes, points per sample after cropping and before subsampling, training subsampling ratios, and input/output channels. ∗ Evaluated at the common checkpoint of 140k steps, reached within one 12-hour run using a 250k-step cosine schedule (Appendix C.1 ).

Figure C.1: Validation curves for all variants on the nine benchmark configurations (seed 42). Training budgets and exceptions are described in § C.1 .

Figure D.1: Surface pressure on a DrivAerML test car. Top: ground truth and the four variants. Bottom: each variant’s absolute error. The car is run_472 . Appendix D.2 states the protocol and repeats the figure on the other eight benchmarks.

Figure D.2: Taylor–Green velocity magnitude on a mid-plane of the $32^{3}$ cube, predicted from $t=1.28$ to $t=1.36$ . Relative $L^{1}$ errors: Baseline $0.149$ , NoTokenAttn $0.139$ , MlpOnly $0.148$ , SliceOnce $0.450$ . SliceOnce smooths the fine vortex structure.

Figure D.3: SHIFT-Wing surface pressure for sample 016040 . Relative $L^{1}$ errors: Baseline $0.00085$ , NoTokenAttn $0.00088$ , MlpOnly $0.0098$ , SliceOnce $0.0086$ . The latter two smooth the leading-edge pressure pattern.

Figure D.4: SHIFT-Wing surface wall shear for the sample in Figure D.3 . Relative $L^{1}$ errors: Baseline $0.057$ , NoTokenAttn $0.057$ , MlpOnly $0.082$ , SliceOnce $0.104$ .

Figure D.5: SHIFT-Wing volume pressure at $y=15.3$ for sample 016032 . Relative $L^{1}$ errors: Baseline $0.0019$ , NoTokenAttn $0.0022$ , MlpOnly $0.039$ , SliceOnce $0.085$ . SliceOnce produces a nearly uniform field.

Figure D.6: SHIFT-Wing volume velocity magnitude on the section in Figure D.5 . Relative $L^{1}$ errors: Baseline $0.174$ , NoTokenAttn $0.177$ , MlpOnly $0.299$ , SliceOnce $0.451$ .

Figure D.7: DrivAerNet++ surface pressure for car N_S_WWC_WM_388 . Relative $L^{1}$ errors: Baseline $0.095$ , MlpOnly $0.168$ , SliceOnce $0.210$ .

Figure D.8: DrivAerNet++ surface wall shear for the car in Figure D.7 . Relative $L^{1}$ errors: Baseline $0.194$ , NoTokenAttn $0.204$ , MlpOnly $0.269$ , SliceOnce $0.476$ .

Figure D.9: DrivAerNet++ volume pressure on the symmetry plane for car N_S_WWC_WM_391 . Relative $L^{1}$ errors: Baseline $0.092$ , NoTokenAttn $0.097$ , MlpOnly $0.242$ , SliceOnce $0.251$ .

Figure D.10: DrivAerNet++ volume velocity magnitude on the plane in Figure D.9 . Relative $L^{1}$ errors: Baseline $0.088$ , NoTokenAttn $0.092$ , MlpOnly $0.194$ , SliceOnce $0.274$ .

Figure D.11: SHIFT-SUV surface pressure for car run_00939 , with surface normals as input. Relative $L^{1}$ errors: Baseline $0.024$ , NoTokenAttn $0.024$ , MlpOnly $0.113$ , SliceOnce $0.195$ .

Figure D.12: SHIFT-SUV surface wall shear for the car in Figure D.11 . Relative $L^{1}$ errors: Baseline $0.256$ , NoTokenAttn $0.257$ , MlpOnly $0.328$ , SliceOnce $0.558$ .

Figure D.13: SHIFT-SUV volume pressure on the symmetry plane for car run_00939 , with signed distance as input. Relative $L^{1}$ errors: Baseline $0.027$ , NoTokenAttn $0.026$ , MlpOnly $0.127$ , SliceOnce $0.192$ .

Figure D.14: SHIFT-SUV volume velocity magnitude on the plane in Figure D.13 . Relative $L^{1}$ errors: Baseline $0.055$ , NoTokenAttn $0.057$ , MlpOnly $0.150$ , SliceOnce $0.385$ .

Figure D.15: DrivAerML surface wall shear for the car in Figure D.1 . Relative $L^{1}$ errors: Baseline $0.107$ , NoTokenAttn $0.115$ , MlpOnly $0.546$ , SliceOnce $0.736$ .

Figure D.16: DrivAerML volume pressure on the symmetry plane for car run_108 . Relative $L^{1}$ errors: Baseline $0.051$ , NoTokenAttn $0.053$ , MlpOnly $0.277$ , SliceOnce $0.474$ .

Figure D.17: DrivAerML volume velocity magnitude on the plane in Figure D.16 . Relative $L^{1}$ errors: Baseline $0.111$ , NoTokenAttn $0.112$ , MlpOnly $0.446$ , SliceOnce $0.701$ .

Benchmark	$p$	$\mathbf{u}$ / $\bm{\tau}_{w}$	max rel. std
DrivAerNet++ surface	$0.10170\pm.00003$	$0.19222\pm.00004$	$0.02\%$
DrivAerNet++ volume	$0.10634\pm.00004$	$0.10096\pm.00002$	$0.04\%$
DrivAerML surface	$0.03876\pm.00006$	$0.11286\pm.00009$	$0.16\%$
DrivAerML volume	$0.05184\pm.00007$	$0.11112\pm.00008$	$0.14\%$
SHIFT-Wing surface	$0.001018\pm.000001$	$0.05863\pm.00001$	$0.10\%$
SHIFT-Wing volume	$0.002251\pm.000001$	$0.17355\pm.00002$	$0.03\%$

Appendix

Table D.1: Subsampling noise of the evaluation metric. Baseline checkpoint (seed 43 on SHIFT-Wing volume), test split, six independent subsample seeds; mean

\pm

std of the per-variable relative

L^{1}

error.

Dataset	Variable	baseline	no-token-attn	MLP-only	untied	frozen	slice-once
Taylor–Green	$\rho$	$0.00023$	$0.00035$	$0.00024$	$0.00029$	$\mathbf{0.00022}$	$0.00026$
	$\mathbf{u}$	$0.1420$	$\mathbf{0.1397}$	$0.1474$	$0.1416$	$0.1459$	$0.3581$
DrivAerNet++ surf.	$p$	$\mathbf{0.1017}$	$0.1057$	$0.1830$	$0.1050$	$0.1047$	$0.2172$
	$\bm{\tau}_{w}$	$\mathbf{0.1922}$	$0.1981$	$0.2759$	$0.1928$	$0.1992$	$0.4635$
DrivAerNet++ vol.	$p$	$0.1064$	$\mathbf{0.0953}$	$0.2188$	$0.0979$	$0.0963$	$0.2253$
	$\mathbf{u}$	$0.1010$	$0.0940$	$0.1639$	$0.0934$	$\mathbf{0.0933}$	$0.2418$

Appendix

Table D.2: Per-variable relative

L^{1}

test error in physical units. Best per row in bold; protocol and reading notes in § D.3 .

Dataset	Variable	baseline	no-token-attn	MLP-only	untied	frozen	slice-once
Taylor–Green	$\rho$	$0.00034$	$0.00044$	$0.00035$	$0.00039$	$\mathbf{0.00031}$	$0.00054$
	$\mathbf{u}$	$0.1850$	$\mathbf{0.1826}$	$0.1906$	$0.1845$	$0.1888$	$0.4470$
DrivAerNet++ surf.	$p$	$\mathbf{0.1509}$	$0.1563$	$0.2510$	$0.1553$	$0.1564$	$0.3479$
	$\bm{\tau}_{w}$	$0.2265$	$0.2316$	$0.3098$	$\mathbf{0.2262}$	$0.2332$	$0.5272$
DrivAerNet++ vol.	$p$	$0.1621$	$\mathbf{0.1462}$	$0.3087$	$0.1481$	$0.1474$	$0.3709$
	$\mathbf{u}$	$0.1337$	$0.1234$	$0.2088$	$0.1229$	$\mathbf{0.1222}$	$0.2770$

Appendix

Table D.3: Per-variable relative

L^{2}

test error in physical units (same protocol as Table D.2 ).

Figure E.1: LinearNO normalization and latent width. (a) Slice and deslice weights in Transolver and LinearNO,with their normalization axes. LinearNO uses latent width

M=\mathrm{key\_ratio}\times d_{\mathrm{head}}

. (b) Validation errors on SHIFT-SUV surface at shared training steps, plotted against runtime estimated from measured iteration times on a four-GH200 node. The settings

\mathrm{key\_ratio}=1

and

4

give

M=32

and

128

, respectively. The diamond marks the completion of

50

k steps for

\mathrm{key\_ratio}=1

step	1k	3k	5k	7k	9k	11k
$\mathrm{key\_ratio}=1$	$0.4952$	$0.3005$	$0.2577$	$0.2210$	$0.2032$	$0.1927$
$\mathrm{key\_ratio}=4$	$0.4714$	$0.2973$	$0.2543$	$0.2232$	$0.2049$	$0.1938$
ratio	$0.95$	$0.99$	$0.99$	$1.01$	$1.01$	$1.01$

Appendix

Table E.1: LinearNO latent width on SHIFT-SUV surface. Validation relative

L^{1}

errors at selected shared checkpoints. Both runs use a

50

k-step cosine schedule. The ratio is the error at

\mathrm{key\_ratio}=4

divided by that at

\mathrm{key\_ratio}=1

level	logits dot	value dots	inputs	gate
ieee (default)	fp32 FMA	fp32 FMA	fp32 or bf16	$1.25\times$ eager fp32
tf32	fp32 FMA	tf32 tensor cores	fp32 or bf16	$10\times$
bf16v	fp32 FMA	bf16 tensor cores	bf16	$2\times$ eager bf16 autocast
bf16	bf16 tensor cores	bf16 tensor cores	bf16	$2\times$ eager bf16 autocast
tf32x3	three tf32 products	three tf32 products	fp32	$10\times$ ; measured $3$ – $7\times$ eager fp32

Appendix

Table F.1: Dot-precision levels of the kernels. Accumulation is fp32 throughout; the gate is the admissible error against an fp64 reference, relative to the corresponding eager computation. ieee is Triton’s FMA path and needs no tensor cores.

fp32
	inference: ms (GB)		training step: ms (GB)
$N$	eager	ours	eager	ours
$262$ k	$58.4$ ( $2.6$ )	$35.3$ ( $2.1$ )	$171$ ( $28.0$ )	$126$ ( $24.0$ )
$1$ M	$229$ ( $10.1$ )	$138$ ( $8.1$ )	OOM	OOM
$2$ M	$475$ ( $20.2$ )	$280$ ( $16.2$ )	OOM	OOM
$4$ M	$980$ ( $40.3$ )	$566$ ( $32.3$ )	OOM	OOM

Appendix

Table F.2: Complete single-GPU sweep behind Table 2 (one job, eager PyTorch, no compilation, no checkpointing). Each cell is median time in ms with peak memory in GB; “OOM”

=

does not fit in

95

GB. “ours” is the exact-fp32-dot tier in the fp32 block and the bf16 tensor-core tier under autocast.

	fp32, peak GB		bf16, peak GB		bf16 step
	eager	ours	eager	ours	speedup
slice count $G$ (depth $L{=}8$ )
$G{=}16$	$26.0$	$24.0$	$17.9$	$\mathbf{15.4}$	$1.31\times$
$G{=}32$	$28.0$	$24.0$	$20.4$	$\mathbf{15.4}$	$1.54\times$
$G{=}64$	$32.0$	$24.0$	$25.4$	$\mathbf{15.4}$	$1.83\times$
$G{=}128$	$41.7$	$24.0$	$36.9$	$\mathbf{15.4}$	$2.18\times$

Appendix

Table F.3: Structural scaling : peak memory of a training step at

N{=}262

k as the two configuration knobs move, each block measured in one job. “OOM”

=

does not fit in

95

GB. Fused peak memory changes little over the tested range of

G

; both grow linearly in depth, with different slopes.

Figure F.1: Step time of the sweeps of Figure 3 (b, c). Median time of a training step against (a) slice count and (b) depth (

N{=}262

k, eager PyTorch against the fused kernels, no compilation or checkpointing; log axes). The speedup grows with

G

, from

1.31\times

G{=}16

2.18\times

G{=}128

under bf16, and is a constant factor in depth.

	fp32, $262$ k	bf16, $262$ k	bf16, $1$ M
eager	$170.5$	$142.9$	$586$
eager $+$ compile	$134.8$	$79.5$	$300$
fused (exact fp32 dots)	$128.4$	$118.9$	$448$
fused $+$ compile $+$ fast dot	$\mathbf{106.6}$	$\mathbf{63.9}$	$\mathbf{241}$
vs. strongest baseline	$1.26\times$	$1.24\times$	$1.25\times$

Appendix

Table F.4: Single-GPU training step (ms) against compiled baselines , medians of

\geq 10

iterations; each precision block is measured in one job, and anchors reproduce across jobs within

1\%

. “fast dot”

=

TF32 (fp32 inputs) or bf16 tensor-core (bf16 inputs) tier.

	bf16, $1$ M	bf16, $4$ M	fp32, $1$ M	fp32, $2$ M
ckpt	$740$	$2999$	$921$	$1874$
ckpt $+$ compile	$365$	$1479$	$696$	$1432$
fused ckpt $+$ compile $+$ fast dot	$\mathbf{278}$	$\mathbf{1106}$	$\mathbf{500}$	$\mathbf{1006}$
vs. ckpt $+$ compile	$1.31\times$	$1.34\times$	$1.39\times$	$1.42\times$

Appendix

Table F.5: Checkpointed training (ms), one job. torch.compile traverses non-reentrant activation checkpointing in this stack, halving the checkpointed baseline; the baseline uses checkpointing and compilation.

inputs / dots	$G$	$N$	eager (ms)	ours (ms)	speedup	eager (GB)	ours (GB)	saved
bf16 / bf16	$256$	$262$ k	$47.5$	$11.9$	$3.98\times$	$13.7$	$2.0$	$85\%$
bf16 / bf16	$256$	$1$ M	$293.7$	$45.5$	$6.45\times$	$54.5$	$8.1$	$85\%$
bf16 / bf16	$512$	$262$ k	$90.6$	$20.0$	$4.53\times$	$26.7$	$2.0$	$92\%$
bf16 / bf16	$512$	$1$ M	OOM	$80.3$	—	—	$8.1$	—
bf16 / bf16	$1024$	$262$ k	$180.2$	$37.6$	$4.79\times$	$52.7$	$2.1$	$96\%$
bf16 / bf16, inference	$256$	$1$ M	$39.4$	$13.3$	$2.96\times$	$17.0$	$2.1$	$88\%$

Appendix

Table F.6: One layer at large slice counts, eager against the

G

-blocked kernels (

H{=}8

D{=}32

, one GH200, medians of ten; training step unless marked). “inputs / dots” is the activation dtype and the dot-precision level of Table F.1 ; “OOM”

=

does not fit in

95

GB.

	step (ms)	Mpts/s	GB/GPU
ckpt $+$ sharding	$3048$	$5.5$	$86.5$
ckpt $+$ sharding $+$ compile	$3043$	$5.5$	$86.6$
fused ckpt $+$ compile $+$ bf16 dot	$\mathbf{1152}$	$\mathbf{14.6}$	$78.6$

Appendix

Table F.7: One node,

4\times

GH200,

16.8

M points, bf16 , all configurations in one job; DDP-equivalent step including the gradient all-reduce. Compilation gives little improvement on the eager sharded pipeline; the operator form composes.

Neural Operators

Neural operators learn mappings between functions, enabling fast surrogates for PDEs and other continuous physical systems.

See all 304 papers

LLT: Local Linear Transformer for PDE Operator Learning
Jul 4, 2026Oded Ovadia, Eli TurkelPDE Surrogate Modeling Transformer Attention
School of Mathematical Sciences, Tel Aviv University, Tel Aviv, Israel
Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations
Sep 30, 2026Ming Zhong, Antonio Colanera, Gianluigi Rozza +1PDE Solving Attention Mechanisms
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 100049, China. · State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China. · International School for Advanced Studies (SISSA), Trieste 34136, Italy. +1
GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer
Dec 23, 2025Corey Adams, Rishikesh Ranade, Ram Cherukuri +1Neural Surrogate Modeling Structural Mechanics
NVIDIA Corporation, Santa Clara, US

Jul 4, 2026J/K move · Enter open · S save

LLT: Local Linear Transformer for PDE Operator Learning

Oded Ovadia, Eli Turkel

School of Mathematical Sciences, Tel Aviv University, Tel Aviv, Israel

PDE Surrogate Modeling Transformer Attention Linear Attention PDE Operator Learning Neural Operators Transformer Neural Operator

Open paper AskPDF arXiv

Neural operators have become a common approach for learning PDE solution maps and accelerating numerical simulations. Transformer-based neural operators are of particular interest, since attention can learn long-range dependencies in the computational domain. However, standard attention has two major limitations when applied to PDEs: it scales quadratically with the number of computational nodes, and it lacks an explicit bias toward local interactions. To address these issues, we introduce Local Linear Transformer (LLT) for PDE operator learning. The architecture combines linear global attention with local spatial mixing, and incorporates coordinate and geometry information. We evaluate LLT on several PDE problems, including elasticity, plasticity, airfoil flow, pipe flow, and Darcy flow. The reference data for these problems span finite-element, finite-volume, and finite-difference discretizations on structured and unstructured meshes. Compared with other neural-operator and transformer baselines from prior studies, LLT achieves competitive or lower relative

L_2

error across these problems. On matched structured discretizations, wall-clock time per training iteration is reduced by factors of 1.8 to 2.5 relative to Transolver. We also scale the approach and apply it to a three-dimensional car aerodynamics dataset with 32,186 unstructured mesh points per sample. Together, these results indicate that LLT provides an accurate and computationally efficient operator for PDE problems across discretizations, mesh types, and problem settings.

Abstract

Neural Operators

Explore similar work

LLT: Local Linear Transformer for PDE Operator Learning

Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations

GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer