Learning Incompressible 3D Flows from Sparse Samples: A Set Transformer Approach to Divergence-Free Kernel Fitting
Authors: Ramzi Idoughi, Radouan Boukharfane, Peter Wonka
Organizations: King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Mohammed VI Polytechnic University (UM6P), Benguerir, Morocco
Recovering divergence-free velocity fields from sparse and corrupted observations requires reconciling physical consistency, expressive representations, and robustness to missing data. We introduce DFK-Former, a set-to-field transformer that reconstructs divergence-free velocity fields from sparse, noisy, and irregularly sampled 3D observations. The model is inherently invariant to the number and ordering of input samples and predicts the parameters of a divergence-free kernel representation, producing continuous, mesh-free velocity fields that satisfy incompressibility exactly by construction rather than approximately through soft constraints or projection operators. Unlike existing divergence-free kernel approaches, which estimate kernel coefficients via a separate optimization for each field, DFK-Former predicts these coefficients directly in a single forward pass. This eliminates costly iterative fitting while providing a learned prior that enables accurate reconstruction even when observations are insufficient to uniquely determine the field. To enhance representational power, we introduce adaptive and anisotropic divergence-free kernels that preserve exact incompressibility guarantees while capturing more complex flow structures. Extensive experiments across varying levels of sparsity, noise, and occlusion, including transfer to real Direct Numerical Simulation (DNS) turbulence data, demonstrate substantial gains in both accuracy and efficiency. DFK-Former reduces endpoint error by up to 4.6x compared to per-field divergence-free-kernel optimization, while reconstructing each field in 7.5 ms instead of 9.8 s, yielding a speedup of more than three orders of magnitude. These results establish direct, feed-forward prediction of divergence-free kernel parameters as a practical and effective paradigm for physically consistent velocity-field reconstruction from sparse observations.
Figures & tables
Figure 1: DFK-Former architecture . Sparse spatiotemporal velocity observations are encoded into tokens and jointly processed with learned kernel-query and register tokens using a Perceiver-style spatiotemporal transformer. Decoding heads predict the parameters of a divergence-free kernel representation, yielding a continuous time-varying incompressible velocity field by construction.
Method
RMSE/ \EPE↓
rel. L2↓
MAE ↓
cosine ↑
angle ( ∘ ) ↓
∥∇⋅v∥2↓
ms/field ↓
div-free?
Wendland RBF (scalar)
0.598
0.736
0.280
0.663
37.9
3.2
26
no
SIREN ( + soft penalty)
0.392
0.519
0.195
0.805
26.7
1.4
53600
no (soft)
Curl-SIREN
0.950
1.352
0.475
0.629
42.0
∼10−14
52700
by construction
Vector-potential INR
0.778
1.092
0.388
0.661
39.1
∼10−15
48000
by construction
DFK fitting (fixed kernels)
0.689
0.924
0.335
0.482
52.9
∼10−15
22
by construction
DFK fitting (adaptive kernels)
0.546
0.719
0.264
0.568
45.5
∼10−15
9821
by construction
Table 1: Main reconstruction benchmark. Reconstruction accuracy, divergence residual, and per-field inference cost of DFK-Former against optimization-based, operator-learning, and physics-constrained baselines. Best in bold , second-best underlined .
Regime
Temporal DFK-Former (ours)
Per-frame DFK
Shared-kernel DFK
ST-SIREN
ST-Curl-SIREN
clean&dense
0.114
0.100
0.083
0.442
0.781
sparse
0.109
0.847
1.093
0.388
0.590
occluded
0.098
0.951
1.511
0.413
0.766
Overall (median \EPE )
0.114
0.767
0.626
0.428
0.842
Overall (mean \EPE )
0.124
20.29 †
0.762
0.466
0.821
Overall (mean angular error)
4.8 ∘
40.0 ∘
25.4 ∘
21.3 ∘
37.1 ∘
Table 2: Temporal flow reconstruction on 60 time-resolved JHTDB flows. Median \EPE by observation regime, plus overall mean \EPE and angular error; lower is better. Best results are in bold , second-best underlined .
Figure 2: Temporal reconstruction in the hardest evaluation regime (Forced MHD (dynamic JHTDB), hard combo : sparse observations, noise, occlusions, and outliers). Top: ground truth and predictions from Temporal DFK-Former, per-frame DFK fitting, iterative DFK fitting, space-time Curl-SIREN, and space-time SIREN. Bottom: input observations and reconstruction errors. DFK-Former remains robust under severe degradation, recovering the dominant flow structures with substantially lower EPE and angular error than competing methods.
Table 5
Target structure
Isotropic
Anisotropic
improvement
Vortex tube (Lamb–Oseen)
0.0763
0.0656
1.16×
Shear layer
0.0719
0.0592
1.22×
Vortex ring
0.0835
0.0623
1.34×
Table 5: Anisotropy vs. a parameter-matched isotropic control on structured analytic flows: our anisotropic K=1024 model against a separately trained isotropic model at K=1542 , matched to within four scalar parameters. rel \EPE , lower is better; improvement is the isotropic/anisotropic ratio.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Setting
Static model: backbone, heads and kernel geometry
Token width dmodel
256
Backbone blocks
6 ; each block = one masked cross-attention sublayer + one self-attention sublayer
Attention heads / head dimension
8 / 32
Feed-forward hidden width
4dmodel=1024 , GELU
Dropout
0
Appendix
Table 6: DFK-Former architecture, as resolved for the reported static run. The upper block is the static model; the lower block gives the temporal model’s additions and the two places where it departs from the static geometry. K is the kernel count and Δ the mean kernel spacing.
Setting
Value
Optimization and schedule
Optimizer
AdamW, library-default β and ϵ ; weight decay 0
Peak learning rate
3×10−4
Schedule
constant to step 60,000 , cosine decay to step 145,000 , held at 10−5 thereafter; no warm-up
Gradient clipping
global norm 1.0
Batch size
32 fields per rank ×4 ranks =128 fields per step
Appendix
Table 7: Training recipe of the reported static model, read from its fully resolved configuration. Ablation arms differ from it only in the single factor each ablation names (Section E ); the temporal model’s departures are in the last block. The auxiliary block acts on the placement and shape heads during training only and has no effect at inference. Per-stage data distributions are in Table 9 ; supervision point counts are in Table 8 .
Figure 3: Training-data pipeline (2D shown, identical principles in 3D). Top: a Gaussian random field splits exactly in Fourier space into the solenoidal target v∗ and an irrotational part, sparse observations are sampled and corrupted, and the resulting measurements are used to reconstruct the clean divergence-free target. Panel divergences are measured spectrally. Bottom: examples from the CAF library.
Symbol
Meaning
Value / range
Domain and precision
L
periodic domain size, [0,L)3
1.0
G
FFT generation grid
643
—
generation / training precision
float64 / float32
Spectral generator (Eq. 10 )
k0
energy-containing wavenumber (cycles/domain)
U[1.5,3.0]
Appendix
Table 8: Data-generation parameters. Ranges are drawn independently per field; U denotes a uniform draw (continuous, or over integers where the range is written {⋅} ). Corruption entries give the values outside the curriculum; per-stage overrides are in Table 9 .
Stage
Steps
Sources (spectral / coherent)
χmax
νmax
M
pocc
Core
60,000
1.0 / −
0.5
0.05
[512,4096]
−
Hard
85,000
0.6 / 0.4
1.5
0.20
[128,2048]
0.5
Cooldown
5,000
0.7 / 0.3
1.0
0.15
[256,4096]
0.5
Appendix
Table 9: Training curriculum. Only the data distribution and loss weights change between stages. “ − ” means the term is off. χmax is the maximum irrotational contamination, νmax the maximum relative noise, pocc the occlusion probability.
Set
Generator
Fields
Role
in-dist
spectral, training band-limit
128
matched-distribution accuracy
CAF
coherent-structure mixture
64
coherent structure (in training)
Taylor–Green
single analytic family
64
per-family accuracy
Lamb–Oseen
single analytic family
64
per-family accuracy
vortex ring
single analytic family
64
per-family accuracy
shear layer
single analytic family
64
per-family accuracy
Appendix
Table 10: Composition of the 582 -flow evaluation benchmark. Synthetic sets use seeds disjoint from training. The CAF and single-family sets draw on the training structure library (Table 9 ), so they measure in-distribution accuracy; held-out families are evaluated in Table 21 . JHTDB sets are real DNS ( Li et al., 2008 ) , resampled to 643 and used for evaluation only.
Figure 4: The mean-flow gap. Kernels alone cannot represent net transport because their sum is zero-mean. Adding a mean-flow term substantially reduces approximation error.
Radius / spacing
channel
transitional b.l.
isotropic turb.
forced MHD
1.5
0.106
0.165
0.483
0.739
2.5
0.037
0.056
0.173
0.261
3.5
0.012
0.019
0.050
0.075
4.5
0.010
0.016
0.040
0.058
6.0
0.009
0.014
0.035
0.051
Appendix
Table 11: Best achievable fit (relEPE) of the anisotropic DFK span to clean real-DNS fields ( Li et al., 2008 ) at fixed K=8192 , as a function of the support radius expressed in kernel spacings. Overlap is the dominant design variable: the floor falls monotonically as the kernels overlap more.
same- K
same-memory
K
isotropic
aniso, equal-volume
aniso, extend-only
aniso, extend-only
256
0.173
0.251
0.139
0.173
512
0.137
0.247
0.092
0.122
1024
0.128
0.282
0.055
0.076
2048
0.124
0.356
0.038
0.045
4096
0.215
0.592
0.049
0.038
Appendix
Table 12: Anisotropic vs isotropic fit (relEPE) on a vortex-tube target: identical centers, radius scale 3.5Δ , Piola construction, aspect ratio 4 , 8000 fit samples, 4000 scoring points, ridge 10−4 . Same-memory reduces the anisotropic count to ⌊7K/11⌋ . The equal-volume column shows that preserving support volume is worse than not using anisotropy at all.
Factor
Change
Δ EPE
Budget
Set encoder
attention → DeepSets pooling
+49.1%
3 k steps
Kernel count
K:512→2048 (no 1024 point)
−12.0%
3 k steps
Adaptive placement
fixed → predicted centers/radii
−7.4%
3 k steps
Width
dmodel:128→384
−4.8%
30 k steps
Fourier position enc.
off → on
−2.4%
30 k steps
Orthogonality loss
λortho:0.1→0
+0.04%
30 k steps
Appendix
Table 13: Controlled architecture matrix: single-factor ablations on a shared reduced-scale base ( K=512 , dmodel=128 , 4 blocks, Core stage only), scored on the 128 -field Gaussian Random Fields set. These are ablation signal runs, not paper-scale numbers; small effects are quoted at the 10× step budget (Section E.2 ). Negative Δ is an improvement.
Figure 5: Axis-isolated robustness sweeps Reconstruction error (EPE; mean ± 95% bootstrap CI; 72 fields per point) as a function of corruption strength along a single axis, with all other factors fixed to nominal settings. DFK-Former (blue) is the strongest, exactly divergence-free feed-forward method. The dashed pink curve is an ablated variant of DFK-Former, while the grey dotted line denotes the oracle representation floor obtained by least-squares fitting the basis to the clean target, i.e., the best achievable error within the representation.
Method
M=64
noise 0.2
occl. 0.8
kpeak[12,18]
∥∇⋅v∥2
DFK-Former (ours)
0.714
0.477
0.713
0.687
2–7×10−7
FNO
0.742
0.501
0.749
0.760
0.40–0.67
DeepONet
0.816
0.723
0.856
0.862
≈0.2
DFK fitting (fixed kernels) ‡
0.804
0.636
0.805
0.842
∼10−16
DFK fitting (adaptive kernels)
1.628
2.167
2.425
2.361
∼10−15
Wendland RBF (scalar)
0.795
0.657
0.803
0.756
1.5–3.1
Appendix
Table 14: Multi-method sweep at each axis’s hard endpoint ( M=64 ; σrel=0.2 at M=512 ; occluded fraction 0.8 ; target kpeak∈[12,18] ). EPE, lower is better; the last column is residual divergence. The row before the floor is our own ablation, not a baseline: on the identical backbone it matches our accuracy and leaks divergence.
Axis
Range (easy → hard)
model EPE
margin vs fit
margin vs aniso. fit
Sample count M
2048→64
0.315→0.713
36.3%→11.2%
→15.8%
Noise ν
0→0.4
0.326→0.390
36.8%→26.5%
→27.7%
Occluded fraction
0→0.5
0.306→0.492
36.2%→26.0%
→29.5%
Target kmin – kmax
[3,5]→[12,18]
0.441→0.544
29.3%→32.0%
→38.0%
Appendix
Table 15: Single-axis corruption sweep: model EPE against per-field DFK fitting (fixed kernels) at its per-point best ridge, 500 fields ×3 seeds per point, 95% bootstrap confidence intervals non-overlapping at every point. The margin is 1−EPEmodel/EPEfit ; the last column repeats it at the hard endpoint against a fit over the model’s own anisotropic basis.
Synthetic (band-limited)
Real DNS (zero-shot)
Method
GRF
CAF
Taylor–Gr.
Lamb–Os.
vtx. ring
shear
ABC
iso.
MHD
chan.
tr. BL
Wendland RBF (scalar)
0.634
0.615
0.607
0.627
0.560
0.622
0.618
0.566
0.553
0.601
0.572
SIREN ( + soft penalty)
0.550
0.537
0.560
0.435
0.444
0.438
0.418
0.256
0.310
0.183
0.180
Curl-SIREN
0.998
1.177
1.167
1.102
1.028
1.108
1.005
0.673
0.729
0.769
0.699
Vector-potential INR
0.819
1.029
0.969
0.943
0.942
0.989
0.821
0.469
0.470
0.623
0.485
DFK fitting (fixed kernels)
0.545
0.643
0.482
0.715
0.692
0.743
0.548
0.752
0.732
0.889
0.842
Appendix
Table 16: Per-set breakdown of the main benchmark ( \EPE , lower is better). Bold marks the column minimum. GRF is the frozen Gaussian Random Fields set and CAF the coherent-structure mixture (Table 10 ). No method has trained on JHTDB, so those four columns are zero-shot throughout.
Table 17: Divergence quality , macro-averaged over the eleven static sets; the L2 column is in Table 1 . Our ∼10−7 is the float32 autograd diagnostic’s floor, not a property of the field: the DFK construction is solenoidal for any coefficients, centers and radii (Section A ).
Three turbulence sets
Ten-family battery
Method
rel. KE ↓
rel. Ω↓
band E(k)↓
rel. Ω↓
band E(k)↓
SIREN ( + soft penalty)
0.077
0.296
0.175
–
–
Curl-SIREN
1.388
4.962
1.576
–
–
DFK fitting (fixed kernels)
0.148
0.486
0.230
–
–
FNO
0.202
0.696
0.258
2.660
0.211
DeepONet
0.388
0.875
0.435
–
–
Appendix
Table 18: Physical-consistency battery , macro-averaged over the three turbulence sets (Gaussian Random Fields, JHTDB isotropic, JHTDB MHD; 128+18+18 fields; spectral curl on the node-aligned 643 grid). The last two columns repeat enstrophy and E(k) over the ten-family battery, available for the three methods run over all families.
Figure 6: Energy-spectrum fidelity. Shell-averaged E(k) (log–log) for ground truth and every method. Both axes are cropped to the band the ground truth carries: the Gaussian Random Fields target is band-limited (Section C.3 ), so an uncropped axis would compress every curve into the top of the panel. DFK-Former tracks the ground truth across the resolved band.
Method
ms/field ↓
scenes/s ↑
peak mem (MB) ↓
GFLOPs ↓
# params (M)
SIREN / Curl-SIREN (per-field fit)
53000
0.019
204
9500
0.20
DFK fitting (fixed kernels)
22
45
1285
81
0
DFK fitting (adaptive kernels)
9821
0.10
623
786
0
DFK-Former (ours)
7.5
134
68
41
10.2
Appendix
Table 19: Efficiency battery Runtime, throughput, peak memory, and computational cost for reconstructing a single scene ( M≈2 k, Q=256 , batch size 1 ). Lower is better except for throughput.
Sparsity axis
Noise axis ( σrel )
Method
1% ( M=64 ) †
2% ( M=128 ) †
5%
20%
50%
0%
5%
10%
20%†
Restricted training — sparsity arm M∈[256,1024] , noise arm σrel≤10% , every stage, retrained from step 0
FNO
0.753
0.665
0.580
0.407
0.362
0.495
0.496
0.497
0.501
DeepONet
0.812
0.782
0.766
0.756
0.750
0.757
0.755
0.748
0.724
DFK-Former (ours)
0.718
0.637
0.551
0.380
0.323
0.461
0.464
0.471
0.492
Full-range training ( M∈[128,2048] , σrel≤20% ) — unrestricted reference
Appendix
Table 20: Generalization to unseen sparsity and noise levels. Restricted models are trained only on moderate sparsity ( M∈[256,1024] ) or noise ( σrel≤10% ) and evaluated both within and beyond that range. Columns marked by † correspond to out-of-distribution settings not observed during training. DFK-Former remains competitive under both sparsity and noise extrapolation, with only modest degradation relative to its full-range counterpart. FNO and DeepONet use identical restricted-training protocols. Results report \EPE averaged over 24 fields and 3 random seeds. Lower is better.
Set
restricted
full library
ratio
Excluded families
Shear layer
0.425
0.054
7.95
Taylor–Green
0.301
0.053
5.71
Vortex ring
0.277
0.062
4.50
Lamb–Oseen
0.249
0.056
4.48
Included families
ABC (the one kept family)
0.074
0.036
2.04
Gaussian Random Fields
0.369
0.343
1.08
Appendix
Table 21: Flow-family generalization under training-set restriction. \EPE on the eleven evaluation sets. The restricted model is trained using only the ABC coherent-flow family, while keeping all other training components, sampling weights, corruption ranges, and spectral-flow data identical to the full setup. The ratio reports restricted/full . Large ratios on withheld families indicate strong transfer from training on diverse coherent structures, whereas ratios near one indicate limited dependence on flow-family diversity. The JHTDB datasets are zero-shot for both models.
Corruption regime
Temporal DFK-Former
Per-frame DFK
Shared-kernel DFK
ST-SIREN
ST-Curl-SIREN
clean&dense
0.114 (0.114)
0.100 (0.102)
0.083 (0.093)
0.442 (0.432)
0.781 (0.727)
contaminated
0.085 (0.122)
0.217 (0.291)
0.165 (0.191)
0.365 (0.448)
0.712 (0.781)
noisy
0.088 (0.097)
0.398 (1.874)
0.268 (0.281)
0.440 (0.482)
0.939 (0.926)
sparse
0.109 (0.129)
0.847 (0.828)
1.093 (1.067)
0.388 (0.453)
0.590 (0.617)
occluded
0.098 (0.110)
0.951 (107.1)
1.511 (1.519)
0.413 (0.451)
0.766 (0.819)
hard combo
0.138 (0.173)
0.916 (11.52)
1.420 (1.419)
0.497 (0.528)
1.029 (1.057)
Appendix
Table 22: Time-varying reconstruction of unobserved real DNS frames , all six corruption regimes (dynamic JHTDB: 60 flows, 180 test observations). Each cell is median (mean) endpoint error at times the model never sees. Per-frame DFK’s mean is inflated by 15/180 numerical failures.
Table 23: Temporal reconstruction by flow type. Median metrics on the dynamic JHTDB benchmark ( 180 test observations). Temporal DFK-Former consistently outperforms per-frame DFK fitting, with the largest gains on wall-bounded flows. The ratio column reports the relative improvement over a per-frame DFK baseline using the same kernel basis.
Corruption regime
Per-frame DFK
F=2 moving iso.
F=7 moving iso.
F=7 shared aniso.
F=2 shared aniso.
+ adaptive placement
clean&dense
0.100
0.115
0.174
0.133
0.114
0.117
contaminated
0.217
0.092
0.172
0.105
0.085
0.100
noisy
0.398
0.093
0.173
0.102
0.089
0.098
hard combo
0.916
0.146
0.222
0.152
0.138
0.152
sparse
0.847
0.118
0.194
0.125
0.109
0.117
occluded
0.951
0.103
0.186
0.117
0.098
0.108
Appendix
Table 24: Ablation of temporal parameterizations on real DNS sequences. Median EPE on the dynamic JHTDB benchmark ( 60 flows, 180 test observations). All temporal variants share the same K=1024 kernel budget, training data, and supervision. The best-performing design uses a shared anisotropic kernel geometry with low-order temporal dynamics ( F=2 ), outperforming both moving-kernel variants and per-frame DFK fitting. Adding adaptive kernel placement provides no benefit on real DNS data. Lower is better.
Figure 7: The well-posed case the model does not win (Forced MHD (dynamic JHTDB), clean&dense ): the DFK-span solves reach the fixed-basis optimum — shared-kernel DFK edging the learned prior out ( 0.207 ) and per-frame DFK matching it ( 0.230 each). The space-time INRs remain far behind ( 0.751 , 1.048 ). The learned prior earns its keep in the ill-posed regimes, not everywhere.
Figure 8: Temporal reconstruction on a representative real-DNS example (Forced MHD (dynamic JHTDB), sparse regime). Top: ground truth and predictions from Temporal DFK-Former, per-frame DFK fitting, iterative DFK fitting, space-time Curl-SIREN, and space-time SIREN. Bottom: sparse observations and reconstruction errors. DFK-Former best preserves the flow structure and achieves the lowest EPE while remaining exactly divergence-free.
Figure 9: The reconstruction as a function of time (Forced MHD (dynamic JHTDB), sparse regime). Columns are three real-DNS times the model never observes — frame midpoints; rows are ground truth, our model, and two baselines. The emitted field is continuous in {\color[rgb]{0,0,0}t} and exactly divergence-free at every {\color[rgb]{0,0,0}t} , not only where it was supervised.
School of Intelligence Science and Technology, Peking University, China · Peking University, China · State Key Laboratory of General Artificial Intelligence, Peking University, China
Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China · Shanghai Academy of Artificial Intelligence for Science, Shanghai, China