Self-Supervised Learning of Universal Latent Spaces for Operator Learning
Organizations: Department of Mechanical and Process Engineering, ETH Zurich, Switzerland · ETH AI Center, Zurich, Switzerland · Seminar for Applied Mathematics, ETH Zurich, Switzerland
Abstract
Neural operators fit their representation of functions to the task they are trained on, so every new task rebuilds a reduced space from labeled data. Following the self-supervised pre-training that made representations shareable in natural language processing and computer vision, we decouple representation learning from operator learning. A functional autoencoder is pre-trained on analytic functions synthesized on the fly, with no solver and no labels, to represent functions on arbitrary point clouds, and the latent space it induces is frozen and reused across tasks. The representation is exactly equivariant to translation, dilation, and amplitude, and reconstructs solutions of partial differential equations (PDEs) and natural images on geometries it never saw. Being task-agnostic, the latent space can be interpreted in its own right. The whole space is used and decodes to valid fields, it preserves the ordering of distances between physical fields, its geodesics interpolate physical evolutions, and edit directions transfer across fields. With frozen representation maps, a dataset is encoded once and a neural operator is trained entirely on cached embeddings, which saves memory and training time. On 28 tasks in fluid dynamics, solid mechanics, acoustics, and the Helmholtz and Poisson equations, this paradigm can exceed the accuracy of state-of-the-art end-to-end architectures, with up to an order of magnitude higher throughput and memory savings that fit models nearly two orders of magnitude larger on consumer-grade GPUs. At inference, latent rollouts run up to an order of magnitude faster than regular rollouts, and chunked inference processes functions of over half a billion points on a single consumer-grade GPU.
Figures & tables
Appendix figures & tables55 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Equation | Points | Description |
| Incompressible flow ( ) | |||
| IEU-BB | incompressible Euler | grid | Flow from a Brownian-bridge random initial condition. |
| IEU-Gauss | incompressible Euler | grid | Flow from Gaussian vortices. |
| IEU-PwC | incompressible Euler | grid | Flow from a piecewise-constant initial condition. |
| IEU-VortexSheet | incompressible Euler | grid | Flow from a perturbed vortex sheet. |
| Dynamic compressible flow ( on grids, on grids) | |||
| Daedalus (ours) | GAOT | RIGNO-12 | ||||||||||||
| Problem | Error [%] | TTT [h] | Error [%] | TTT [h] | Error [%] | TTT [h] | ||||||||
| Incompressible flow | ||||||||||||||
| IEU-BB | 2.00 | 2.5 | 2.83 | 42% | 8.7 | 3.5 | 12.8 | 540% | 26 | 10.6 | ||||
| IEU-Gauss | 4.63 | 2.7 | 6.97 | 51% | 5.1 | 1.9 | 37.6 | 713% | 26 | 9.7 | ||||
| IEU-PwC | 2.72 | 2.5 | 3.01 | 11% | 5.1 | 2.0 | 27.4 | 907% | 26 | 10.5 | ||||
| IEU-VortexSheet | 1.66 | 2.6 | 0.578 | 65% | 5.7 | 2.2 | 5.78 | 248% | 31 | 12.0 | ||||
| Latent loss | Physical loss | Cooldown | Unfrozen cooldown | |||||
| Problem | Error [%] | TTT [h] | Error [%] | TTT [h] | Error [%] | TTT [h] | Error [%] | TTT [h] |
| Evolution problems | ||||||||
| AWA-Layer | 7.14 | 2.1 | 12.2 | 2.9 | 7.06 | 2.4 | 7.15 | 6.0 |
| IEU-BB | 2.27 | 2.2 | 1.86 | 3.5 | 2.00 | 2.5 | 2.23 | 6.3 |
| IEU-Gauss | 4.84 | 2.1 | 5.05 | 4.1 | 4.63 | 2.7 | 4.74 | 6.5 |
| IEU-VortexSheet | 2.04 | 2.2 | 2.13 | 3.5 | 1.66 | 2.6 | 1.68 | 5.9 |
PDE Operator Learning
Partial differential equation (PDE) operator learning builds models that map inputs such as coefficients and initial conditions to solution fields.
See all 250 papers