Neural surrogates based on implicit neural representations (INRs) are typically trained on mini-batches spread over the whole domain, since a generic INR loses accuracy when every point of a batch lies in one small region. We show that low-rank neural representations (LRNRs), INRs whose weight matrices are products of shared bases with coefficients emitted by a small hypernetwork, can be trained with spatially local mini-batches. Each step uses the points of one time snapshot of an adaptive-mesh numerical solution inside one small cell, with snapshots and cells drawn according to the refinement of the mesh. We posit that such training is enabled by the structural property of LRNRs, as a local step reaches the shared bases only through the r coefficients of each layer, and keeps one cell from rewriting the network. By a Kolmogorov-width argument, we show that when restricted to a small cell, the hidden states of LRNRs lie close to a low-dimensional subspace, which suggests the local steps only affect low-rank subspaces of the shared bases. On six hyperbolic-conservation-law problems with solutions computed using adaptive mesh refinement, one local training recipe reaches 1.1-2.3% relative L2 error with a tenth of the parameters of a dense network of the same width. Finally, we provide a comparison by training a hypernetwork-SIREN, a DeepONet, and a Fourier neural operator (FNO) of the same size with the same local batches and training budget. We observe that these baselines reach 16-78% where the LRNR reaches 4.9%. Interestingly, we also observe that local training has a significant effect on the per-cell low-dimensional structure even in a wide generic hypernetwork-SIREN.
Figures & tables
Figure 1. Top: the sampled points (crosses) of a uniform mini-batch spread evenly over the domain and ignore the adaptive mesh (gray dots, left), while a spatially local mini-batch takes all the points of one time snapshot inside one cell, here k=13 ; each of the 25 cells has its own projector P1,...,P25 (right). Below each, one layer of the computation: the plain layer on the uniform batch, and on the local batch the plain layer together with a twin pass that projects the layer’s output with PkPk⊤ after the activation, the loss asking the two outputs to agree. Bottom: locality of the representation, from finite elements, whose basis functions have local support, through the LRNR, whose hidden states are close to a few directions within each cell, to a generic INR, in which every weight acts on the whole domain; in each panel the highlighted curves are basis functions that are active in the shaded cell.
Figure 2. The local spatial sampling, on frame 64 ( t=0.64 ) of the Euler quadrants. Top left: the solver’s adaptive points (171,000 here, colored by density) under the two-level tiling: 25 coarse cells (thick) and 100 fine cells (thin) on [0,1]2 , each level translated by its own random offset of up to a quarter cell, redrawn every time the tiling is applied. Top right: the number of solver points in each fine cell (gray, log scale) is the sampler’s score; the cells at or above its 0.8 quantile with at least 100 points (red, 20 of 100) are used in the draw list ten extra times. Bottom left: one step is one such draw, the points of one frame inside one cell (here 4,100 points at the shock–contact junction), on which one gradient step is taken with that cell’s projector resident. Bottom right: the frame is drawn with probability proportional to Nk (blue; 49,000 to 236,000 points per frame) and, in the final stages, five times more often within five frames of either end (red).
Table 2. The six problems and the area-weighted relative L2 error of the trained models on the training data, at the solver’s points (rms over the times t0,t20,t40,t60,t80 ). The pre-boost column is the floor of each ladder before the end frames were over-sampled; the Euler ladder was trained without that over-sampling, so it has no pre-boost value.
Figure 3. Relative L2 error by time of the six final models, at every fourth frame: all variables together (black) and, for the problems with several variables, each variable. A variable’s error is left out at the times where the variable itself is (nearly) zero: the radial momentum before the shock reaches the bubble, the inclusions’ fields at t=0 and their y -velocity before the pulse reaches the discs.
Problem
81 training times
80 halfway times
Euler quadrants
0.8%
0.8%
Periodic acoustics
1.8%
1.8%
Swirling advection
1.3%
1.6%
Periodic Burgers
1.4%
1.8%
Shock–bubble
1.4%
1.4%
Acoustics with inclusions
2.1%
2.1%
Table 3. Relative L2 error of the final models (area-weighted, root mean square over the frames) at the 81 training times and at the 80 times halfway between them. The halfway solutions were computed by restarting the solver from the training run at the preceding training time and advancing it by half an output interval.
Figure 4. Euler quadrants, density: solver (top row), LRNR (middle) and their difference (bottom) at four times on the full domain. The dashed square is the window of Fig. 5 .
Figure 5. As Fig. 4 , inside the Kelvin–Helmholtz window [0.25,0.62]×[0.38,0.75] .
Figure 6. Acoustics with inclusions, pressure: as Fig. 4 , from t=0.27 (the field is zero until the pulse arrives).
Figure 7. As Fig. 6 , inside the window on the slow, dense disc and its surroundings, [0.25,0.65]×[0.1,0.5] , which the pulse has barely reached at t=0.27 (hence the large relative error of that panel on a near-zero field).
Figure 8. Per layer, the ranked contribution of the 50 coefficients (top) and the time courses sℓ,i(t) of the eight leading coefficients (bottom), for the Euler and inclusions models (acoustics and advection in Fig. 24 , Burgers and shock–bubble in Fig. 25 , Appendix E ).
Figure 9. Hyper-modes of the six models, one row each: the singular values of the coefficient trajectory (left), the time courses of the four leading modes (middle, told apart by line style) and the distribution of each mode’s energy over the layers (right, told apart by hatching).
Figure 10. Euler quadrants, density, perturbed along the hyper-modes at four times. Rows: the model u(t) ; the field at s(t)−0.02σ1v1 and at s(t)+0.02σ1v1 (the title gives the time t′ on the model’s own trajectory nearest to the perturbed field); the field at s(t)+0.02σ2v2 and the change it makes, as a fraction of ∥u∥ .
Figure 11. The width ladder of the most complicated cell of each problem: the leaf’s error against the solver’s points in the cell, its agreement with the full model, and the model’s own error (dashed), relative L2 pooled over all times, against the neurons per layer. The leaves themselves, their difference from the model and the errors over time are in Figs. 34 – 36 (Appendix E ).
Figure 12. Singular values of the hidden states over (t,x) per layer for the three baselines (normalized to the largest). In the LRNR the pre-activations of every layer lie in an r=50 -dimensional space by construction; its hidden states are compared in Table 13 .
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13. Training histories over the whole ladders (Euler and acoustics; advection and Burgers in Fig. 14 , shock–bubble and inclusions in Fig. 15 ): the sampled-cell misfit (gray raw, blue smoothed) and the projection misfit (orange), and the three regularizers (unweighted); dashed lines mark the stage boundaries. The learning rate follows the stage ladders above: cosine stages decay from 10−4 , constant-rate stages run at 10−5 .
Figure 14. Training histories of the advection and Burgers ladders (as Fig. 13 ).
Figure 15. Training histories of the shock–bubble and inclusions ladders (as Fig. 13 ).
Figure 16. Periodic acoustics, pressure: as Fig. 4 .
Figure 17. As Fig. 16 , inside the central window [−0.5,0.5]2 where the periodic images of the ring cross.
Figure 18. Swirling advection: as Fig. 4 .
Figure 19. As Fig. 18 , inside the 0.4 -wide square holding the most solver points at the final time, the swirl center.
Figure 20. Periodic Burgers: as Fig. 4 .
Figure 21. As Fig. 20 , inside the 0.4 -wide square holding the most solver points at the final time, on the leading shock.
Figure 22. Shock–bubble interaction, density, drawn at the physical aspect of the tube [0,2]×[0,0.5] (the symmetry axis at the bottom): solver, LRNR and their difference (columns) at four times (rows) on the full domain; the dashed rectangle is the window of Fig. 23 .
Figure 23. As Fig. 22 , inside the window [0.6,1.4]×[0,0.25] that the vortex ring crosses.
Figure 24. As Fig. 8 , for the acoustics and advection models.
Figure 25. As Fig. 8 , for the Burgers and shock–bubble models.
Figure 26. As Fig. 10 , inside the Kelvin–Helmholtz window.
Figure 27. Periodic acoustics, pressure, perturbed along the hyper-modes (layout of Fig. 10 ).
Figure 28. Swirling advection, perturbed along the hyper-modes (layout of Fig. 10 ).
Figure 29. As Fig. 28 , inside the swirl-center window.
Figure 30. Periodic Burgers, perturbed along the hyper-modes (layout of Fig. 10 ).
Figure 31. Shock–bubble interaction, density, perturbed along the hyper-modes at the first two times, drawn at the physical aspect of the tube. Each time is a column holding the five rows of Fig. 10 : the model; −0.02σ1v1 and +0.02σ1v1 with the nearest trajectory time t′ ; +0.02σ2v2 ; and the change it makes.
Figure 32. As Fig. 31 , at the last two times.
Figure 33. Acoustics with inclusions, pressure, perturbed along the hyper-modes (layout of Fig. 10 ).
Figure 34. The most complicated cell of the Euler and acoustics problems: the full model, the 300-neuron FastLRNR leaf and their difference at the final time (top row of each), and the error against the solver and the agreement with the model over time for every width of the ladder, with the ladder pooled over time (bottom row of each).
Figure 35. As Fig. 34 , for the advection and Burgers cells.
Figure 36. As Fig. 34 , for the shock–bubble and inclusions cells; the per-time panels of the inclusions cell omit the times at which it is empty.
Training implicit neural representations (INRs) to capture fine-scale details typically relies on iterative backpropagation and is often hindered by spectral bias when the target exhibits highly non-uniform frequency content. We propose ELM-INR, a backpropagation-free INR that decomposes the domain into overlapping subdomains and fits each local problem using an Extreme Learning Machine (ELM) in closed form, replacing iterative optimization with stable linear least-squares solutions. This design yields fast and numerically robust reconstruction by combining local predictors through a partition of unity. To understand where approximation becomes difficult under fixed local capacity, we analyze the method from a spectral Barron norm perspective, which reveals that global reconstruction error is dominated by regions with high spectral complexity. Building on this insight, we introduce BEAM, an adaptive mesh refinement strategy that balances spectral complexity across subdomains to improve reconstruction quality in capacity-constrained regimes.
Implicit neural representations (INRs) offer a powerful framework for modeling high-dimensional physical fields, but their training remains inefficient because evaluating dense coordinate grids is computationally expensive. Existing coordinate-pruning strategies rely on local error functions that can oversample correlated points and incur substantial overhead from full-grid error evaluation. We introduce Refined Grid Sampling (RGS), which combines stratified grid sampling, error-based refinement, and dynamic grid-resolution adjustment. This design reduces spatial redundancy without evaluating errors over the full grid. On high-resolution Navier--Stokes fields, RGS achieves higher reconstruction accuracy within a fixed wall-clock budget and reaches the accuracy of full-grid training up to 6× faster than the evaluated baselines.
Guang Zhao, Xihaier Luo, Sean McCue +5
Brookhaven National Laboratory Upton, NY 11973, USA · University of California San Diego La Jolla, CA 92093, USA · Los Alamos National Laboratory Los Alamos, NM 87545, USA
Implicit Neural Representations (INRs) have emerged as promising surrogates for large 3D scientific simulations due to their ability to continuously model spatial and conditional fields, yet they face a critical fidelity-speed dilemma: deep MLPs suffer from high inference cost, while efficient embedding-based models lack sufficient expressiveness. To resolve this, we propose the Decoupled Representation Refinement (DRR) architectural paradigm. DRR leverages a deep refiner network, alongside non-parametric transformations, in a one-time offline process to encode rich representations into a compact and efficient embedding structure. This approach decouples slow neural networks with high representational capacity from the fast inference path. We introduce DRR-Net, a simple network that validates this paradigm, and a novel data augmentation strategy, Variational Pairs (VP) for improving INRs under complex tasks like high-dimensional surrogate modeling. Experiments on several ensemble simulation datasets demonstrate that our approach achieves state-of-the-art fidelity, while being up to 27× faster at inference than high-fidelity baselines and remaining competitive with the fastest models. The DRR paradigm offers an effective strategy for building powerful and practical neural field surrogates and INRs in broader applications, with a minimal compromise between speed and quality.
Tianyu Xiong, Skylar Wurster, Han-Wei Shen
Department of Computer Science and Engineering Ohio State University Columbus, OH 43210, USA · Adobe San Francisco, CA, USA