We present a hypernetwork that generates the parameters of an implicit neural representation (INR) for a given signal in a single forward pass, without test-time optimization. Existing hypernetworks for this problem either ignore the chain-rule structure of the test-time optimization that produces an INR or, when they preserve it, instantiate the structure with MLPs whose parameter count grows quartically in target-network width. Our central contribution is to show that this parameter-count cost is structural to the MLP instantiation, not to the optimization-oriented principle of preserving chain-rule structure, and to resolve it by replacing the MLP body with attention. The resulting hypernetwork combines three design choices: an output-neuron-centric tokenization that aligns the token axis with each layer's output-neuron axis; an information-flow consistency rule that fixes the role of every cross-attention module from the representation its output should occupy; and a token-space iterative rollout that realizes the five quantities one optimization step on the target INR computes as five cross-attention modules with parameter count independent of target-network width. Across five 2D and 3D INR generation benchmarks, our method outperforms the state-of-the-art single-pass and optimization-oriented baselines; scales to target-network widths at which the MLP-based optimization-oriented paradigm exhausts GPU memory; and produces the highest result on NeRF generation without ground-truth INR weights supervision.
Figures & tables
Figure 1: One iterative module of the proposed hypernetwork HΩ , illustrated for a three-layer MLP target INR ( L=3 ); the construction extends to arbitrary L . At iteration t , the forward chain (left, modules ϕi ) produces the pseudo-activation tokens τit from the parameter tokens ωi(t) , seeded by τ0t from zζ through the Cross-Attention (CA) block. The backward block (right) realizes one chain-rule step at each layer: ψiτ and ψiω produce the local Jacobian tokens Ji−1τ,t and Jiω,t , which ψi−1δ and ψi∇ contract with δit to produce the propagated loss-signal δi−1t and the parameter-gradient tokens ∇ωiL ; the backward recursion is seeded by δLt , formed from τLt and zζ via the concat-plus-CA module. The ADD block then applies the refinement with predicted step size and forwards to the next module.
Figure 2: Two tokenization schemes for the layer- i weight matrix Wi . Left (ours): the output-neuron-centric scheme groups each row of Wi into one token ωi,j , placing each output neuron’s parameters into a single, co-adapted unit. Right: the input-neuron-centric scheme groups each column of Wi into one token, collecting parameters that originate from one input neuron to multiple output neurons that may encode unrelated features and are therefore not co-adapted.
Dataset
Metric
VAMoH
HyPoGen
LDMI
Ours
CelebA-HQ
PSNR ↑
21.3
16.2
24.8
27.7
SSIM ↑
0.6303
0.4550
0.6947
0.8635
LPIPS ↓
0.1332
0.1463
0.0556
0.0296
ERA5
PSNR ↑
39.0
40.0
44.6
49.3
SSIM ↑
0.8090
0.9947
0.9952
0.9991
LPIPS ↓
0.1663
0.0008
0.0003
0.0001
Table 1: Quantitative comparison on 2D INR reconstruction on CelebA-HQ, ERA5, and ImageNet.
Figure 3: Visual comparison of generated INRs across four datasets. Left: 2D INR generation on (a) CelebA-HQ and 3D shape reconstruction on (b) ShapeNet Chairs. Right: 2D INR generation on (a) ImageNet and novel view synthesis on (b) Pre-trained NeRF. Columns show HyPoGen, LDMI, Ours, and the ground truth (GT). Qualitative results on ERA5, as well as additional samples for all four datasets shown here, are provided in Appendix 6.8.5 .
Dataset
Metric
VAMoH
HyPoGen
LDMI
Ours
ShapeNet Chairs
Accuracy ↑
96.8%
95.2%
97.3%
97.9%
PSNR ↑
15.6
19.0
22.4
23.8
Pre-trained NeRF
Metric
HyperDiff.
HyPoGen
LDMI
Ours
Weights only
PSNR ↑
20.0
19.9
19.3
22.0
Images only
PSNR ↑
N/A
24.9
26.2
28.8
Weights & Images
PSNR ↑
17.5
25.5
26.5
27.8
Table 2: Quantitative comparison on 3D INR generation. 3D shape reconstruction on ShapeNet Chairs (top) and novel view synthesis on Pre-trained NeRF (bottom) under three supervision settings. “HyperDiff.” denotes HyperDiffusion.
Params (M)
GPU Memory (GB)
Infer Time @ b=16 (ms)
Dataset
HyPoGen
LDMI
Ours
HyPoGen
LDMI
Ours
HyPoGen
LDMI
Ours
CelebA-HQ
128.8
40.5
41.2
0.72
0.35
0.42
6.6
17.2
24.5
ImageNet
578.7
544.9
124.5
8.20
8.12
5.51
66.8
84.5
58.5
ERA5
93.1
48.9
21.3
0.59
0.51
0.33
21.2
25.0
24.6
Shapes3D
8.3
30.9
4.1
0.82
0.79
0.80
4.7
8.3
10.2
Pretrained NeRF
1049.9
515.5
482.5
14.51
12.47
2.51
2382
2541
2018
Table 3: Computational efficiency evaluated at batchsize 16 on each dataset.
Variant
CelebA-HQ
ImageNet
NeRF
Ours (Full)
27.7
22.9
27.8
Column-wise Tokenization
25.3
19.8
25.1
Reversed Q-KV
23.5
18.3
22.9
Table 4: Ablations on the two architectural principles of Sec. 3 . PSNR ( ↑ ) on CelebA-HQ, ImageNet, and Pre-trained NeRF ( Weights & Images ).
Figure 4: Empirical scalability analysis. (a) Performance versus training-data ratio on NeRF generation, testing how accuracy improves with data scale. (b) Parameter count and PSNR versus target-INR width on CelebA-HQ at batch size 64, testing the parameter-count prediction of Eq. 16 : HyPoGen’s parameter count grows rapidly while ours grows mildly with much more gains in PSNR.
Original dimensions
Under compression
HyPoGen (MLP body)
O(ni2⋅ni−12)
O(ni2⋅d2)
Ours (attention body)
O(ni−12)
O(d2)
Structural difference
factor of ni2
factor of ni2
Table 5: Parameter count of the core iterative body per target-network layer per refinement step, for HyPoGen’s MLP-based core and our attention-based core, in the original target-network dimensions and under row-level compression to embedding dimension d . Encoders and decoders are shared between the two architectures and excluded from the count.
Standard layer
Cross-attention module ϕi
Linear mixing
Wihi−1
ωiτi−1⊤/d
Non-linearity
activation σ
row-wise softmax
Output-unit alignment
j -th row →j -th neuron
j -th query →j -th output token
Table 6: Component-by-component correspondence between a standard INR layer (Eq. 3 ) and the cross-attention module ϕi (Eq. 6 ) that realizes the forward block.
Module
Surrogate of
Output representation
V=K
Q
ψiτ
∂hi/∂hi−1
layer- (i−1) activation
τi−1
ωi
ψiω
∂hi/∂Wi
layer- i parameter
ωi
τi−1
ψi−1δ
δi⋅∂hi/∂hi−1
layer- (i−1) activation
Ji−1τ
δi
ψi∇
δi⋅∂hi/∂Wi
layer- i parameter
Jiω
δi
Table 7: QKV assignments of the four backward modules, derived from information-flow consistency given the output representation each module’s target-network counterpart lives in.
Target-INR width
128
384
512
768
1024
LDMI (original)
22.1
–
–
25.3
–
LDMI + Adaptive Scaling
20.4
22.8
25.2
25.3
26.1
Table 8: Width-scaling stability of LDMI with and without our adaptive scaling sub-module on CelebA-HQ at batch size 64, with SIREN as the target INR. PSNR (dB) is reported; “–” marks runs that diverge at iteration zero because the decoded Wi0 exits the SIREN-safe regime. The original LDMI fails on three of the five widths in the sweep, while LDMI augmented with SW(i) from Eq. 23 trains across the full range.
CelebA-HQ 64 × 64
ImageNet
ERA5
Chairs
Pre-trained NeRF
Learning rate
5e-5
5e-5
5e-5
5e-5
5e-5
Optimizer
Adam
Adam
Adam
Adam
Adam
Mini-batch size
64
128
128
32
96
Task Tokenizer dim
128
256
128
64
768
Each Block Hidden dim
128
768
128
64
768
Transformer layers
1
1
1
1
1
Table 9: Model hyperparameter across different datasets.
Dataset
Encoder
CelebA-HQ
2D Conv VAE
ERA5
2D Conv VAE
ImageNet
VQ-VAE (pre-trained, frozen)
ShapeNet Chairs
3D Conv VAE
Pre-trained NeRF
DINO ViT-B/16 + adaLN-Zero pose cond.
Table 10: Encoder used for each dataset.
Figure 5: Additional visual comparison on CelebA-HQ.
Figure 6: Additional visual comparison on ERA5.
Figure 7: Additional visual comparison on ImageNet.
Figure 8: Additional visual comparison on ShapeNet Chairs.
Figure 9: Additional visual comparison on Pre-trained NeRF.
Supervision
Metric
Trans-INR
GINR-IPC
Ours
Weights only
PSNR ↑
16.5
17.1
22.0
Images only
PSNR ↑
21.9
26.1
28.8
Weights & Images
PSNR ↑
16.6
18.2
27.8
SSIM ↑
0.8659
0.8899
0.948
LPIPS ↓
0.2618
0.1855
0.0742
Table 11: Additional NeRF-specific baselines on the Pre-trained NeRF dataset under three supervision settings. Our method outperforms both Trans-INR and GINR-IPC across all settings.
CelebA-HQ
ImageNet
NeRF
Variant
Params
PSNR
Params
PSNR
Params
PSNR
Full model
41.2
27.7
124.5
22.9
482.5
27.8
w/o Forward module
33.1
24.9
101.2
18.5
441.9
24.6
w/o Backward module
38.4
24.4
110.1
18.1
456.8
24.2
Table 12: Ablations on the structural blocks of the iterative module of Sec. 3.4 . Each variant disables one structural block—the forward chain, the role-specific backward decomposition, while keeping all other components and total compute matched to the full model. PSNR ( ↑ ) on CelebA-HQ, ImageNet, and Pre-trained NeRF ( Weights & Images ).
Variant
CelebA-HQ
ImageNet
NeRF
Ours (Full, T=3 )
27.7
22.9
27.8
Single Layer Optimization ( T=1 )
27.4
22.1
26.7
Table 13: Ablations on the structural blocks of the iterative module of Sec. 3.4 . PSNR ( ↑ ) on CelebA-HQ, ImageNet, and Pre-trained NeRF ( Weights & Images ).
Dataset
Seed 444
Seed 555
Seed 777
Seed 888
Seed 999
Mean ± Std
Pre-trained NeRF
27.7
27.8
27.7
27.8
27.8
27.76 ± 0.05
CelebA-HQ
27.8
27.7
27.7
27.5
27.6
27.66 ± 0.10
ERA5
48.9
49.3
49.4
49.2
49.5
49.26 ± 0.23
Table 14: PSNR stability across random seeds on multiple datasets. Each row reports our method’s PSNR under five independent training runs that differ only in the random seed used for initialization, together with the mean and standard deviation. NeRF results are reported under the Weights & Images supervision setting.
ImageNet
Pre-trained NeRF
Metric
Token Comp.
Layer Fus.
Ours
Token Comp.
Layer Fus.
Ours
PSNR (dB)
22.7
22.6
22.9
29.0 †
29.2 †
28.8
Infer Time @ b=16 (ms)
54.1
41.2
58.5
2098
2135
2018
Params (M)
132.82
99.43
124.5
484.39
491.43
482.5
Table 15: Efficiency comparison (batch size: 16) under two practical engineering strategies for scaling to wider or deeper target networks. NeRF results are reported under the Images only supervision setting.
Neural surrogates based on implicit neural representations (INRs) are typically trained on mini-batches spread over the whole domain, since a generic INR loses accuracy when every point of a batch lies in one small region. We show that low-rank neural representations (LRNRs), INRs whose weight matrices are products of shared bases with coefficients emitted by a small hypernetwork, can be trained with spatially local mini-batches. Each step uses the points of one time snapshot of an adaptive-mesh numerical solution inside one small cell, with snapshots and cells drawn according to the refinement of the mesh. We posit that such training is enabled by the structural property of LRNRs, as a local step reaches the shared bases only through the r coefficients of each layer, and keeps one cell from rewriting the network. By a Kolmogorov-width argument, we show that when restricted to a small cell, the hidden states of LRNRs lie close to a low-dimensional subspace, which suggests the local steps only affect low-rank subspaces of the shared bases. On six hyperbolic-conservation-law problems with solutions computed using adaptive mesh refinement, one local training recipe reaches 1.1-2.3% relative L2 error with a tenth of the parameters of a dense network of the same width. Finally, we provide a comparison by training a hypernetwork-SIREN, a DeepONet, and a Fourier neural operator (FNO) of the same size with the same local batches and training budget. We observe that these baselines reach 16-78% where the LRNR reaches 4.9%. Interestingly, we also observe that local training has a significant effect on the per-cell low-dimensional structure even in a wide generic hypernetwork-SIREN.
Implicit neural representations (INRs) have emerged as powerful tools for encoding signals, yet dominant MLP-based designs often suffer from slow convergence, and how their accuracy depends on their size is known only empirically. We introduce FUTON (Fourier Tensor Network), which models signals as generalized Fourier series whose coefficients are parameterized by a low-rank tensor decomposition, read out by a linear or MLP head. FUTON combines complementary inductive biases: Fourier bases capture smoothness and periodicity, while the low-rank parameterization enforces low-dimensional spectral structure. Beyond universal approximation, we show that with a linear head, the best FUTON's error splits exactly into the truncation of the signal's series and the best low-rank error of its coefficients, which we bracket from the samples before training, with a FUTON that attains the upper bound. On image fitting, occupancy volumes, and novel view synthesis, against twelve INRs of matched size, FUTON achieves the best average quality, reaches the final quality of each task's strongest baseline 4-34x sooner than that baseline, infers fastest, and trains in the least memory.
Implicit Neural Representations (INRs) have been proven successful in encoding continuous signals through coordinate-based networks, yet facing a spectral dilemma: periodic activations capture fine details but act as all-pass filters that memorise noise, while spatially compact activations regularise effectively but suffer from low-frequency bias. Existing attempts to resolve this trade-off introduce computational overhead or tuning frailty. We propose to model each neuron's activation as the steady-state response of a sinusoidally-forced damped harmonic oscillator, whose amplitude naturally governs the network's spectral selectivity during training. By jointly optimising the oscillator parameters alongside the network weights, our method adapts to the target signal's spectral content without explicit regularisation. Initialised in the stopband, the network exhibits a coarse-to-fine learning curriculum that progressively expands its spectral gate, capturing low-frequency structures first and high-frequency details only when justified by the reconstruction objective. Comprehensive experiments show that our approach consistently achieves state-of-the-art or competitive results against established INRs, while requiring no task-specific tuning of any hyperparameters.
Alex Costanzino, Pierluigi Zama Ramirez, Giuseppe Lisanti +1
CVLab, University of Bologna · Ca’ Foscari University of Venice