NOIR: Neural Operator mapping for Implicit Representations
Authors: Sidaty El Hadramy, Nazim Haouchine, Michael Wehrli, Philippe C. Cattin
Organizations: Department of Biomedical Engineering, University of Basel, Allschwil, Switzerland · Harvard Medical School, Brigham and Women’s Hospital, Boston, United States
This paper presents NOIR, a framework that reframes core medical imaging tasks as operator learning between continuous function spaces, challenging the prevailing paradigm of discrete grid-based deep learning. Instead of operating on fixed pixel or voxel grids, NOIR embeds discrete medical signals into shared Implicit Neural Representations and learns a Neural Operator that maps between their latent modulations, enabling resolution-independent function-to-function transformations. We evaluate NOIR across multiple 2D and 3D downstream tasks, including segmentation, shape completion, image-to-image translation, and image synthesis, on several public datasets such as Shenzhen, OASIS-4, SkullBreak, fastMRI, as well as an in-house clinical dataset. It achieves competitive performance at native resolution while demonstrating strong robustness to unseen discretizations, and empirically satisfies key theoretical properties of neural operators. The project page is available here: https://github.com/Sidaty1/NOIR-io.
Figures & tables
Figure 1: Downstream tasks solved by NOIR . By formulating each task as operator learning between function spaces, NOIR addresses segmentation, completion, synthesis, and translation across in 2D and 3D, with inherent robustness to resolution changes.
Figure 2: Architecture of NOIR . Discrete input ( fid ) and output ( gid ) signals are embedded as continuous functions ( fi∣X and gi∣X ) using INRs with shared dataset-level parameters and signal-specific modulations. A neural operator maps the modulations of the input ( zin ) and output ( zout ) INRs, resulting in a continuous function-to-function mapping applicable to multiple downstream tasks.
Figure 3
Dataset
Method
Res = 10%
Res = 25%
Res = 100%
DSC ↑
IoU ↑
DSC ↑
IoU ↑
DSC ↑
IoU ↑
Shenzhen [ 28 ]
U-Net [ 51 ]
0.75
0.62
0.92
0.87
0.95
0.91
ViT [ 8 ]
0.91
0.84
0.93
0.87
0.94
0.90
AttU-Net [ 46 ]
0.01
0.01
0.74
0.64
0.95
0.91
FNO [ 37 ]
0.92
0.86
0.94
0.88
0.94
0.89
NOIR (Ours)
0.93
0.88
0.94
0.88
0.94
0.88
Table 1: Performance of NOIR and baseline methods across segmentation and anatomy completion tasks.
Figure 3: Qualitative results for segmentation and anatomy completion. Rows correspond to Shenzhen binary segmentation, OASIS-4 multi-label segmentation, and SkullBreak anatomy completion, respectively. A randomly selected test sample is shown with the input image, ground truth, NOIR results, and comparison methods.
Figure 4: Qualitative results for image synthesis (top row) and image translation (bottom row). A randomly selected test sample is shown, including the input image, ground truth, NOIR results, and results from comparison methods.
Task
Method
Res = 25%
Res = 50%
Res = 100%
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
US Synthesis
U-Net [ 51 ]
17.51
0.84
17.45
0.84
17.54
0.89
ViT [ 8 ]
28.22
0.78
28.31
0.79
28.36
0.79
AttU-Net [ 46 ]
27.76
0.87
28.29
0.89
28.55
0.90
FNO [ 37 ]
27.37
0.73
27.94
0.75
28.15
0.76
DDPM [ 26 ]
26.35
0.65
26.51
0.65
26.91
0.64
Table 2: Image-to-image translation performance for domain adaptation and ultrasound synthesis. PSNR and SSIM are reported across different resolutions.
Figure 5: Resolution robustness of NOIR. (a) Dice distributions across discretizations show stable performance with mean Dice around 0.93 – 0.94 . (b) Qualitative example demonstrating consistent segmentation predictions across different input resolutions.
Dataset
Operator T
DSC ↑
IoU ↑
Shenzhen ( p=64 )
Linear Regression
0.93
0.87
SVR
0.94
0.88
Random Forest
0.93
0.86
MLP
0.94
0.88
OASIS-4 ( p=2048 )
Linear Regression
0.80
0.68
SVR
0.79
0.66
Table 3: Ablation study on NO model selection.
Dataset
In
Out
#HL
HS
HAct
LAct
Lat
#HHL
HHS
LRL
LR
Opt
Shenzhen
2
1
6
256
Sine
Sigmoid
64
1
64
10−2
5×10−6
AdamW
OASIS-4
2
1
6
256
Sine
Sigmoid
2048
1
64
10−2
5×10−6
AdamW
SkullBreak
3
1
10
256
Sine
Softmax
64
1
64
10−2
5×10−6
AdamW
fastMRI
2
1
10
256
Sine
Sigmoid
4096
1
64
10−2
5×10−6
AdamW
Ultrasound
2
1
10
256
Sine
Sigmoid
64
1
64
10−2
5×10−6
AdamW
Table S1 : Training configuration for input implicit neural representations (INRs).
Dataset
In
Out
#HL
HS
HAct
LAct
Lat
#HHL
HHS
LRL
LR
Opt
Shenzhen
2
2
6
256
Sine
Softmax
64
1
64
10−2
5×10−6
AdamW
OASIS-4
2
5
10
256
Sine
Softmax
2048
1
64
10−2
5×10−6
AdamW
SkullBreak
3
2
6
256
Sine
Softmax
64
1
64
10−2
5×10−6
AdamW
fastMRI
2
1
10
256
Sine
Sigmoid
4096
1
64
10−2
5×10−6
AdamW
Ultrasound
2
1
10
256
Sine
Sigmoid
64
1
64
10−2
5×10−6
AdamW
Table S2 : Training configuration for output implicit neural representations (INRs).
Dataset
In
Out
#HL
HD
Act
Res
LR
Opt
WD
Dropout
Shenzhen
64
64
1
128
SiLU
True
10−4
AdamW
10−4
0
OASIS-4
2048
2048
3
2048
SiLU
True
10−4
AdamW
10−4
0.2
SkullBreak
64
64
1
2048
SiLU
True
10−4
AdamW
10−4
0
fastMRI
4096
4096
1
2048
SiLU
True
5×10−5
AdamW
10−4
0.1
Ultrasound
64
64
3
512
ReLU
True
7×10−4
AdamW
10−5
0
Table S3 : Training configuration for the neural operator (NO) mapping input to output INRs.
Dataset
Image Size
Train
Val
Test
Num. Points
Shenzhen [ 28 ]
200×200
0.8
0.1
0.1
40000
OASIS-4 [ 42 ]
160×192
0.7
0.15
0.15
30720
SkullBreak [ 32 ]
512×512×512
From [ 32 ]
From [ 32 ]
From [ 32 ]
40000
fastMRI [ 31 ]
128×128
0.8
0.1
0.1
10000
Ultrasound (in-house)
64×64
0.8
0.1
0.1
1820
Table S4 : Dataset information including training, validation, and test splits, image sizes, and number of points/slices used for each experiment.
Input INR
Output INR
Dataset
PSNR ↑
SSIM ↑
DSC
IoU
PSNR ↑
SSIM ↑
DSC ↑
IoU ↑
Shenzhen [ 28 ]
22.72
0.66
–
–
–
–
0.98
0.97
OASIS-4 [ 42 ]
24.76
0.79
–
–
–
–
0.91
0.84
SkullBreak [ 32 ]
–
–
0.87
0.77
–
–
0.90
0.83
fastMRI [ 31 ]
22.26
0.65
–
–
29.09
0.71
–
–
Ultrasound
33.77
0.94
–
–
33.48
0.70
–
–
Table S5 : Reconstruction performance of the input and output INRs across datasets.
Figure S1 : Reconstruction results from the output INR for the Shenzhen dataset. Each example shows the ground truth image and the reconstruction produced by the output INR.
Figure S2 : Reconstruction results from the output INR for the OASIS-4 dataset. Ground truth segmentations are compared with the corresponding INR reconstructions.
Figure S3 : Reconstruction results from the output INR for the fastMRI dataset. The reconstructed images are compared with the ground truth MRI slices.
Figure S4 : Reconstruction results from the output INR for the ultrasound dataset, showing the ability of the INR to approximate the ultrasound signal from the learned latent representation.
Figure S5 : Confusion matrices summarizing pairwise discrepancies between latent modulation vectors obtained at resolutions 32×32 , 64×64 , 128×128 , 160×160 , and 200×200 . Left: discrepancies between optimized input modulation vectors. Right: discrepancies between predicted output modulation vectors produced by the neural operator.
Dataset
NOIR
U-Net
ViT
AttU-Net
FNO
DDPM
Shenzhen [ 28 ]
8.30×10−1
3.06×10−3
3.07×10−3
4.72×10−3
2.94×10−3
–
OASIS-4 [ 42 ]
2.41×10−1
2.69×10−3
5.75×10−3
3.57×10−3
2.63×10−3
–
SkullBreak [ 32 ]
3.45×10−1
2.13×10−1
1.06×10−1
2.38×10−1
1.23×10−1
–
fastMRI [ 31 ]
1.58
1.89×10−3
2.71×10−3
2.78×10−3
1.59×10−3
5.27
Ultrasound
5.93×10−1
3.00×10−3
4.04×10−3
4.69×10−3
1.62×10−3
2.72×10−1
Table S6 : Prediction time comparison (in seconds).
Implicit neural representations (INRs) have emerged as a powerful paradigm for medical imaging via physics-informed unsupervised learning. Classical INRs optimize an entire network from scratch for each subject, leading to inefficient training and suboptimal imaging quality. Recent initialization-based approaches attempt to inject population priors into pre-trained networks, yet they rely on high-quality images and often suffer from catastrophic forgetting during fine-tuning. We present DisINR, a novel INR framework that explicitly disentangles shared and subject-specific representations. DisINR introduces a shared encoder-decoder pair and subject-specific encoders, whose features are jointly decoded for image reconstruction. By integrating differentiable forward models, it pre-trains the shared modules directly from limited raw measurements, removing the need for pre-acquired high-quality images. During test-time adaptation, only the subject-specific encoder is optimized, while the shared pair remains frozen, effectively preserving learned priors. Extensive evaluations on three representative medical imaging tasks show that DisINR significantly outperforms state-of-the-art INRs in both reconstruction accuracy and efficiency.
Qing Wu, Xuanyu Tian, Chenhe Du +4
Medical AI Lab, Ant Group · ShanghaiTech University · Shanghai Jiao Tong University
Continuous parameterization of medical data has emerged as a powerful paradigm for resolution-independent image representation. While Implicit Neural Representations offer high fidelity and compact storage, their reliance on global Multi-Layer Perceptrons incurs sizeable computational costs, large memory requirements, and extensive optimization times. As medical imaging trends towards ever-more detailed, high-resolution volumes, these costs impose significant bottlenecks in the applicability of implicit approaches. Recently, explicit Gaussian-based primitives have revolutionized the representation learning paradigm by trading deep network evaluations for localized, rasterization-friendly primitives. In this paper, we present a comprehensive, cross-dimensional evaluation of Gaussian representations against implicit approaches for medical imaging applications. First, we outline a theoretical overview on the mathematical properties offered by explicit primitives beyond what is capable under the implicit neural paradigm. Subsequently, we benchmark the computational performance on two demanding image datasets: 2D microscopy histology and 3D lung Computed Tomography (CT). Our experiments demonstrate that Gaussian representations consistently match or surpass reconstruction metrics compared to implicit methods across all compression factors, while displaying significantly lower optimization times, and memory requirements. Together with the compelling mathematical properties offered by explicit primitives, these findings motivate the wider adoption of Gaussian representations and position them as an attractive direction for future research in medical imaging.
Nil Stolt-Ansó, Maik Dannecker, Wenqi Huang +2
Technical University Munich, Germany · University of Zurich, Switzerland · Imperial College London, United Kingdom
Neural operators (NOs) are designed to learn maps between infinite-dimensional function spaces. We propose a novel reframing of their use. By introducing an auxiliary base-space, any finite-dimensional function can be viewed as an operator acting by composition on functions of the base-space. Through a range of benchmarks on analytic functions of increasing complexity and dimensionality, we demonstrate that NOs can match or outperform standard multilayer perceptrons and Kolmogorov--Arnold Networks in accuracy while requiring significantly fewer parameters and training time. As a real-world application, we apply a two-dimensional Tensorized Fourier Neural Operator (TFNO) to the nuclear chart, learning a correction to state-of-the-art nuclear mass models as a partially observed residual field. A TFNO ensemble reaches a held-out root-mean-square error of 198.2 keV, placing it among the best recent neural-network approaches while retaining high parameter efficiency and short training times. More broadly, these results introduce NOs as a scalable framework for finite-dimensional function interpolation, from analytic benchmarks to structured scientific data.
Department of Physics, CCTP and ITCP, University of Crete, 71303, Greece · Theoretical Physics Department, CERN, Geneva, Switzerland · Physik-Institut, Universität Zürich, 8057 Zürich, Switzerland