Learning based methods are now ubiquitous for solving inverse problems, but their deployment in real-world applications is often hindered by the lack of ground truth references for training. Recent self-supervised learning strategies offer a promising alternative, avoiding the need for ground truth. However, most existing methods are limited to linear inverse problems. This work extends self-supervised learning to the non-linear problem of recovering audio and images from clipped measurements, by assuming that the signal distribution is approximately invariant to changes in amplitude. We provide sufficient conditions for learning to reconstruct from saturated signals alone and a self-supervised loss that can be used to train reconstruction networks. Experiments on both audio and image data show that the proposed approach is almost as effective as fully supervised approaches, despite relying solely on clipped measurements for training.
Figures & tables
Figure 1: Example in 3 dimensions illustrating Theorem 1 . The black set presents a particularly challenging scenario in which all saturated signals are projected to the same point. In contrast, with high probability on A , the colored set enables recovery of more signals with moderate norms, as points in X beyond a certain radius— which is connected to the signal norm when A is Gaussian—are all projected onto a single corner, showing the non-injectivity beyond this radius.
Figure 2: LNMC compared to LMC . On the left: Curve of the LNMC in one dimension for two examples in both cases saturated and not. On the right: Curve of the new LMC for the same two examples.
Figure 3: An example where the reconstruction is learned through the bias (third column) which prevents the reconstruction for low amplitude measurements. We thus observe that removing the bias makes the network naturally homogeneous and improves the reconstruction. On the first row, we consider a signal x , on the second row 10x .
Figure 4: An image with its associated blending mask. The mask is white where a channel is saturated and black when none is saturated.
Figure 5: Example where two different signals x1,x2 have the same measurement η(x1),η(x2) and so the network fθ fails to recover them both (the second and fourth columns). Adding randomness implies that the two measurements are not equal anymore, and so a network fθ′ can reconstruct the original signal (the third and fifth columns).
Figure 6: Plot of the dynamic range of fθ(y) depending on dynamic range of x , where y=η(Ax) . Left: The training is done with the losses LNMC+LEI . Right: The training is done with loss LMC+LEI . A perfect plot should be the identity line.
Figure 7: Average reconstruction performance as a function of saturated part v and signal set dimension k for both supervised (left) and self-supervised (right) methods. The value indicated corresponds to the mean SDR over the test dataset.
Methods
SDR ↑
PESQ ↓
Identity
4.84±2.12
2.7±0.65
Social Sparsity
9.92±4.46
2.09±0.95
Supervised
11.69±2.25
1.94±0.75
Supervised 5%
9.79±1.59
2.28±0.78
Supervised + LEI
11.72±2.23
1.92±0.74
Proposed self-supervised
10.48±2.20
2.20±0.80
Table 1: Reconstruction performance, SDR and PESQ are averaged over all music test dataset.
Methods
SDR (dB)
Identity
6.54±2.34
Supervised (trained on music)
10.94±2.00
Proposed self-supervised (trained on music and voice)
11.92±2.46
Table 2: Average SDR performance on the test dataset, which includes both music and voice recordings.
Methods
SDR ↑
PESQ ↓
Identity
5.00±0
2.9±0.73
Supervised
6.70±1.69
2.22±0.52
Proposed self-supervised
7.78±8.64
2.47±1.12
Table 3: Average SDR and PESQ performance on the test MAESTRO dataset.
Figure 8: Image reconstruction on the test dataset. An exposition correction ( Wikipedia, ) is applied to see details better.
Methods
PSNR ↑
NIQE ↓
Identity
29.54±6.95
4.21±1.96
PnP DPIR
32.11±5.80
4.06±1.93
Supervised
36.67±6.19
4.04±1.74
Proposed self-supervised
35.04±6.33
4.10±1.85
Table 4: Reconstruction performance, PSNR is averaged over all test dataset images.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Section
λ
pg
Architecture
Learning rate
Epochs
Batch size
Section 5.2.1
1
U(0.1,2)
Bias-free U-Net, 4 down + 4 up blocks
5e−4
300
50
Section 5.2.2
1
U(0.5,1.5)
MLP, input size 100, hidden size 100, depth 5
1e−4
300
100
Section 5.3
0.1
U(0.1,2.0)
Bias-free U-Net, 5 down + 5 up blocks
5e−4
100
35
Section 5.4
0.1
U(0.2,1.5)
Bias-free U-Net, 4 down + 4 up blocks
5e−5
400
12
Appendix
Table 5: Hyperparameters for different experiments.
Figure 9: SDR performance on the evaluation dataset during training for supervised and self-supervised methods.
Figure 10: Image reconstruction on the test dataset. Artifacts can appear in saturation zone.
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Self-supervised learning for imaging inverse problems is increasingly important in photon-limited settings, where acquiring clean ground truth is impractical and reconstruction must remain stable under dataset and acquisition shifts. This challenge is amplified under Poisson noise, whose signal-dependent statistics interact with sampling operators (e.g., CFA mosaicing). Meanwhile, foundation vision encoders trained at web scale offer distortion-invariant, content-related representations that generalize well across domains, suggesting a promising route to build priors that transfer beyond the training distribution without expensive fine-tuning. This paper proposes an ADMM-inspired unrolled plug-and-play solver for Poisson inverse problems that decouples a closed-form data-consistency update from a parameter-efficient prior. The prior is implemented as a lightweight decoder operating on frozen CLIP RN50 dense multi-scale features, adapting foundation representations with less trainable parameters. For self-supervision, the method integrates GR2R measurement-domain re-corruption with an Equivariant Imaging regularizer via virtual acquisitions. Experiments on Poisson CFA demosaicing and deblurring show competitive quality, improved robustness under shifts, and self-supervised performance approaching supervised training.
Laura C. Diaz-Delgado, Emmanuel Martinez, Henry Arguello
Department of Computer Science, Universidad Industrial de Santander, Colombia
Self-supervised image denoising methods have traditionally relied on architectural constraints, pseudo-pair constructions, or specialized loss functions to avoid the trivial identity mapping. Among these, approaches such as Noisier2Noise or R2R create training pairs by adding synthetic noise to noisy images. While effective, these recorruption-based approaches require precise knowledge of the noise distribution, which is often unavailable. We present Learning to Recorrupt (L2R), a self-supervised framework that does not require exact knowledge of the noise distribution. Our method introduces a learnable recorruptor jointly optimized with the denoiser through a min--max saddle-point objective. The proposed method achieves state-of-the-art performance among methods without prior knowledge of the noise distribution across unconventional and heavy-tailed noise distributions, such as log-gamma and Laplace, as well as spatially correlated noise, while obtaining competitive results on real-world denoising benchmarks characterized by signal-dependent noise.
Brayan Monroy, Jorge Bacca, Julián Tachella
Universidad Industrial de Santander, Colombia · CNRS, ENS de Lyon, France