Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems

Laura C. Diaz-Delgado; Emmanuel Martinez; Henry Arguello

eess.IVAug 20, 2026

Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems

Authors: Laura C. Diaz-Delgado, Emmanuel Martinez, Henry Arguello

Organizations: Department of Computer Science, Universidad Industrial de Santander, Colombia

Abstract

Self-supervised learning for imaging inverse problems is increasingly important in photon-limited settings, where acquiring clean ground truth is impractical and reconstruction must remain stable under dataset and acquisition shifts. This challenge is amplified under Poisson noise, whose signal-dependent statistics interact with sampling operators (e.g., CFA mosaicing). Meanwhile, foundation vision encoders trained at web scale offer distortion-invariant, content-related representations that generalize well across domains, suggesting a promising route to build priors that transfer beyond the training distribution without expensive fine-tuning. This paper proposes an ADMM-inspired unrolled plug-and-play solver for Poisson inverse problems that decouples a closed-form data-consistency update from a parameter-efficient prior. The prior is implemented as a lightweight decoder operating on frozen CLIP RN50 dense multi-scale features, adapting foundation representations with less trainable parameters. For self-supervision, the method integrates GR2R measurement-domain re-corruption with an Equivariant Imaging regularizer via virtual acquisitions. Experiments on Poisson CFA demosaicing and deblurring show competitive quality, improved robustness under shifts, and self-supervised performance approaching supervised training.

Figures & tables

Figure 1: Overview of the proposed method. The reconstruction network with $T$ iterations alternates between: (i) a data-consistency update $\mathbf{x}^{t}$ , and (ii) a CLIP-based prior update $\mathbf{z}^{t}$ . The prior consists of a frozen CLIP encoder coupled with a learnable decoder. After $T$ iterations, the updates yield the reconstruction $\hat{\mathbf{x}}$ .

Method		Poisson Noise
	$\gamma$	BSDS500		DIV2K
	$\gamma$	PSNR [dB]	SSIM	PSNR [dB]	SSIM
DPIR [ 26 ]	0.01	25.96 $\pm$ 1.98	0.7039 $\pm$ 0.0887	25.95 $\pm$ 2.82	0.7093 $\pm$ 0.1244
Transfer CLIP [ 5 ]		27.88 $\pm$ 3.18	0.7833 $\pm$ 0.1068	27.50 $\pm$ 4.44	0.7841 $\pm$ 0.1006
GSPnP [ 10 ]		29.25 $\pm$ 2.16	0.8061 $\pm$ 0.0647	29.36 $\pm$ 2.62	0.8237 $\pm$ 0.0605
RAM [ 23 ]		30.46 $\pm$ 2.07	0.8622 $\pm$ 0.0545	30.48 $\pm$ 2.51	0.8729 $\pm$ 0.0568

Table 1: Demosaicing performance under Poisson noise.

Figure 2: Qualitative comparison on BSDS500 for Poisson demosaicing with shot-noise level $\gamma=0.01$ . Columns show the mosaiced noisy Measurement , reconstructions by DPIR [ 26 ] , Transfer CLIP [ 24 , 5 ] , GSPnP [ 10 ] , and RAM [ 23 ] , followed by the proposed method (Ours (Self)) and its supervised counterpart (Ours (Sup)), and the Reference . PSNR/SSIM are overlaid for direct visual comparison.

Method		Poisson Noise
	$\gamma$	BSDS500		DIV2K
	$\gamma$	PSNR [dB]	SSIM	PSNR [dB]	SSIM
DPIR [ 26 ]	0.01	28.81 $\pm$ 2.60	0.7949 $\pm$ 0.0699	28.25 $\pm$ 3.66	0.7894 $\pm$ 0.0816
Transfer CLIP [ 5 ]		24.63 $\pm$ 2.73	0.7389 $\pm$ 0.0605	24.33 $\pm$ 3.75	0.7298 $\pm$ 0.0932
GSPnP [ 10 ]		29.37 $\pm$ 2.73	0.8149 $\pm$ 0.0701	28.90 $\pm$ 3.65	0.8143 $\pm$ 0.0785
RAM [ 23 ]		28.67 $\pm$ 2.57	0.8028 $\pm$ 0.0667	28.40 $\pm$ 3.35	0.8078 $\pm$ 0.0672

Table 2: Deblurring performance under Poisson noise.

Figure 3: Qualitative comparison on BSDS500 for Poisson deblurring with shot-noise level $\gamma=0.01$ . Columns show the blurred noisy Measurement , reconstructions by DPIR [ 26 ] , Transfer CLIP [ 24 , 5 ] , GSPnP [ 10 ] , and RAM [ 23 ] , followed by Ours (Self), Ours (Sup), and the Reference . Per-image PSNR/SSIM are overlaid for direct visual comparison.

Method	#Params [M]	TFLOPs	Time [s]
DPIR [ 26 ]	32.64	11.49	0.5193
Transfer CLIP [ 5 ]	10.99	9.00	2.4683
GSPnP [ 10 ]	17.01	9.12	0.9247
RAM [ 23 ]	34.13	0.32	0.8026
Ours	10.99	0.09	0.0242

Table 3: Runtime and complexity comparison (test on size

3\times 256\times 256).

Setup	Loss	PSNR [dB]	SSIM
Untrained CLIP (Baseline)	$\mathcal{L}_{\text{sup}}$	21.91	0.7064
+ Pretraining	$\mathcal{L}_{\text{MC}}$	21.42	0.5134
+ Pretraining	$\mathcal{L}_{\text{GR2R}}$	27.45	0.7992
+ Pretraining	EI [ 3 ]	28.41	0.7746
+ Pretraining	REI [ 4 ]	30.31	0.8600
DnCNN	$\mathcal{L}_{\text{self}}$	29.90	0.8474

Table 4: Ablation study on training objectives and pretraining strategies. This ablation evaluates different backbones for the unrolled algorithm and different cost functions to assess their effectiveness on Poisson Demosaicing with

\gamma=0.01

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Zero-shot self-supervised reconstruction on real photon-limited data from the SID dataset. A $512\times 512$ patch is reserved for testing, while the remaining patches from the same low-light image are used to train with $\mathcal{L}_{\mathrm{self}}$ . The Poisson scaling parameter is estimated per RGB channel, and an affine color transform is applied for sRGB visualization and metric evaluation. The result illustrates the adaptability of the proposed method to real photon-limited sensor statistics without clean-image supervision.

Method		Poisson Noise
	$\gamma$	Demosaicing		Deblurring
	$\gamma$	PSNR [dB]	SSIM	PSNR [dB]	SSIM
DPIR	0.02	26.75 $\pm$ 1.83	0.7463 $\pm$ 0.0813	27.89 $\pm$ 2.17	0.7738 $\pm$ 0.0684
Transfer CLIP		24.53 $\pm$ 2.91	0.6608 $\pm$ 0.1315	22.96 $\pm$ 2.33	0.6909 $\pm$ 0.0700
GSPnP		28.31 $\pm$ 1.76	0.7798 $\pm$ 0.0726	28.44 $\pm$ 2.22	0.7962 $\pm$ 0.0690
RAM		28.73 $\pm$ 1.94	0.8084 $\pm$ 0.0041	27.39 $\pm$ 2.45	0.7540 $\pm$ 0.0728

Appendix

Table 5: Additional BSDS500 results at intermediate Poisson noise levels.

Figure 5: Qualitative comparison on BSDS500 for Poisson demosaicing with shot-noise level $\gamma=0.05$ . Columns show the mosaiced noisy Measurement , reconstructions by DPIR, Transfer CLIP, GSPnP, and RAM, followed by the proposed method trained self-supervised (Ours (Self)) and its supervised counterpart (Ours (Sup)), and the Reference . Per-image PSNR/SSIM are overlaid for direct visual comparison.

Figure 6: Qualitative comparison on DIV2K for Poisson demosaicing with shot-noise level $\gamma=0.01$ . Columns show the mosaiced noisy Measurement , reconstructions by DPIR, Transfer CLIP, GSPnP, and RAM, followed by the proposed method trained self-supervised (Ours (Self)) and its supervised counterpart (Ours (Sup)), and the Reference . Per-image PSNR/SSIM are overlaid for direct visual comparison.

Figure 7: Qualitative comparison on DIV2K for Poisson demosaicing with shot-noise level $\gamma=0.05$ . Columns show the mosaiced noisy Measurement , reconstructions by DPIR, Transfer CLIP, GSPnP, and RAM, followed by the proposed method trained self-supervised (Ours (Self)) and its supervised counterpart (Ours (Sup)), and the Reference . Per-image PSNR/SSIM are overlaid for direct visual comparison.

Figure 8: Qualitative comparison on BSDS500 for Poisson deblurring with shot-noise level $\gamma=0.05$ . Columns show the blurred noisy Measurement , reconstructions by DPIR, Transfer CLIP, GSPnP, and RAM, followed by the proposed method trained self-supervised (Ours (Self)) and its supervised counterpart (Ours (Sup)), and the Reference . Per-image PSNR/SSIM are overlaid for direct visual comparison.

Figure 9: Qualitative comparison on DIV2K for Poisson deblurring with shot-noise level $\gamma=0.01$ . Columns show the blurred noisy Measurement , reconstructions by DPIR, Transfer CLIP, GSPnP, and RAM, followed by the proposed method trained self-supervised (Ours (Self)) and its supervised counterpart (Ours (Sup)), and the Reference . Per-image PSNR/SSIM are overlaid for direct visual comparison.

Figure 10: Qualitative comparison on DIV2K for Poisson deblurring with shot-noise level $\gamma=0.05$ . Columns show the blurred noisy Measurement , reconstructions by DPIR, Transfer CLIP, GSPnP, and RAM, followed by the proposed method trained self-supervised (Ours (Self)) and its supervised counterpart (Ours (Sup)), and the Reference . Per-image PSNR/SSIM are overlaid for direct visual comparison.

Inverse Problems

Inverse problems infer hidden causes, parameters, or signals from indirect, noisy measurements using physical models, priors, and statistical methods.

See all 245 papers

Learning to reconstruct from saturated data: audio declipping and high-dynamic range imaging
Feb 25, 2026Victor Sechaud, Laurent Jacques, Patrice Abry +1Inverse Problems Self-Supervised Learning
CNRS and ENS de Lyon, Laboratoire de physique, Lyon, France · UCLouvain, ICTEAM, Louvain-la-Neuve, Belgium · CIFAR Fellow
$\mathrm{S}^3$ -PnP: Plug-and-Play Denoising for Scientific Imaging with Self-Supervised Score Priors
Oct 13, 2026Yukun Cui, Jiangshe Zhang, Shuang Xu +6Image Restoration Image Denoising
Xi’an Jiaotong University · Northwestern Polytechnical University · Nanyang Technological University +3
Hallucination-Aware Diffusion Sampling for Inverse Problems via Robust Prior Updates
Jun 1, 2026Pengfei Jin, Yiqi Tian, Kailong Fan +2Diffusion-Based Inverse Problems Hallucination Mitigation
Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114 · Department of Industrial Engineering, University of Pittsburgh, Pittsburgh, PA 15261

Feb 25, 2026J/K move · Enter open · S save

Learning to reconstruct from saturated data: audio declipping and high-dynamic range imaging

Victor Sechaud, Laurent Jacques, Patrice Abry, Julián Tachella

CNRS and ENS de Lyon, Laboratoire de physique, Lyon, France · UCLouvain, ICTEAM, Louvain-la-Neuve, Belgium · CIFAR Fellow

Inverse Problems Self-Supervised Learning

Open paper AskPDF arXiv

Learning based methods are now ubiquitous for solving inverse problems, but their deployment in real-world applications is often hindered by the lack of ground truth references for training. Recent self-supervised learning strategies offer a promising alternative, avoiding the need for ground truth. However, most existing methods are limited to linear inverse problems. This work extends self-supervised learning to the non-linear problem of recovering audio and images from clipped measurements, by assuming that the signal distribution is approximately invariant to changes in amplitude. We provide sufficient conditions for learning to reconstruct from saturated signals alone and a self-supervised loss that can be used to train reconstruction networks. Experiments on both audio and image data show that the proposed approach is almost as effective as fully supervised approaches, despite relying solely on clipped measurements for training.

Abstract

Inverse Problems

Explore similar work

Learning to reconstruct from saturated data: audio declipping and high-dynamic range imaging

S3\mathrm{S}^3S3-PnP: Plug-and-Play Denoising for Scientific Imaging with Self-Supervised Score Priors

Hallucination-Aware Diffusion Sampling for Inverse Problems via Robust Prior Updates

$\mathrm{S}^3$ -PnP: Plug-and-Play Denoising for Scientific Imaging with Self-Supervised Score Priors