Organizations: School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, Guangdong, China · Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · School of Electronics and Information Engineering, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, Guangdong, China
Deep unfolding networks (DUNs), combining conventional iterative optimization algorithms and deep neural networks into a multi-stage framework, have achieved remarkable accomplishments in Image Restoration (IR), such as spectral imaging reconstruction, compressive sensing and super-resolution.It unfolds the iterative optimization steps into a stack of sequentially linked blocks.Each block consists of a Gradient Descent Module (GDM) and a Proximal Mapping Module (PMM) which is equivalent to a denoiser from a Bayesian perspective, operating on Gaussian noise with a known level.However, existing DUNs suffer from two critical limitations: (i) their PMMs share identical architectures and denoising objectives across stages, ignoring the need for stage-specific adaptation to varying noise levels; and (ii) their chain of structurally repetitive blocks results in severe parameter redundancy and high memory consumption, hindering deployment in large-scale or resource-constrained scenarios.To address these challenges, we introduce generalized Deep Low-rank Adaptation (LoRA) Unfolding Networks for image restoration, named LoRun, harmonizing denoising objectives and adapting different denoising levels between stages with compressed memory usage for more efficient DUN.LoRun introduces a novel paradigm where a single pretrained base denoiser is shared across all stages, while lightweight, stage-specific LoRA adapters are injected into the PMMs to dynamically modulate denoising behavior according to the noise level at each unfolding step.This design decouples the core restoration capability from task-specific adaptation, enabling precise control over denoising intensity without duplicating full network parameters and achieving up to N times parameter reduction for an N-stage DUN with on-par or better performance.Extensive experiments conducted on three IR tasks validate the efficiency of our method.
Figures & tables
Fig. 1: PSNR-Params.-FLOPs comparison of our LoRun and SOTAs in CASSI task. Our LoRun reduces the parameters of traditional DUN methods by nearly 70% while maintaining comparable performance.
Fig. 2: Structure comparison between the previous DUNs and our hierarchical deep LoRA-unfolding networks. Previous DUNs adopt independent parameters to all stages while our LoRun first leverages pre-trained denoiser for each stage then is trained with different LoRA layers for different denoising abilities.
Fig. 3: Representation for LoRA adopted in our LoRun framework.
Fig. 4: Illustration of the proposed LoRun method. (a) The gradient descent module. (a-1) The gradient descent process based on the PGD algorithm. (a-2) The gradient descent process based on the HQS algorithm. (b) The proximal mapping module (denoiser). (c) Framework of the proposed deep LoRA-Unfolding network. Our LoRun first adopts the same most frozen-shared backbone denoiser among all stages, followed by fine-tuning the denoising level and direction of each stage with small learnable-separate low-rank modules. This strategy mostly reduces the parameters and accelerates convergence.
Datasets
CS Ratios
AMP-Net
CASNet
DGU-Net +
TransCS
AutoBCS
CSformer
NesTD-Net
Block-9
LoRun-9
TIP’21
TIP’22
CVPR’22
TIP’22
TCYB’23
TIP’23
TIP’24
Ours
Ours
-
-
0.94M
16.90M
6.81M
1.49M
2.01M
6.65M
5.93M
15.96M
5.23M
General100
0.01
22.71/ 0.68
23.48/0.65
22.86/0.62
21.66/0.54
22.24/0.62
23.35/ 0.69
23.14/0.62
23.89 /0.65
23.92 /0.65
0.04
26.96/0.77
28.50/ 0.82
27.92/ 0.81
27.25/0.74
27.91/0.80
28.61 /0.80
28.58/ 0.82
28.60/ 0.82
28.70 / 0.82
0.10
30.62/0.88
32.78/ 0.91
32.74/ 0.91
32.17/0.87
32.70/ 0.89
32.97 /0.88
32.85 / 0.91
32.82/ 0.91
32.78/ 0.91
0.25
36.01/0.95
38.07/ 0.97
37.55/ 0.96
36.95/ 0.96
36.71/0.96
36.51/ 0.96
38.42 / 0.97
38.17 / 0.96
38.16/ 0.97
TABLE I: Average PSNR (dB)/SSIM performance comparisons of recent CS methods on various datasets at different CS ratios.
Fig. 5: CS results under ratio β=25% of different methods and datasets, from top to bottom General100 , Set11 and Set14 .
Model
Method
β =1%
β =4%
β =10%
β =25%
β =30%
Params.
Transformer
Block-9
20.73 0.54
24.50 0.74
28.44 0.88
33.63 0.95
34.72 0.95
0.10M
LoRun-9 γ =10
21.02 0.55
25.17 0.75
29.14 0.89
34.16 0.95
35.54 0.95
0.06M
U-Net
Block-9
21.14 0.56
25.60 0.79
29.57 0.89
35.00 0.96
36.36 0.96
4.65M
LoRun-9 γ =10
21.13 0.56
25.45 0.79
29.29 0.89
34.75 0.95
36.17 0.96
1.42M
TABLE II: Comparisons on Set11 under different CS ratios for CS based on Transformer and U-Net denoiser. PSNR (upper entry), SSIM (lower entry) and parameters are reported.
-
DeSCI
λ -Net
TSA-Net
PnP-CASSI
DIP-HSI
BIRNAT
GAP-Net
DAUHST
DWMT
PSRSCI
RCUMP
Block-9
LoRun-9
Category
Model
CNN
CNN
PnP
PnP
RNN
DUN
DUN
Transformer
Diffusion
DUN
DUN
DUN
-
-
Supervised
Supervised
Zero-Shot
Zero-Shot
-
Supervised
Supervised
Supervised
Supervised
Supervised
Supervised
Supervised
Reference
TPAMI’19
ICCV’19
ECCV’20
PR’21
ICCV’21
TPAMI’23
IJCV’23
NeurIPS’22
AAAI’24
ICLR’25
TIP’24
Ours
Ours
Params.
-
62.64M
44.25M
33.85M
33.85M
4.40M
4.27M
6.15M
15.47M
1312M
9.03M
5.18M
1.62M
scene01
28.38/0.80
30.10/0.85
32.31/0.89
29.09/0.80
31.32/0.86
36.79/0.95
33.63/0.91
37.25/ 0.96
36.46/ 0.96
37.18/ 0.96
37.63/ 0.96
38.88 / 0.97
38.78 / 0.97
scene02
26.00/0.70
28.49/0.81
31.03/0.86
28.05/0.71
25.89/0.70
37.89/0.96
33.19/0.90
39.02/ 0.97
37.75/0.96
38.74/ 0.97
39.98/ 0.97
41.70 / 0.98
41.68 / 0.98
TABLE III: Comparisons of PSNR (dB)/SSIM between LoRun ( γ=10 ) and SOTA methods on 10 simulation scenes in KAIST for CASSI .
Fig. 6: CASSI results of scene01 in different bands, from top to bottom λ∈{481.5\text{,}\mathrm{nm},492.5\text{,}\mathrm{nm},522.5\text{,}\mathrm{nm},567.5\text{,}\mathrm{nm},604.5\text{,}\mathrm{nm}} .
Fig. 7: CASSI results of different scenes in different bands, from top to bottom λ∈{487.0\text{,}\mathrm{nm},498.0\text{,}\mathrm{nm},584.5\text{,}\mathrm{nm},492.5\text{,}\mathrm{nm},476.5\text{,}\mathrm{nm}} .
Fig. 8: CASSI measurements of different scenes, each scene has the size of 256×310 .
Method
scene01
scene02
scene03
scene04
scene05
Avg
Params.
Block-9
34.87 0.94
35.32 0.94
35.48 0.95
40.55 0.97
32.74 0.95
35.79 0.95
4.80M
LoRun-9 γ =10
34.20 0.93
34.40 0.92
35.23 0.95
41.26 0.98
32.15 0.94
35.45 0.94
1.43M
TABLE IV: Comparisons on CASSI datasets with U-Net denoiser. PSNR (upper entry), SSIM (lower entry) and parameters are reported.
Fig. 9: Presentation of 4 isotropic Gaussian kernels with different widths (0.7, 1.2, 1.6, 2.0), 4 anisotropic Gaussian kernels and 4 motion blur kernels.
Method
SF
K1
K2
K3
K4
K5
K6
K7
K8
K9
K10
K11
K12
Params.
Block-9
×2
29.90
30.31
29.79
28.70
28.41
28.29
28.49
27.63
29.57
29.49
29.54
29.52
5.3M
×3
26.70
27.37
27.51
27.46
27.27
27.21
27.30
27.10
26.89
26.78
26.68
26.52
×4
24.75
25.60
25.84
25.94
25.86
25.75
25.79
25.95
25.18
24.92
24.97
24.96
LoRun-9 γ=10
×2
29.88
30.17
29.73
28.72
28.45
28.25
28.42
27.60
29.03
29.31
29.43
29.43
1.96M
×3
26.73
27.38
27.52
27.47
27.29
27.25
27.22
27.02
26.65
26.68
26.43
26.33
×4
24.74
25.65
25.89
25.99
25.90
25.80
25.75
25.99
25.05
24.82
24.72
24.83
TABLE V: Comparisons between our LoRun ( γ=10 ) and Block-9 for BSD68 datasets SR in different SFs and different blur kernels.
Fig. 10: SR results of different blur kernels and SFs. (Up: K5 , SF= ×2 . Down: K9 , SF= ×4 .)
Method
β =1%
β =4%
β =10%
β =25%
β =30%
β =40%
β =50%
LoRun-9 β -pretrained=1%
21.07 0.55
25.41 0.79
29.15 0.89
34.47 0.95
35.72 0.96
38.09 0.97
40.26 0.98
LoRun-9 β -pretrained=50%
21.08 0.55
25.66 0.79
29.57 0.89
35.07 0.96
36.31 0.96
38.65 0.98
40.58 0.98
TABLE VI: Ablation experiments on β -pretrained based on the U-Net denoiser in CS .
Method
β =1%
β =4%
β =10%
β =25%
β =30%
β =40%
β =50%
LoRun-9 γ =10
21.02 0.55
25.17 0.75
29.14 0.89
34.16 0.95
35.54 0.95
37.88 0.97
40.10 0.98
LoRun-9 γ =20
21.08 0.55
25.17 0.75
29.13 0.89
34.24 0.95
35.56 0.95
37.95 0.97
40.14 0.98
LoRun-9 γ =80
20.97 0.55
25.01 0.74
28.83 0.88
33.93 0.95
35.30 0.95
37.74 0.97
39.98 0.98
TABLE VII: Ablation on γ based on the Transformer denoiser in CS .
Method
baby
bird
head
Average
Params.
Block-1
29.62
27.55
28.55
28.57
0.59M
Block-1 w/o GDM
29.14
27.28
25.80
27.41
0.59M
TABLE VIII: Ablation analysis for GDM in SR with SF = ×3 on Set5 .
Fig. 11: Loss curves with training iterations among different training strategies.
K=1
K=4
K=9
K=12
Original
31.08 0.9032
32.28 0.9085
32.84 0.9128
32.91 0.9133
LoRun (Ours)
31.28 0.9062
32.24 0.9090
32.78 0.9125
32.81 0.9130
TABLE IX: Ablation analysis for the effect of the number K of blocks.
Fig. 12: Visualization of the selected LoRA modules.
Fig. 13: Structures of the Transformer denoiser and its components. (a) The Transformer denoiser which aims to demonstrate the performance of our LoRun in simple but classical modules. (b) The feed forward network of the Transformer. (c) The attention network induced with LoRA of the Transformer.
Fig. 14: Structures of the U-Net denoiser and its components. (a) The U-Net denoiser which aims to demonstrate the performance of our LoRun in simple but classical modules. (b) The double-convolution module induced with LoRA.
Fig. 15: Structures of the components of the denoiser for CASSI task. (a) The convolution modulated block of the denoiser. (b) The spatial alignment block. (c) The window-based spectra wise self-attention with LoRA.
Fig. 16: Structures of the denoiser and its components for CS task. (a) The depth-wise channel attention block. (b) The multi-scale transformer module with LoRA. (c) The denoiser which is the dual path fusion block for CS task.
Fig. 17: Structures of the denoiser and its components for SR task. This architecture is based on the convolution-LoRA module and upsample block.
Deep unfolding turns iterative algorithms into trainable networks, combining model-based structure with learned components for efficient inference and optimization.
Degraded images not only reduce visual quality but also impair downstream high-level vision tasks. Task-driven image restoration (TDIR) addresses this issue by jointly optimizing restoration quality and task performance. Recent works show that pretrained diffusion priors benefit TDIR, yet diffusion-based restoration is inherently stochastic, as the sampling process depends on a random noise term, which can undermine task consistency. In this paper, we show that a deterministic, noise-free one-step forward pass with pretrained diffusion priors can substantially improve TDIR, but the benefit critically depends on the adaptation module: LoRA yields consistent gains, whereas ControlNet-style conditioning does not. This enables one-step forwarding that surpasses conventional multi-step diffusion TDIR baselines. Furthermore, we introduce a task-preserving GAN training strategy that improves perceptual quality without sacrificing task performance. Extensive experiments on classification, segmentation, and detection demonstrate consistent gains over prior TDIR methods, and we further validate generalization on real-world degraded images and OCR.
Jaeha Kim, Kyoung Mu Lee
Dept. of ECE & ASRI, Seoul National University, Seoul, Korea · IPAI, Seoul National University, Seoul, Korea
All-in-One image restoration aims to develop a unified restoration framework for handling diverse degradation types. Existing end-to-end methods usually regard the restoration process as a black-box mapping, lacking an explicit optimization interpretation. Although deep unfolding provides an interpretable iterative modeling paradigm for image restoration, existing methods mostly rely on fixed degradation assumptions or predefined degradation information, making them difficult to adapt to unified restoration requirements under complex degradations and locally damaged content. This limitation restricts their performance in degradation suppression and structural detail recovery. To address these issues, this paper proposes DVANet, a deep unfolding network inspired by the half-quadratic splitting optimization algorithm, which formulates unified image restoration under complex degradations as a collaborative unfolding process between degradation-aware observation consistency and visual-prior-guided reconstruction. Specifically, in the degradation-aware observation consistency branch, a degradation representation module is employed to extract global degradation attributes and local degradation cues, and degradation-conditioned mapping is used to enhance the model's adaptability to different degradation types. In the visual-prior-guided reconstruction branch, DINOv3 is introduced to provide structural and semantic information as hierarchical visual priors, thereby complementing the missing structural information in damaged regions and improving detail recovery. Extensive experiments demonstrate that DVANet achieves superior or competitive performance on multi-scenario degradation and cross-domain image restoration tasks, showing favorable degradation adaptability and generalization ability.
Yanjie Tu, Qingsen Yan, Axi Niu +3
School of Computer Science, Northwestern Polytechnical University, Xi’an 710000, China · Shenzhen Research Institute of Northwestern Polytechnical University, Shenzhen 518057, China · State Key Laboratory of Internet of Things for Smart City, University of Macau, Macau 999078, China
Compressed sensing (CS) reconstructs images from highly limited measurements, but existing deep unfolding methods are typically driven by first-order optimization and weakly exploit the optimization states generated during reconstruction. To address these limitations, we propose a Newton deep unfolding network (NDU-Net), which, to the best of our knowledge, is the first deep unfolding framework that leverages second-order optimization for CS reconstruction. Specifically, NDU-Net introduces a Newton update (NU) module to estimate Newton-type update directions and generate optimization states that characterize the current reconstruction process. Furthermore, a Newton-guided multi-scale prior (MP) module is designed to incorporate these optimization states into multi-scale feature restoration, thereby enabling the learned prior to adapt to the current reconstruction stage. Experimental results under different CS ratios confirm that our proposed NDU-Net achieves promising reconstruction performance and exhibits enhanced robustness. Our code is available at https://github.com/xianchaoxiu/DNU-Net.
Changhua He, Xianchao Xiu
School of Mechatronic Engineering and Automation, Shanghai University