Recent progress in image restoration has converged on all-in-one architectures that jointly handle multiple degradations within a single network. These methods are effective on static benchmarks but target a closed-world setting that assumes simultaneous access to every target degradation at training time. In practice, degradations are encountered sequentially as field-deployed systems progressively face new environmental conditions, and historical training data is often unavailable due to privacy or storage constraints. Accommodating a new degradation then requires either retraining on the union of all prior data, which is often costly or infeasible, or fine-tuning, which causes catastrophic forgetting. We formulate multi-degradation image restoration as a continual domain-incremental learning problem, in which degradations arrive incrementally and prior data is unavailable. Our proposed Restoring without Forgetting (RwF) framework learns a lightweight adapter for each new degradation, eliminating forgetting by construction at a fraction of the cost of dedicated per-domain networks. To isolate degradation learning from dataset variation, we construct a benchmark spanning five degradation domains under shared image content. At test time, an unsupervised routing mechanism identifies the appropriate restoration path for unknown inputs without requiring domain labels. Across the five-domain sequence, RwF improves final average PSNR over naive sequential fine-tuning by 15.25 dB and 11.83 dB on the Restormer and NAFNet backbones, respectively. The framework transfers to eleven canonical real-degradation benchmarks (3,465 images) at 89.5% routing accuracy with only a +0.94 dB oracle PSNR gap, establishing, to our knowledge, the first systematic baseline for continual multi-degradation image restoration.
Figures & tables
Figure 1: (a) Three paradigms for multi-degradation image restoration: One-By-One (one network per degradation), All-In-One (a single jointly trained network), and RwF (Ours), which freezes a pretrained backbone and attaches lightweight degradation-specific adapters routed without supervision at inference. (b) Average incremental PSNR across five sequentially learned degradations (Noise, +Blur, +Rain, +Haze, +Lowlight). Sequential fine-tuning forgets as degradations accumulate, while RwF holds performance across all domains.
Figure 2: RwF Framework: a denoising-pretrained encoder–decoder backbone is frozen, and each new degradation trains a network-spanning path of low-rank adapters under strict parameter isolation, leaving prior paths and the backbone unchanged. A domain prototype is formed by pooling first-encoder-stage features. At inference, an unknown input is routed by cosine similarity to the matching path, which restores the image.
PSNR (dB)
SSIM
Method
Backbone
AB↑
Aˉ↑
F↓
AB↑
Aˉ↑
F↓
Sequential FT
NAFNet
12.20
20.79
15.83
0.5697
0.6262
0.3287
Restormer
13.62
24.16
22.05
0.6148
0.7102
0.3705
EWC
NAFNet
11.63
20.78
16.00
0.5376
0.6256
0.3366
Restormer
12.00
21.26
14.43
0.4521
0.6133
0.3193
LwF
NAFNet
18.10
22.14
0.22
0.5559
0.6591
0.0305
Table 1: Continual learning comparison across the five-domain sequence ( D1→D5 : noise, blur, rain, haze, low-light) under the oracle setting.
PSNR (dB) ↑
SSIM ↑
Domain
Test set
N
ID Acc. (%)
Pred.
Oracle
Pred.
Oracle
D1 (noise)
CBSD68
68
92.6
28.76
30.18
0.8448
0.8713
Kodak24
24
100.0
31.66
31.66
0.8754
0.8754
Urban100
100
90.0
28.74
30.75
0.8622
0.8906
D2 (blur)
RealBlur-J
980
93.6
26.55
26.57
0.8349
0.8347
RealBlur-R
980
94.9
32.78
33.88
0.8962
0.9379
Table 2: Domain-agnostic evaluation of RwF-Restormer on eleven real-degradation benchmarks. Routing accuracy and reconstruction quality are reported under both domain-agnostic (Pred.) and oracle settings.
Figure 3: A D4 hazy input (left) misrouted to the D3 derain path is still largely restored (right).
Table 6
Figure 4: Qualitative comparison across all five degradation domains for a shared scene. Top: degraded inputs. Middle: sequentially fine-tuned Restormer after the full D1→D5 schedule. Bottom: RwF-Restormer under domain-agnostic inference, with the router’s prediction ( →Dt ✓). Per-image PSNR is shown in the lower-left of each restored panel.
Adapting image restoration models to a stream of new tasks without revisiting past data remains challenging due to catastrophic forgetting. In this work, we propose Restoring without Forgetting (RwF), a filter-level continual adaptation framework for image restoration built upon a critical observation: task-specific knowledge is centered in a small subset of filters and can be separated from those reconstructing general content. RwF first performs parameter-space integrated gradients attribution to localize degradation-critical filters in a coarse-to-fine manner. It then adapts to new tasks by generating task-specific filters from a filter bank using compact factorized low-rank transformations, further augmented with cross-task attention and prototypical contrastive learning, and lastly assembles them back only at localized positions. Experiments on six restoration tasks show that RwF effectively avoids forgetting and achieves competitive restoration quality against all-in-one methods that have full data access, and outperforms LoRA-style adaptation with ∼10× fewer additional parameters. Code is available at https://github.com/funkdub/Restoring-without-Forgetting.
Xin Feng, Jin Zhao, Yizhen Zhang +3
School of Informatics, University of Edinburgh · Baidu Inc. · Tsinghua University +1
Image restoration models are typically trained with a fixed set of capabilities. When new restoration requirements emerge, existing solutions usually train additional models or jointly retrain the original model with both new and historical data. Instead of designing another restoration backbone, we investigate how a trained restorer can continually acquire new capabilities without forgetting those learned previously. We propose RestoreMore, a continual capability-expansion framework that preserves the pretrained restoration model as a frozen capability anchor and learns residual expansion modules for newly arriving degradations. RestoreMore introduces a capability-oriented bi-level routing mechanism at multiple feature stages. The first routing level identifies restoration capabilities relevant to the current input, while the second selects and combines a sparse set of complementary degradation experts. This design enables newly introduced tasks to selectively reuse historical restoration knowledge and progressively enriches the expert bank available for subsequent restoration tasks. Extensive experiments on a wide range of restoration benchmarks demonstrate that RestoreMore consistently acquires new restoration abilities while preserving and improving previously learned capabilities.
Hu Gao, Yulong Chen, Lizhuang Ma
Department of Computer Science Shanghai Jiao Tong University Shanghai, China · Department of Architecture and Design Harbin Institute of Technology Heilongjiang, China
Image restoration aims to recover high quality images from inputs degraded by various factors, such as adverse weather, blur, or low light. While recent studies have shown remarkable progress across individual or unified restoration tasks, they still suffer from limited generalization and inefficiency when handling unknown or composite degradations. To address these limitations, we propose RAR, a Restore, Assess and Repeat process, that integrates Image Quality Assessment (IQA) and Image Restoration (IR) into a unified framework to iteratively and efficiently achieve high quality image restoration. Specifically, we introduce a restoration process that operates entirely in the latent domain to jointly perform degradation identification, image restoration, and quality verification. The resulting model is fully trainable end to end and allows for an all-in-one assess and restore approach that dynamically adapts the restoration process. Also, the tight integration of IQA and IR into a unified model minimizes the latency and information loss that typically arises from keeping the two modules disjoint, (e.g. during image and/or text decoding). Extensive experiments show that our approach consistent improvements under single, unknown and composite degradations, thereby establishing a new state-of-the-art.
I-Hsiang Chen, Isma Hadji, Enrique Sanchez +5
Samsung AI Center Cambridge · National Taiwan University · Technical University of Iasi +3