Recognition pipelines typically adopt a restore-then-recognize workflow, yet decades of experience show that generating visually pleasing images seldom translates to improved recognition. We propose a Joint Restoration-for-Recognition (JR2) paradigm: restore only what downstream tasks truly require, with task signals dictating where, how much, and whether restoration is necessary. JR2 rests on three pillars: (i) Physics, employing optics-accurate turbulence simulation, extensible to blur and noise, to ground restoration in real image formation; (ii) Neuroscience, drawing on selective attention and neuroplasticity to direct model capacity toward identity-critical regions and frames while bypassing already-clean inputs; and (iii) Vision & Learning, coupling recognition loss end-to-end through restoration and alignment so that low-level edits maximize high-level identity stability. Evaluations on IARPA-BRIAR show consistent improvements (e.g., [email protected]% FAR +0.6; FNIR@1% FPIR -2.5), while a quality gate skips ~70% of clean frames, reducing cost. Ablations confirm physics priors enhance realism, joint training prevents catastrophic forgetting, and selective restoration suffices in many cases. We conclude that better-looking images are neither necessary nor sufficient; restoration modules must be task-driven, selective, and physically aware. Code, pretrained models, and recipes are provided for integration.
Figures & tables
Figure 1: High-level illustration of the challenges and motivation behind restoration-for-recognition tasks. (a) The distortion and blur in the degraded input could harm the recognition model a lot. (b) A two-stage pipeline in which an independent restoration model preprocessed the degraded input before recognition. While this can improve visual quality, it often fails to preserve the identity-specific features needed for accurate recognition. (c) Our joint restoration and recognition framework is driven by interdisciplinary insights from neuroscience, physics-grounded signal processing, and computer vision. Permission granted by the subject for use of imagery in publications.
Figure 2: Examples from the BRIAR dataset. The top rows show image examples with varying levels of degradation, pose, and occlusion. The bottom row presents a corresponding sequence of video frames. Permission granted by the subject for use of imagery in publications.
Figure 3: Why restoration may hurt recognition? Whenever restoration is performed, the prior used in the restoration has already implicitly introduced bias. This bias, if added wrongly, can lead to false recognition in downstream.
Figure 4: Restoration does not always improve recognition. (a): while the image quality is clearly improved, the similarity score actually drops. Please zoom in for best visual experience. (b): even though there is no noticeable change in terms of image quality, the recognition rate drops significantly after restoration. Please zoom in for best visualization. Images shown with subject permission for publication.
Figure 5: The overall tri-pillar pipeline consists of: (i) an optics-accurate turbulence simulator (PATSv5) Roddier, 1981 ; Chimitt et al., 2024 , (ii) a neuroscience-inspired, attention-guided video selection module Friston, 2010 ; Bastos et al., 2012 , and (iii) a task-aware co-optimization loop with differentiable alignment Wang et al., 2018 ; Deng et al., 2019 . Images shown with subject permission for publication.
Table 1: Comparison Before and After Using co-optimization strategy, tested on Test Set EVP 4.2.0 Reduced. Each column adds on top of the previous configuration.
Figure 6: Principle of atmospheric turbulence simulation Chimitt & Chan, 2020 . In optics perspective the PSF is caused by distortion on the phase plane, and phase distortion can be represented by linear combination of Zernike basis.
Face Retrieval
Degraded
Internalizing
Explicit Modeling
PATSv3 [45]
PATSv5 [70]
PATSv3 [45]
PATSv5 [70]
Rank 5
37.75%
38.83%
39.18%
39.01%
39.19%
Rank 10
40.59%
41.83%
42.18%
41.99%
42.16%
Rank 20
45.29%
46.40%
46.70%
46.79%
46.68%
Table 2: Face recognition is evaluated on degraded data using restoration models trained with (1) internalized synthetic data and (2) explicit loop modeling. Two different turbulence simulators, PATSv3 and PATSv5, are employed. To minimize confounding factors introduced by non-turbulence content, the experiment is conducted exclusively on a curated subset of long-range turbulence videos.
Figure 7: The principle of explicitly modeling degradation in the restore-degrade loop Jaiswal et al., 2023 , where a physics-based simulator re-applies degradation to the restored output to compute consistency loss. Images shown with subject permission for publication.
Figure 8: Co-optimization pipeline for restoration and recognition. The input video undergoes either co-trained restoration or pre-trained restoration, with distillation loss supervising the co-trained model. The restored video is processed through differentiable landmark alignment before face recognition, which is optimized using AdaFace loss at the classification layer. Not all frames may have a face detection, and such frames are filtered out before proceeding to the face recognition module. Images shown with subject permission for publication.
Table 3: Comparison among separately pretrained restoration, fine-tuning the restoration model using recognition loss only, co-optimization with a restore-first strategy, and co-optimization with an align-first strategy, evaluating both recognition performance and visual quality (PSNR ↑ and SSIM ↑ ).
Method
w/o Restoration
w/ Restoration
R20 ↑
TAR ↑
TPIR ↑
R20 ↑
TAR ↑
TPIR ↑
ViT + Aug.
65.09
28.99
16.70
67.23
31.13
18.08
+ LQ Data
73.90
32.58
17.70
74.53
32.83
17.88
+ LQ + Restoration
73.21
35.22
19.16
73.58
35.53
19.30
Table 4: Ablation on training data and augmentation. Evaluation without vs with restoration. R20, TAR, and TPIR denote Rank-20, TAR@FAR=0.001, and TPIR@FNIR=0.01, respectively.
Table 5: Comparison of the co-optimization framework under different weights λ , evaluating both recognition performance and visual quality (PSNR ↑ and SSIM ↑ ).
Figure 9: Trade-off between perceptual quality (PSNR) and recognition accuracy as the loss weight λ varies. While PSNR remains stable or improves with larger λ , recognition performance can degrade — highlighting the need to balance restoration and task objectives.
Figure 10: The human visual system employs attentional selection to regulate the processing of visual input, assigning greater cognitive resources to stimuli that are salient or relevant to current goals, while attenuating the impact of irrelevant or distracting signals.
Figure 11: Illustration of the temporal sampling strategy , which significantly reduces computational cost by selecting only a subset of representative frames, while preserving the original motion information.
Figure 12: Side-by-side comparisons of input images and Grad-CAM visualizations before and after image restoration. With restoration, the input images (left) exhibit noticeable improvements in clarity and detail post-restoration, with sharper features and reduced blur. The Grad-CAM outputs (right) highlight areas of importance for the model’s prediction, revealing that the restoration enhances the attention to key regions of interest, particularly around facial features. The alignment of two images is different as images are restored before alignment. Images shown with subject permission for publication.
Table 6: Comparison Before and After Using Video Restoration Classifier, tested on Test Set EVP 4.2.0 Reduced. VIDCLS refers to the proposed Video Restoration Classifier.
Figure 13: Video Restoration Classifier (VIDCLS). The input video is sampled and processed by two branches: Rank Prediction and Score Prediction, both using a pre-trained Video Swin Transformer (SwinT) as the backbone. The decision to restore is based on the predicted rank and score — restoration is skipped if the rank is low or the score is high; otherwise, restoration is applied.
Figure 14: The confusion matrix and examples used to train the video restoration classifier. For most videos that already have high quality (Rank 1) before restoration, their rank remains 1 after restoration, indicating that restoration is unnecessary. Images shown with subject permission for publication.
Trainable Layers
Validation Accuracy
Early Layers
73.22%
Middle Layers
73.38%
Late Layers
73.54%
All Layers
73.62%
Table 7: Face recognition metrics when fine-tuning different components of the restoration network with co-training on a BRS validation set.
Dataset
# Subjects
# Media (videos/images)
BRIAR-BRC
995
162,949
MSU-BRC
452
5,599/17,593
Accenture-BRC
512
21,948/21,204
Kitware-BRC
509
568,263
USC-BRC
290
26,222
STR-BRC
436
8,394/25,135
Table 8: Training Dataset Statistics from BRIAR dataset.
Image restoration aims to recover high quality images from inputs degraded by various factors, such as adverse weather, blur, or low light. While recent studies have shown remarkable progress across individual or unified restoration tasks, they still suffer from limited generalization and inefficiency when handling unknown or composite degradations. To address these limitations, we propose RAR, a Restore, Assess and Repeat process, that integrates Image Quality Assessment (IQA) and Image Restoration (IR) into a unified framework to iteratively and efficiently achieve high quality image restoration. Specifically, we introduce a restoration process that operates entirely in the latent domain to jointly perform degradation identification, image restoration, and quality verification. The resulting model is fully trainable end to end and allows for an all-in-one assess and restore approach that dynamically adapts the restoration process. Also, the tight integration of IQA and IR into a unified model minimizes the latency and information loss that typically arises from keeping the two modules disjoint, (e.g. during image and/or text decoding). Extensive experiments show that our approach consistent improvements under single, unknown and composite degradations, thereby establishing a new state-of-the-art.
I-Hsiang Chen, Isma Hadji, Enrique Sanchez +5
Samsung AI Center Cambridge · National Taiwan University · Technical University of Iasi +3
Image restoration is fundamentally constrained by the tradeoff between distortion and perception: minimizing pixel-wise error yields over-smoothed results, whereas optimizing for perceptual realism often introduces structural deviations. Recent approaches attempt to balance this tradeoff via posterior sampling or multi-stage generative pipelines, yet remain computationally expensive and architecturally complex. To overcome these limitations, we propose PCFlow (Perceptually Consistent Flow Matching), a unified framework that directly parameterizes a continuous transport from degraded observations to clean targets, jointly optimizing distortion and perceptual quality. While its latent consistency flow objective drives stable and efficient few-step inference, a Latent Consistency Perceptual Loss (LCPL) imposes semantic constraints directly on the guiding velocity field, steering the dynamics toward visually sharp data manifolds. Furthermore, recognizing the inherent conflict between structural and perceptual consistencies, we integrate a conflict-free gradient projection strategy to stabilize the multi-objective optimization landscape. Combined with lightweight, convolution-only backbone, PCFlow achieves competitive performance across diverse restoration tasks at a fraction of traditional computational costs.
Sangwoo Jo, Donggeun Ko, Jayeon Kang +3
Korea University, Seoul, South Korea · Aim Future, Seoul, South Korea
Adapting image restoration models to a stream of new tasks without revisiting past data remains challenging due to catastrophic forgetting. In this work, we propose Restoring without Forgetting (RwF), a filter-level continual adaptation framework for image restoration built upon a critical observation: task-specific knowledge is centered in a small subset of filters and can be separated from those reconstructing general content. RwF first performs parameter-space integrated gradients attribution to localize degradation-critical filters in a coarse-to-fine manner. It then adapts to new tasks by generating task-specific filters from a filter bank using compact factorized low-rank transformations, further augmented with cross-task attention and prototypical contrastive learning, and lastly assembles them back only at localized positions. Experiments on six restoration tasks show that RwF effectively avoids forgetting and achieves competitive restoration quality against all-in-one methods that have full data access, and outperforms LoRA-style adaptation with ∼10× fewer additional parameters. Code is available at https://github.com/funkdub/Restoring-without-Forgetting.
Xin Feng, Jin Zhao, Yizhen Zhang +3
School of Informatics, University of Edinburgh · Baidu Inc. · Tsinghua University +1