cs.CVSep 12, 2026
SaveRestore What Matters: Lessons from Joint Restoration and Recognition
Organizations: The University of Texas at Austin, Austin, USA · Purdue University, West Lafayette, USA · Michigan State University, Lansing, USA
Abstract
Recognition pipelines typically adopt a restore-then-recognize workflow, yet decades of experience show that generating visually pleasing images seldom translates to improved recognition. We propose a Joint Restoration-for-Recognition (JR) paradigm: restore only what downstream tasks truly require, with task signals dictating where, how much, and whether restoration is necessary. JR rests on three pillars: (i) Physics, employing optics-accurate turbulence simulation, extensible to blur and noise, to ground restoration in real image formation; (ii) Neuroscience, drawing on selective attention and neuroplasticity to direct model capacity toward identity-critical regions and frames while bypassing already-clean inputs; and (iii) Vision & Learning, coupling recognition loss end-to-end through restoration and alignment so that low-level edits maximize high-level identity stability. Evaluations on IARPA-BRIAR show consistent improvements (e.g., [email protected]% FAR +0.6; FNIR@1% FPIR -2.5), while a quality gate skips ~70% of clean frames, reducing cost. Ablations confirm physics priors enhance realism, joint training prevents catastrophic forgetting, and selective restoration suffices in many cases. We conclude that better-looking images are neither necessary nor sufficient; restoration modules must be task-driven, selective, and physically aware. Code, pretrained models, and recipes are provided for integration.
Figures & tables
Figure 1: High-level illustration of the challenges and motivation behind restoration-for-recognition tasks. (a) The distortion and blur in the degraded input could harm the recognition model a lot. (b) A two-stage pipeline in which an independent restoration model preprocessed the degraded input before recognition. While this can improve visual quality, it often fails to preserve the identity-specific features needed for accurate recognition. (c) Our joint restoration and recognition framework is driven by interdisciplinary insights from neuroscience, physics-grounded signal processing, and computer vision. Permission granted by the subject for use of imagery in publications.
Figure 2: Examples from the BRIAR dataset. The top rows show image examples with varying levels of degradation, pose, and occlusion. The bottom row presents a corresponding sequence of video frames. Permission granted by the subject for use of imagery in publications.
Figure 3: Why restoration may hurt recognition? Whenever restoration is performed, the prior used in the restoration has already implicitly introduced bias. This bias, if added wrongly, can lead to false recognition in downstream.
Figure 4: Restoration does not always improve recognition. (a): while the image quality is clearly improved, the similarity score actually drops. Please zoom in for best visual experience. (b): even though there is no noticeable change in terms of image quality, the recognition rate drops significantly after restoration. Please zoom in for best visualization. Images shown with subject permission for publication.
Figure 5: The overall tri-pillar pipeline consists of: (i) an optics-accurate turbulence simulator (PATSv5) Roddier, 1981 ; Chimitt et al., 2024 , (ii) a neuroscience-inspired, attention-guided video selection module Friston, 2010 ; Bastos et al., 2012 , and (iii) a task-aware co-optimization loop with differentiable alignment Wang et al., 2018 ; Deng et al., 2019 . Images shown with subject permission for publication.
| Metric | Degraded | Restoration | VIDCLS | Co-Optimize |
| 1:1 [email protected]%FAR | 62.6% | 63.5% | 63.3% | 64.1% |
| 1:N Rank Top 20 | 87.2% | 86.5% | 86.9% | 87.4% |
| 1:N Search FNIR@1%FPIR | 51.2% | 51.6% | 49.9% | 49.9% |
| v.s. Degraded | – | 0.07% | 0.57% | 1.0% |
Table 1: Comparison Before and After Using co-optimization strategy, tested on Test Set EVP 4.2.0 Reduced. Each column adds on top of the previous configuration.
Figure 6: Principle of atmospheric turbulence simulation Chimitt & Chan, 2020 . In optics perspective the PSF is caused by distortion on the phase plane, and phase distortion can be represented by linear combination of Zernike basis.
| Face Retrieval | Degraded | Internalizing | Explicit Modeling | ||
| PATSv3 [45] | PATSv5 [70] | PATSv3 [45] | PATSv5 [70] | ||
| Rank 5 | 37.75% | 38.83% | 39.18% | 39.01% | 39.19% |
| Rank 10 | 40.59% | 41.83% | 42.18% | 41.99% | 42.16% |
| Rank 20 | 45.29% | 46.40% | 46.70% | 46.79% | 46.68% |
Table 2: Face recognition is evaluated on degraded data using restoration models trained with (1) internalized synthetic data and (2) explicit loop modeling. Two different turbulence simulators, PATSv3 and PATSv5, are employed. To minimize confounding factors introduced by non-turbulence content, the experiment is conducted exclusively on a curated subset of long-range turbulence videos.
Figure 7: The principle of explicitly modeling degradation in the restore-degrade loop Jaiswal et al., 2023 , where a physics-based simulator re-applies degradation to the restored output to compute consistency loss. Images shown with subject permission for publication.
Figure 8: Co-optimization pipeline for restoration and recognition. The input video undergoes either co-trained restoration or pre-trained restoration, with distillation loss supervising the co-trained model. The restored video is processed through differentiable landmark alignment before face recognition, which is optimized using AdaFace loss at the classification layer. Not all frames may have a face detection, and such frames are filtered out before proceeding to the face recognition module. Images shown with subject permission for publication.
| Metric | Pretrained | Fine-tune | Co-optimize Restore-first | Co-optimize Align-first |
| 1:1 [email protected]%FAR | 63.5% | 63.6% | 64.1% | 63.7% |
| 1:N Rank Top 20 | 86.5% | 86.8% | 87.4% | 86.5% |
| 1:N Search FNIR@1%FPIR | 51.6% | 52.5% | 49.9% | 51.4% |
| PSNR | 31.94 | 15.69 | 32.00 | 31.59 |
| SSIM | 0.8799 | 0.2386 | 0.8790 | 0.8785 |
Table 3: Comparison among separately pretrained restoration, fine-tuning the restoration model using recognition loss only, co-optimization with a restore-first strategy, and co-optimization with an align-first strategy, evaluating both recognition performance and visual quality (PSNR and SSIM ).
| Method | w/o Restoration | w/ Restoration | ||||
| R20 | TAR | TPIR | R20 | TAR | TPIR | |
| ViT + Aug. | 65.09 | 28.99 | 16.70 | 67.23 | 31.13 | 18.08 |
| + LQ Data | 73.90 | 32.58 | 17.70 | 74.53 | 32.83 | 17.88 |
| + LQ + Restoration | 73.21 | 35.22 | 19.16 | 73.58 | 35.53 | 19.30 |
Table 4: Ablation on training data and augmentation. Evaluation without vs with restoration. R20, TAR, and TPIR denote Rank-20, TAR@FAR=0.001, and TPIR@FNIR=0.01, respectively.
| Metric | |||
| 1:1 [email protected]%FAR | 60.9% | 60.9% | 61.5% |
| 1:N Rank Top 20 | 87.4% | 87.4% | 87.2% |
| 1:N Search [email protected]%FPIR | 55.4% | 55.3% | 54.8% |
| PSNR | 32.00 | 32.05 | 32.07 |
| SSIM | 0.8790 | 0.8795 | 0.8803 |
Table 5: Comparison of the co-optimization framework under different weights , evaluating both recognition performance and visual quality (PSNR and SSIM ).
Figure 9: Trade-off between perceptual quality (PSNR) and recognition accuracy as the loss weight varies. While PSNR remains stable or improves with larger , recognition performance can degrade — highlighting the need to balance restoration and task objectives.
Figure 10: The human visual system employs attentional selection to regulate the processing of visual input, assigning greater cognitive resources to stimuli that are salient or relevant to current goals, while attenuating the impact of irrelevant or distracting signals.
Figure 11: Illustration of the temporal sampling strategy , which significantly reduces computational cost by selecting only a subset of representative frames, while preserving the original motion information.
Figure 12: Side-by-side comparisons of input images and Grad-CAM visualizations before and after image restoration. With restoration, the input images (left) exhibit noticeable improvements in clarity and detail post-restoration, with sharper features and reduced blur. The Grad-CAM outputs (right) highlight areas of importance for the model’s prediction, revealing that the restoration enhances the attention to key regions of interest, particularly around facial features. The alignment of two images is different as images are restored before alignment. Images shown with subject permission for publication.
| Metric | Degraded | Restore All | VIDCLS |
| 1:1 [email protected]%FAR | 62.6% | 63.5% | 63.3% |
| 1:N Rank Top 20 | 87.2% | 86.5% | 86.9% |
| 1:N Search FNIR@1%FPIR | 51.2% | 51.6% | 49.9% |
| vs. Degraded | – | 0.07% | 0.57% |
Table 6: Comparison Before and After Using Video Restoration Classifier, tested on Test Set EVP 4.2.0 Reduced. VIDCLS refers to the proposed Video Restoration Classifier.
Figure 13: Video Restoration Classifier (VIDCLS). The input video is sampled and processed by two branches: Rank Prediction and Score Prediction, both using a pre-trained Video Swin Transformer (SwinT) as the backbone. The decision to restore is based on the predicted rank and score — restoration is skipped if the rank is low or the score is high; otherwise, restoration is applied.
Figure 14: The confusion matrix and examples used to train the video restoration classifier. For most videos that already have high quality (Rank 1) before restoration, their rank remains 1 after restoration, indicating that restoration is unnecessary. Images shown with subject permission for publication.
| Trainable Layers | Validation Accuracy |
| Early Layers | 73.22% |
| Middle Layers | 73.38% |
| Late Layers | 73.54% |
| All Layers | 73.62% |
Table 7: Face recognition metrics when fine-tuning different components of the restoration network with co-training on a BRS validation set.
| Dataset | # Subjects | # Media (videos/images) |
| BRIAR-BRC | 995 | 162,949 |
| MSU-BRC | 452 | 5,599/17,593 |
| Accenture-BRC | 512 | 21,948/21,204 |
| Kitware-BRC | 509 | 568,263 |
| USC-BRC | 290 | 26,222 |
| STR-BRC | 436 | 8,394/25,135 |
Table 8: Training Dataset Statistics from BRIAR dataset.