Smartphone cameras have gained immense popularity with the adoption of high-resolution and high-dynamic range imaging. As a result, high-performance camera Image Signal Processors (ISPs) are crucial in generating high-quality images for the end user while keeping computational costs low. In this paper, we propose DRIFT (Deep Restoration, ISP Fusion, and Tone-mapping): an efficient AI mobile camera pipeline that generates high quality RGB images from hand-held raw captures. The first stage of DRIFT is a Multi-Frame Processing (MFP) network that is trained using a adversarial perceptual loss to perform multi-frame alignment, denoising, demosaicing, and super-resolution. Then, the output of DRIFT-MFP is processed by a novel deep-learning based tone-mapping (DRIFT-TM) solution that allows for tone tunability, ensures tone-consistency with a reference pipeline, and can be run efficiently for high-resolution images on a mobile device. We show qualitative and quantitative comparisons against state-of-the-art MFP and tone-mapping methods to demonstrate the effectiveness of our approach.
Figures & tables
Figure 2: Overview of the training and inference pipelines for DRIFT-MFP. During training, we model handshake motions using homographies from real data and apply them to tripod captures of corresponding short exposure and long exposure images with equalized brightness.
Figure 3: Overview of the training and inference pipelines for DRIFT-TM. We train DRIFT-TM using a computationally expensive referenece tone-map. DRIFT-TM learns the residual enhancements from a light-weight tonemap allowing more robust learning as well as tunability by modulating the enhancements.
Figure 4: DRIFT Tone-map network architecture. We incorporate a local encoder that encodes a full-resolution image per tile, a global encoder that encodes a low-resolution full image, and a metadata encoder to encode capture metadata.
Figure 5: Denoising results across various scenes. For each scene (row), the columns correspond to: (a) The low-quality input image, (b) BIPNet, (c) MPRNet (d)Burstormer, (e) Restormer (f) NAFNet (g) Our proposed method DRIFT-MFP, and (h) The Ground Truth (GT). Our method consistently outperforms the baselines in terms of visual quality and fidelity to the GT.
Figure 6: 4x SR results across various scenes. NAFNET trained with LPIPS achieves the best FID score but produces artifacts as seen in the second row while our method maintains fidelity to GT without generating unnatural artifacts.
Method
LPIPS
FID
PSNR
SSIM
BIPNET [ 9 ]
0.09
11.68
36.32
0.97
Burstormer [ 10 ]
0.04
6.18
37.06
0.98
MPRNet [ 48 ]
0.08
12.40
36.76
0.97
Restormer [ 49 ]
0.05
8.49
36.36
0.97
NAFNet (L1)
0.10
29.00
35.48
0.90
NAFNet (L1+GAN)
0.15
45.42
34.01
0.88
Table 1: Quantitative Evaluation of Multi-frame Denoising
Method
LPIPS
FID
PSNR
SSIM
BIPNET [ 9 ]
0.16
21.91
34.79
0.94
FBANet [ 42 ]
0.19
21.60
32.05
0.94
Burstormer [ 10 ]
0.09
12.27
35.07
0.95
MPRNet [ 48 ]
0.13
21.53
35.19
0.95
Restormer [ 49 ]
0.12
16.28
35.25
0.95
NAFNet (L1)
0.11
17.82
37.75
0.96
Table 2: Evaluation of Multi-frame 4x Super-Resolution
NAFNET
Same
DRIFT-MFP
User Preference
28.2%
8.8%
63.0%
Table 3: Results of user study on DRIFT-MFP denoising and super-resolution
Figure 7: Non-reference tone-mapping methods comparisons. Our method consistently outperforms the other ones in visual quality.
Figure 8: Visual comparison with state-of-the-art supervised learning based tone-mapping methods (top: image, bottom: difference map from GT). The results indicate that the both LLF-LUT and TMO-GAN have an inferior ability to match the tone-mapping operation to the GT compared to our proposed method.
Method
TMQI-Q
TMQI-S
TMQI-N
IQATM [ 17 ]
0.805
0.713
0.330
Self-TMO [ 39 ]
0.759
0.725
0.091
TMO-GAN [ 52 ]
0.801
0.757
0.253
DRIFT-TM
0.845
0.791
0.421
GT
0.847
0.792
0.432
Table 4: Quantitative results of the non-reference comparison experiments with the TMQI metric.
Method
PSNR
SSIM
LLF-LUT ( [ 50 ] )
30.89
0.95
TMO-GAN ( [ 52 ] )
30.83
0.96
DRIFT-TM (w/o maps)
34.06
0.98
DRIFT-TM (w/o global and meta)
39.66
0.99
DRIFT-TM (w/o meta)
40.43
0.99
DRIFT-TM
40.59
0.99
Table 5: Quantitative results comparing the prior methods with our proposed method and its ablation study variations.
Figure 9: Illustration showing the value of co-designing DRIFT-MFP and DRIFT-TM. Co-designing allows us to adjust the contrast-enhancement and rendering strengths based on the noise levels in the input and DRIFT-MFP output resulting in reduced noise and artifact enhancement in the sky regions as shown.
Figure 10: Visual comparison for ablation study (top: image, bottom: difference map from GT). While the complete proposed method, DRIFT-TM, produces a visual match to the GT, the ablations show progressively more deviation such as tiling artifacts in ”DRIFT w/o global and meta” as well as large color deviations in ”DRIFT w/o maps”.
Figure 11: Visual comparison showing incorrect metadata at inference: Top (image), bottom (GT difference map). ”Meta low/high” denote DRIFT-TM inference with altered ISO/exposure. Incorrect metadata is non-catastrophic, but accurate metadata improves tone-mapping precision.
Figure 12: Illustration of our method’s tunability: left shows contrast adjustment; right shows HDR effect adjustment.