Burst image restoration aims to reconstruct a high-quality image from burst images, which are typically captured using manually designed exposure settings. Although these exposure settings significantly influence the final restoration performance, the problem of finding optimal exposure settings has been overlooked. In this paper, we present Dynamic Exposure Burst Image Restoration (DEBIR), a novel burst image restoration pipeline that enhances restoration quality by dynamically predicting exposure times tailored to the shooting environment. In our pipeline, Burst Auto-Exposure Network (BAENet) estimates the optimal exposure time for each burst image based on a preview image, as well as motion magnitude and gain. Subsequently, a burst image restoration network reconstructs a high-quality image from burst images captured using these optimal exposure times. For training, we introduce a differentiable burst simulator and a three-stage training strategy. Our experiments demonstrate that our pipeline achieves state-of-the-art restoration quality. Furthermore, we validate the effectiveness of our approach on a real-world camera system, demonstrating its practicality.
Figures & tables
Figure 1: Overview of our pipeline. BAENet predicts the exposure times of each burst image from a preview image. Differentiable Burst Simulator generates burst images according to the exposure times. The restoration network then reconstructs a high-quality image from them. During inference, the simulator is removed, and the restoration network processes real burst images captured by our camera system.
Figure 2: Qualitative results on our test set. See the supplementary material for more results.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
Digital-Gimbal [ 4 ]
33.87
0.9309
0.187
Active S-L [ 16 ]
33.89
0.9379
0.176
Average AE [ 2 ]
34.69
0.9484
0.157
Gradient AE [ 12 ]
34.86
0.9494
0.156
Exposure Bracket [ 6 , 17 ]
35.04
0.9481
0.164
DEBIR (Ours)
35.32
0.9519
0.154
Table 1: Quantitative comparisons on our test set. The best and second-best results are in bold and blue, respectively.
Figure 3: Analysis of predicted exposure times. (a) Scatter plots of exposure times of each frame ti and motion magnitude mp . (b) Exposure time histograms of the first frame ( t1 ) for the minimum and maximum preview image gains gp . The unit of exposure times is 1/1920 sec., and gp is normalized to [0,1] .
Figure 4: Qualitative results using a real-world camera system.
Methods
NIQE ↓ [ 24 ]
BRISQUE ↓ [ 23 ]
TOPIQ ↑ [ 9 ]
Exposure Bracket
6.57
50.66
0.339
DEBIR (Ours)
6.34
46.90
0.363
Table 2: Quantitative results using a real-world camera system.
Inputs
w/o Prev. Image
w/o Gain
w/o Motion Info.
Full
PSNR ↑
34.80
35.21
35.13
35.32
Table 3: Ablation study on the impact of each input in BAENet. The best and second-best results are in bold and blue, respectively.
# of burst images
2
4
6
8
Exposure Bracket [ 6 , 17 ]
33.59
35.04
35.69
35.89
DEBIR (Ours)
34.22
35.32
35.84
36.11
Table 4: Analysis of the impact of the number of burst images on performance in terms of PSNR. The best and second-best results are in bold and blue, respectively.
Training Strategy
PSNR ↑
SSIM ↑
LPIPS ↓
S1, S2-2
34.93
0.9482
0.162
S1, S2-1, S2-2
35.01
0.9489
0.160
S1, S2-2, S3
35.16
0.9502
0.157
S1, S2-1, S2-2, S3
35.32
0.9519
0.154
S1, S2-1, E2E
34.99
0.9496
0.158
S1, S2-1, S2-2, E2E
35.11
0.9500
0.158
Table 5: Ablation study on training strategy. S1: Pre-training the restoration network. S2-1: Training BAENet with Lwarm-up . S2-2: Training BAENet with LDEBIR . S3: Fine-tuning the restoration network. E2E: end-to-end training of BAENet and the restoration network. The best and second-best results are in bold and blue.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Components
Motion estimation
BAENet
Burst image restoration
FLOPs (G)
65
6
1769
Appendix
Table S1 : Computational cost of each module in our pipeline.
Figure S1 : Qualitative comparison with another predefined-exposure schedule on our test set.
Metric
{8,24,40,56}
{8,16,32,64}
DEBIR (Ours)
PSNR ↑
35.04
35.07
35.32
SSIM ↑
0.9481
0.9482
0.9519
LPIPS ↓
0.164
0.163
0.154
Appendix
Table S2 : Quantitative comparison with another predefined-exposure schedule on our test set. The best and second-best results are in bold and blue, respectively.
Figure S2 : Qualitative comparison with Exposure Bracket [ 6 , 17 ] on the new test set with stronger noise.
Methods
Original (Tab. 1 )
New
1
2
3
4
5
{8,24,40,56}
35.75
35.40
35.15
34.35
34.70
32.04
{8,16,32,64}
35.81
35.43
35.20
34.36
34.71
32.00
DEBIR (Ours)
35.97
35.64
35.43
34.65
35.03
32.87
Appendix
Table S3 : PSNR comparison with Exposure Bracket. Columns 1–5 correspond to ISO-based subsets of our test set, while column 6 corresponds to a new test set with stronger noise. The best and second-best results are shown in bold and blue, respectively.
Test set
Dig. Gim. [ 4 ]
Act. S-L [ 16 ]
Avg. AE [ 2 ]
Grad. AE [ 12 ]
Exp. Brk. [ 6 , 17 ]
DEBIR (Ours)
REDS + Basler
32.53
32.44
33.44
33.68
33.60
33.86
REDS + Pixel
31.44
31.26
32.29
32.44
32.45
32.50
Appendix
Table S4 : Quantitative comparison of PSNR on new test sets. The best and second-best results are in bold and blue, respectively.
i -th burst image
w/o I1
w/o I2
w/o I3
w/o I4
Full
PSNR ↑
13.71
32.12
33.91
34.89
35.32
SSIM ↑
0.1059
0.9161
0.9367
0.9461
0.9519
LPIPS ↓
0.627
0.198
0.183
0.167
0.154
Appendix
Table S5 : Analysis of the impact of each burst image on performance. Here, Ii refers to the i -th burst image. The best and second-best results are in bold and blue, respectively.
Figure S3 : Additional qualitative results on our test set.
Figure S4 : Additional qualitative results using a real-world camera system.