Organizations: School of Electronic Information, Wuhan University, Wuhan 430072, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China
By combining complementary benefits of short- and long-exposure images, Dual-Exposure Imaging (DEI) enhances image quality in low-light scenarios. However, existing DEI approaches inevitably suffer from producing artifacts due to spatial displacement from scene motion and image feature discrepancies from different exposure times. To tackle this problem, we propose a novel Event-based DEI (E-DEI) algorithm, which reconstructs high-quality images from dual-exposure image pairs and events, leveraging high temporal resolution of event cameras to provide accurate inter-/intra-frame dynamic information. Specifically, we decompose this complex task into an integration of two sub-tasks, i.e., event-based motion deblurring and low-light image enhancement tasks, which guides us to design E-DEI network as a dual-path parallel feature propagation architecture. We propose a Dual-path Feature Alignment and Fusion (DFAF) module to effectively align and fuse features extracted from dual-exposure images with assistance of events. Furthermore, we build a real-world Dataset containing Paired low-/normal-light Images and Events (PIED). Experiments on multiple datasets show the superiority of our method. The code and dataset are available at github.
Figures & tables
Fig. 1: (a) Illustration of the proposed Event-based Dual-Exposure Imaging (E-DEI) strategy. (b-c) The short- and long-exposure image pairs are continuously captured under the dynamic scenes. The corresponding image reconstruction results via (d) the state-of-the-art Frame-based DEI (F-DEI) method and (e) our E-DEI. Due to the missing inter/intra-frame motion information, F-DEI suffers from texture blur and artifacts, while E-DEI can successfully generate the high-quality visual images.
Fig. 2: Comparison between (a) existing Frame-based Dual-Exposure Imaging (F-DEI) and (b) our Event-based Dual-Exposure Imaging (E-DEI). The key idea of our E-DEI is to introduce the high temporal resolution event data to complement the missing intra- and inter-frame motion information. As a result, our algorithm produces sharp and clear images without artifacts.
Fig. 3: (a) The overall architecture of the proposed network of Event-based Dual-Exposure Imaging (E-DEI), which is a dual-path feature propagation and supervision framework. (b) The proposed Dual-path Feature Alignment and Fusion (DFAF) module, where features of the Deblurring path are spatially aligned to the short-exposure target time with features of the Enhancement path as the temporal reference, and then we integrate features of the two paths for the high-quality reconstruction.
Fig. 4: Details of the proposed Cross-Guided Fusion (CGF) module, which is designed to fuse the results and features of the Deblurring and Enhancement paths for the final result.
Fig. 5: (a) The illustration of a beam splitter-based hybrid camera system for building the PIED dataset. (b) PIED contains the normal- and low-light image pairs (upper row) and the corresponding events (bottom row) with three diverse motion patterns.
Dataset
Image
Resolution
FPS
Motion pattern
LL
NL
SDE [ 21 ]
✓
✓
346 × 260
< 30
Camera motion
EVFI-LL [ 20 ]
✓
✗
784 × 611
64
Moving object
RLED [ 19 ]
✗
✓
1120 × 660
10,25
Camera+Object
Ours
✓
✓
1280 × 720
72
Camera+Object
TABLE I: Comparisons of our dataset with the existing event-based low-light datasets. LL/NL means low-/normal-light images
Fig. 6: Statistical properties of the E-DEI dataset. Distributions of (a) Motion magnitude, (b) Illumination, (c) Texture richness, and (d) Event firing rate.
Method
Event
REDS-LL
PIED
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
Low-light image enhancement
Uretinex-Net [ 23 ]
✗
25.621
0.9020
25.829
0.8613
EMNet [ 70 ]
✗
30.317
0.9253
26.776
0.8982
EvLowLight [ 18 ]
✓
25.519
0.9217
26.808
0.8666
EvLight [ 21 ]
✓
31.211
0.9375
28.203
0.9102
Motion deblurring
MSSNet [ 58 ]
✗
30.181
0.8883
28.890
0.8538
TABLE II: Quantitative comparisons of the proposed E-DEI to the state-of-the-art methods on the REDS-LL and PIED datasets. The best performance is highlighted in bold and second best is underlined .
Fig. 7: Qualitative comparisons with state-of-the-art methods on the REDS-LL (upper low) and PIED (lower row) datasets. For a better view, details are zoomed in and the brightness of short-exposure images is rescaled.
Ex.
Event
PSNR ↑
SSIM ↑
Deblurring
Enhancement
1
✗
✗
34.522
0.9471
2
✓
✗
37.756
0.9684
3
✗
✓
36.281
0.9601
4
✓
✓
37.895
0.9695
TABLE III: Ablation study on the feeding strategies of the event streams.
Architecture
L^s
L^
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
Serial pipeline
35.984
0.9581
36.702
0.9658
Parallel design (Ours)
37.895
0.9695
37.904
0.9695
TABLE IV: Quantitative comparisons of the proposed E-DEI, which is a parallel architecture, to the serial pipeline design.
Ex.
Backbone
DFAF module
PSNR ↑
SSIM ↑
Enc.
Dec.
DA
CAF
1
SP
SP
✗
✗
37.376
0.9657
2
DP
SP
✓
✓
37.637
0.9673
3
DP
DP
✗
✗
36.199
0.9591
4
DP
DP
✓
✗
35.876
0.9645
5
DP
DP
✗
✓
37.624
0.9674
TABLE V: Ablation study of the dual-path backbone and modules of our network on the REDS-LL dataset. Enc. and Dec. indicate the architecture of the encoder and decoder. DP and SP mean the dual- and single-path backbone.
Fig. 8: Qualitative ablations of the backbone and modules of our network and their absolute differences to the ground-truth clear image. Enc. and Dec. indicate the architecture of the encoder and decoder.
Fig. 9: Visualization of intermediate features from the DFAF module of our E-DEI. (a) and (d) the input dual-exposure image pairs. (b-c) Fl and Fl′ are the features before and after the DA block in the Delurring path. (e-f) Fs and Fs′ are the features before and after the CAF block in the Enhancement path.
Fig. 10: Evaluation of our E-DEI supervised by Eq. 20 with the different values of the balancing parameter λ3 in the first stage training.
Fig. 11: (a) Sensitivity analysis of the E-DEI network to the temporal misalignment between cross-modal image pairs and event streams. (b) Generalization of the E-DEI network to the unseen exposure-time ratios.
Method
Event
Param
Runtime
FLOPs
(M) ↓
( ms ) ↓
(G) ↓
Uretinex-Net [ 23 ]
✗
0.34
123.7
1601.25
EMNet [ 70 ]
✗
12.52
774.4
1036.33
EvLowLight [ 18 ]
✓
15.03
766.9
4091.00
EvLight [ 21 ]
✓
22.73
674.6
2731.53
MSSNet [ 58 ]
✗
15.59
5.4
2040.46
TABLE VI: Quantitative comparisons of the network parameter and inference efficiency on the REDS-LL datasets. The items of runtime and FLOPs are evaluated with the input images and events at the resolution of 1280×720 .
Fig. 12: Failure cases under non-ideal scenes, e.g. , extremely dim environment (upper), blocked and discarded events (middle), and coupled degradations (lower).
National Key Lab of Multispectral Information Intelligent Processing Technology School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore
College of Computer and Data Science, Fuzhou University, Fuzhou 350108, China · College of Computer Science and Technology, Shandong Technology and Business University, Yantai 264003, China