Recently, Spectral Compressive Imaging (SCI) has achieved remarkable success, unlocking significant potential for dynamic spectral vision. However, existing reconstruction methods, primarily image-based, suffer from two limitations: (i) Encoding process masks spatial-spectral features, leading to uncertainty in reconstructing missing information from single compressed measurements, and (ii) The frame-by-frame reconstruction paradigm fails to ensure temporal consistency, which is crucial in the video perception. To address these challenges, this paper seeks to advance spectral reconstruction from the image level to the video level, leveraging the complementary features and temporal continuity across adjacent frames in dynamic scenes. Initially, we construct the first high-quality dynamic hyperspectral image dataset (DynaSpec), comprising 30 sequences obtained through frame-scanning acquisition. Subsequently, we propose the Propagation-Guided Spectral Video Reconstruction Transformer (PG-SVRT), which employs a spatial-then-temporal attention to effectively reconstruct spectral features from abundant video information, while using a bridged token to reduce computational complexity. Finally, we conduct simulation experiments to assess the performance of four SCI systems, and construct a DD-CASSI prototype for real-world data collection and benchmarking. Extensive experiments demonstrate that PG-SVRT achieves superior performance in reconstruction quality, spectral fidelity, and temporal consistency, while maintaining minimal FLOPs. Project page: https://github.com/nju-cite/DynaSpec
Figures & tables
Figure 1: Spectral compressive imaging and reconstruction. (a) SCI principle. (b) Image-based methods, with issues of uncertain reconstruction and temporal inconsistency (flickering intensity curves). (c) Video-based reconstruction, where information complementarity enhances completeness and temporal consistency (smooth intensity curves).
Figure 2: The proposed DynaSpec dataset. (a) Dynamic HSIs sequences acquired frame by frame to simulate the diverse motion of real-world scenarios. (b) A display of the 30 scenes.
Figure 3: Illustration of PG-SVRT. (a) and (c) The components of MGDP and CDBP. (b) PG-SVRT framework and key components.
Figure 4: Details of the CDPB, which consists primarily of CDPA and MDFFN. (a) CDPA is a spatial-then-temporal attention mechanism, where the blue line represents spatial feature processing and the red line indicates temporal feature processing. (b) Illustration of MDFFN.
PMVIS
SD-CASSI
NDSSI
DD-CASSI
PSNR ↑
28.45
37.78
37.84
41.52
SSIM ↑
0.8456
0.9700
0.9825
0.9893
SAM ↓
5.4162
4.0737
5.4091
3.9084
ST-RRED ↓
459.49
23.21
91.8
23.25
Table 1: Quantitative comparison of representative SCI architectures. All systems are solved via PG-SVRT under a unified mathematical framework to exclude algorithmic bias.
Figure 5: Measurements of different SCI systems
Method
MST-L CVPR 2022
CST-L ECCV 2022
DAUHST NeurIPS 2022
GAP-Net IJCV 2023
DADF-Plus-3 TMI 2023
RDLUF CVPR 2023
PADUT ICCV 2023
S2 -Transfor. TPAMI 2024
SSR CVPR 2024
DPU CVPR 2024
DPU ∗ CVPR 2024
PG-SVRT Ours
PSNR-K ↑
39.99
39.93
38.98
36.92
38.23
39.26
38.61
33.26
39.04
40.02
40.50
41.23
SSIM-K ↑
0.9881
0.9864
0.9832
0.9755
0.9832
0.9860
0.9828
0.9617
0.9842
0.9856
0.9853
0.9882
SAM-K ↓
3.8248
4.1342
5.4514
6.1204
4.7676
4.2932
4.7154
8.0837
5.2201
5.2250
5.1685
3.805
ST-RRED-K ↓
30.99
35.11
37.27
85.34
48.45
39.06
47.19
155.82
38.29
25.90
26.71
19.35
PSNR-D ↑
39.58
40.06
40.39
39.38
39.00
39.26
40.41
37.10
39.66
41.01
41.36
41.82
SSIM-D ↑
0.9873
0.9876
0.9883
0.9851
0.9861
0.9863
0.9881
0.9786
0.9873
0.9893
0.9889
0.9904
Table 2: Quantitative comparisons of several SOTA methods and PG-SVRT. The suffix -K denotes results on the KAIST, while -D represents evaluations on the DynaSpec testset. The best and second best results are highlighted in bold and underline, respectively.
Figure 6: Reconstruction results of PG-SVRT and comparison methods on the KAIST and DynaSpec test sets. The bottom-left corner of each subplot presents an enlarged detail view, while the bottom-right corner shows the difference with the GT. It is evident that, while all methods benefit from DD-CASSI and are able to recover structural details, our method achieves the superior fidelity.
Figure 7: Reconstruction of real measurements using comparison methods and PG-SVRT, with pseudo-RGB images generated from the reconstructed HSIs to assess reconstruction quality across all bands. Compared to other methods, PG-SVRT results exhibit fewer artifacts.
Figure 8: Complex real-world scenes reconstructed by PG-SVRT show clear structures and fine details in both the pseudo-RGB and HSIs.
Baseline
+ CDPA
+ MGDP
+ MDFFN
PSNR
39.97
41.30
41.41
41.52
SSIM
0.9827
0.9884
0.9886
0.9893
SAM
5.5312
4.3224
4.2513
3.9084
STRRED
43.90
25.44
24.63
23.25
Params
2.17 M
1.92 M
1.92 M
2.48M
GFLOPs
30.11
21.11
21.31
28.18
Table 3: Break-down ablation study of PG-SVRT, "+" indicates adding or replacing modules relative to the baseline.
Parallel
T-S
T-S w/ P
S-T
S-T w/ P
PSNR
41.35
41.04
41.47
41.08
41.52
SSIM
0.9886
0.9877
0.9892
0.9880
0.9893
SAM
4.1518
4.4995
3.9605
4.4128
3.9084
STRRED
26.00
28.67
26.60
26.40
23.25
Params
2.85 M
2.60 M
2.48 M
2.60 M
2.48 M
GFLOPs
33.23
30.16
28.18
30.16
28.18
Table 4: Comparison of spatiotemporal processing strategies.