Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
Authors: Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, Luc Van Gool
Organizations: Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University · ETH Zürich
Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement in the coded aperture snapshot spectral imaging (CASSI) system. The HSI representations are highly similar and correlated across the spectral dimension. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. However, existing CNN-based methods show limitations in capturing spectral-wise similarity and long-range dependencies. Besides, the HSI information is modulated by a coded aperture (physical mask) in CASSI. Nonetheless, current algorithms have not fully explored the guidance effect of the mask for HSI restoration. In this paper, we propose a novel framework, Mask-guided Spectral-wise Transformer (MST), for HSI reconstruction. Specifically, we present a Spectral-wise Multi-head Self-Attention (S-MSA) that treats each spectral feature as a token and calculates self-attention along the spectral dimension. In addition, we customize a Mask-guided Mechanism (MM) that directs S-MSA to pay attention to spatial regions with high-fidelity spectral representations. Extensive experiments show that our MST significantly outperforms state-of-the-art (SOTA) methods on simulation and real HSI datasets while requiring dramatically cheaper computational and memory costs. Code and pre-trained models are available at https://github.com/caiyuanhao1998/MST/ . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST
Figures & tables
Figure 1: PSNR-Params-FLOPS comparisons of CNN-based HSI reconstruction algorithms. The vertical axis is PSNR (in dB performance), the horizontal axis is FLOPS (computational cost), and the circle radius is Params (memory cost). Our proposed Mask-guided Spectral-wise Transformers (MSTs) outperform previous methods while requiring significantly cheaper FLOPS and Params.
Figure 2: Illustration of the proposed method. Our Mask-guided Spectral-wise Multi-head Self-Attention (MS-MSA) is motivated by the HSI characteristics and CASSI system. (a) The representations of HSIs are spatially sparse while spectrally correlated. (b) The CASSI system uses a mask to modulate the HSIs. (c) Our MS-MSA in stage 0 of MST. (c1) S-MSA treats each spectral feature as a token and calculates self-attention along the spectral dimension. (c2) Mask-guided Mechanism directs the Spectral-wise MSA to pay attention to spatial regions with high-fidelity HSI representations. Some components are omitted for simplification. Please refer to the text for details.
Figure 3: The overall architecture of MST. (a) MST adopts a U-shaped structure that consists of an encoder, a bottleneck, and a decoder. (b) MSAB is composed of a Feed-Forward Network (FFN), an MS-MSA, and two layer normalization. (c) The components of FFN.
TwIST [ 4 ]
GAP-TV [ 57 ]
DeSCI [ 27 ]
λ -net [ 36 ]
HSSP [ 49 ]
DNU [ 50 ]
DIP-HSI [ 35 ]
TSA-Net [ 33 ]
DGSMP [ 18 ]
MST-S
MST-M
MST-L
Scene
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
1
25.16
0.700
26.82
0.754
27.13
0.748
30.10
0.849
31.48
0.858
31.72
0.863
32.68
0.890
32.03
0.892
33.26
0.915
34.71
0.930
35.15
0.937
35.40
0.941
2
23.02
0.604
22.89
0.610
23.04
0.620
28.49
0.805
31.09
0.842
31.13
0.846
27.26
0.833
31.00
0.858
32.09
0.898
34.45
0.925
35.19
0.935
35.87
0.944
3
21.40
0.711
26.31
0.802
26.62
0.818
27.73
0.870
28.96
0.823
29.99
0.845
31.30
0.914
32.25
0.915
33.06
0.925
35.32
0.943
36.26
0.950
36.51
0.953
4
30.19
0.851
30.65
0.852
34.96
0.897
37.01
0.934
34.56
0.902
35.34
0.908
40.54
0.962
39.19
0.953
40.54
0.964
41.50
0.967
42.48
0.973
42.27
0.973
5
21.41
0.635
23.64
0.703
23.94
0.706
26.19
0.817
28.53
0.808
29.03
0.833
29.79
0.900
29.39
0.884
28.86
0.882
31.90
0.933
32.49
0.943
32.77
0.947
Table 1: Quantitative results on 10 scenes in simulation. PSNR and SSIM are reported. MSTs significantly surpass other competitors.
Figure 4: Reconstructed simulation HSI comparisons of Scene 5 with 4 out of 28 spectral channels. 7 SOTA algorithms and our MST-L are included. The spectral curves (bottom-left) are corresponding to the selected green box of the RGB image. Zoom in for a better view.
Figure 5: Reconstructed real HSI comparisons of Scene 3 with 4 out of 28 spectral channels. Seven SOTA algorithms and our MST-L are included. MST-L reconstructs more detailed contents and suppresses more noise. Please zoom in for better visualization performance.
Table 2: Ablations. We train models on CAVE [ 39 ] and test on KAIST [ 10 ] in simulation. PSNR, SSIM, Params, and FLOPS are reported.
Figure 6: Visualization of the correlation coefficients among spectral channels of HSIs reconstructed by models using different MSAs. The correlation coefficient map of the model equipped with our S-MSA is the most similar one to that of the ground truth.
Figure 7: Visual analysis of the feature maps of the last MSAB in MST-S. The top row shows the original RGB images. The middle and bottom rows exhibit the feature maps without and with MM. The model using MM pays more high-fidelity attention to details.
Many algorithms have been developed to solve the inverse problem of coded aperture snapshot spectral imaging (CASSI), i.e., recovering the 3D hyperspectral images (HSIs) from a 2D compressive measurement. In recent years, learning-based methods have demonstrated promising performance and dominated the mainstream research direction. However, existing CNN-based methods show limitations in capturing long-range dependencies and non-local self-similarity. Previous Transformer-based methods densely sample tokens, some of which are uninformative, and calculate the multi-head self-attention (MSA) between some tokens that are unrelated in content. This does not fit the spatially sparse nature of HSI signals and limits the model scalability. In this paper, we propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST), firstly embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing. Comprehensive experiments show that our CST significantly outperforms state-of-the-art methods while requiring cheaper computational costs. The code and models will be released at https://github.com/caiyuanhao1998/MST . An online Gradio demo is available at https://huggingface.co/spaces/CaiYuanhao/MST
Yuanhao Cai, Jing Lin, Xiaowan Hu +5
Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University +1
Existing leading methods for spectral reconstruction (SR) focus on designing deeper or wider convolutional neural networks (CNNs) to learn the end-to-end mapping from the RGB image to its hyperspectral image (HSI). These CNN-based methods achieve impressive restoration performance while showing limitations in capturing the long-range dependencies and self-similarity prior. To cope with this problem, we propose a novel Transformer-based method, Multi-stage Spectral-wise Transformer (MST++), for efficient spectral reconstruction. In particular, we employ Spectral-wise Multi-head Self-attention (S-MSA) that is based on the HSI spatially sparse while spectrally self-similar nature to compose the basic unit, Spectral-wise Attention Block (SAB). Then SABs build up Single-stage Spectral-wise Transformer (SST) that exploits a U-shaped structure to extract multi-resolution contextual information. Finally, our MST++, cascaded by several SSTs, progressively improves the reconstruction quality from coarse to fine. Comprehensive experiments show that our MST++ significantly outperforms other state-of-the-art methods. In the NTIRE 2022 Spectral Reconstruction Challenge, our approach won the First place. Code and pre-trained models are publicly available at https://github.com/caiyuanhao1998/MST-plus-plus . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST-plus-plus
Yuanhao Cai, Jing Lin, Zudi Lin +5
Shenzhen International Graduate School, Tsinghua University · Harvard University · CVL, ETH Zürich +1
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. Experiments show that DAUHST significantly surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models will be released at https://github.com/caiyuanhao1998/MST . An online Gradio demo is available at https://huggingface.co/spaces/CaiYuanhao/MST
Yuanhao Cai, Jing Lin, Haoqian Wang +5
Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University +2