Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction
Authors: Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, Luc Van Gool
Organizations: Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University · ETH Zürich
Many algorithms have been developed to solve the inverse problem of coded aperture snapshot spectral imaging (CASSI), i.e., recovering the 3D hyperspectral images (HSIs) from a 2D compressive measurement. In recent years, learning-based methods have demonstrated promising performance and dominated the mainstream research direction. However, existing CNN-based methods show limitations in capturing long-range dependencies and non-local self-similarity. Previous Transformer-based methods densely sample tokens, some of which are uninformative, and calculate the multi-head self-attention (MSA) between some tokens that are unrelated in content. This does not fit the spatially sparse nature of HSI signals and limits the model scalability. In this paper, we propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST), firstly embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing. Comprehensive experiments show that our CST significantly outperforms state-of-the-art methods while requiring cheaper computational costs. The code and models will be released at https://github.com/caiyuanhao1998/MST . An online Gradio demo is available at https://huggingface.co/spaces/CaiYuanhao/MST
Figures & tables
Figure 1: Diagram of our coarse-to-fine learning scheme. (a) HSIs exhibit high spatial sparsity. The image is firstly partitioned into non-overlapping patches. Then the informative patches (yellow) with HSI signals are screened out and fed into the self-attention mechanism. (b) Tokens closely correlated in content are clustered into the same bucket ( B1∼B5 ). Then the self-attention is calculated inside each bucket .
Figure 2: A schematic diagram of CASSI.
Figure 3: Framework of our method. (a) Spectra-aware screening mechanism (SASM) and the network architecture of CST. (b) The components of spectra-aware hashing attention block (SAHAB), which is the basic unit of CST. (c) spectra-aggregation hashing multi-head self-attention (SAH-MSA) is the key component embedding in SAHAB.
Algorithms
Params
GFLOPS
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
Avg
TwIST [ 3 ]
-
-
25.16 0.700
23.02 0.604
21.40 0.711
30.19 0.851
21.41 0.635
20.95 0.644
22.20 0.643
21.82 0.650
22.42 0.690
22.67 0.569
23.12 0.669
GAP-TV [ 94 ]
-
-
26.82 0.754
22.89 0.610
26.31 0.802
30.65 0.852
23.64 0.703
21.85 0.663
23.76 0.688
21.98 0.655
22.63 0.682
23.10 0.584
24.36 0.669
DeSCI [ 50 ]
-
-
27.13 0.748
23.04 0.620
26.62 0.818
34.96 0.897
23.94 0.706
22.38 0.683
24.45 0.743
22.03 0.673
24.56 0.732
23.59 0.587
25.27 0.721
λ -net [ 67 ]
62.64M
117.98
30.10 0.849
28.49 0.805
27.73 0.870
37.01 0.934
26.19 0.817
28.64 0.853
26.47 0.806
26.09 0.831
27.50 0.826
27.13 0.816
28.53 0.841
HSSP [ 81 ]
-
-
31.48 0.858
31.09 0.842
28.96 0.823
34.56 0.902
28.53 0.808
30.83 0.877
28.71 0.824
30.09 0.881
30.43 0.868
28.78 0.842
30.35 0.852
DNU [ 82 ]
1.19M
163.48
31.72 0.863
31.13 0.846
29.99 0.845
35.34 0.908
29.03 0.833
30.87 0.887
28.99 0.839
30.13 0.885
31.03 0.876
29.14 0.849
30.74 0.863
Table 1: Comparisons of Params, FLOPS, PSNR (upper entry in each cell), and SSIM (lower entry in each cell) of different methods on 10 simulation scenes (S1 ∼ S10). Best results are in bold. * denotes setting the sparsity ratio to 0.
Figure 4: Reconstructed simulation HSI comparisons of Scene 2 with 4 out of 28 spectral channels. 7 SOTA methods and our CST-L are included. The spectral curves (bottom-left) are corresponding to the selected green box of the RGB image. Please zoom in.
Figure 5: CST vs . MST.
Figure 6: Reconstructed real HSI comparisons of Scene 1 with 4 out of 28 spectral channels. Seven SOTA algorithms and our CST-L are included. Our CST-L are more favorable to restore detailed contents and remove noise. Zoom in for a better view.
Table 2: Ablations. Models are trained on CAVE and test on KAIST in simulation.
Figure 7: Visual analysis of uniform sparsity scheme, random sparsity scheme, and our SASM. We visualize the sparsity masks produced by different sparsity schemes. Yellow indicates the patch is selected while green means vice versa. Only our SASM can generate accurate response to the informative regions with HSI information.
Figure 8: Parameter analysis of sparsity ratio σ , round number R , patch size M , and loss weight λ . The vertical axis is PNSR (in dB performance). The circle radius is FLOPS (computational cost). To achieve a better trade-off between performance and computational complexity, we finally set σ = 0.5, R = 2, M = 16, and λ = 2.
Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement in the coded aperture snapshot spectral imaging (CASSI) system. The HSI representations are highly similar and correlated across the spectral dimension. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. However, existing CNN-based methods show limitations in capturing spectral-wise similarity and long-range dependencies. Besides, the HSI information is modulated by a coded aperture (physical mask) in CASSI. Nonetheless, current algorithms have not fully explored the guidance effect of the mask for HSI restoration. In this paper, we propose a novel framework, Mask-guided Spectral-wise Transformer (MST), for HSI reconstruction. Specifically, we present a Spectral-wise Multi-head Self-Attention (S-MSA) that treats each spectral feature as a token and calculates self-attention along the spectral dimension. In addition, we customize a Mask-guided Mechanism (MM) that directs S-MSA to pay attention to spatial regions with high-fidelity spectral representations. Extensive experiments show that our MST significantly outperforms state-of-the-art (SOTA) methods on simulation and real HSI datasets while requiring dramatically cheaper computational and memory costs. Code and pre-trained models are available at https://github.com/caiyuanhao1998/MST/ . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST
Yuanhao Cai, Jing Lin, Xiaowan Hu +5
Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University +1
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. Experiments show that DAUHST significantly surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models will be released at https://github.com/caiyuanhao1998/MST . An online Gradio demo is available at https://huggingface.co/spaces/CaiYuanhao/MST
Yuanhao Cai, Jing Lin, Haoqian Wang +5
Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University +2
Existing leading methods for spectral reconstruction (SR) focus on designing deeper or wider convolutional neural networks (CNNs) to learn the end-to-end mapping from the RGB image to its hyperspectral image (HSI). These CNN-based methods achieve impressive restoration performance while showing limitations in capturing the long-range dependencies and self-similarity prior. To cope with this problem, we propose a novel Transformer-based method, Multi-stage Spectral-wise Transformer (MST++), for efficient spectral reconstruction. In particular, we employ Spectral-wise Multi-head Self-attention (S-MSA) that is based on the HSI spatially sparse while spectrally self-similar nature to compose the basic unit, Spectral-wise Attention Block (SAB). Then SABs build up Single-stage Spectral-wise Transformer (SST) that exploits a U-shaped structure to extract multi-resolution contextual information. Finally, our MST++, cascaded by several SSTs, progressively improves the reconstruction quality from coarse to fine. Comprehensive experiments show that our MST++ significantly outperforms other state-of-the-art methods. In the NTIRE 2022 Spectral Reconstruction Challenge, our approach won the First place. Code and pre-trained models are publicly available at https://github.com/caiyuanhao1998/MST-plus-plus . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST-plus-plus
Yuanhao Cai, Jing Lin, Zudi Lin +5
Shenzhen International Graduate School, Tsinghua University · Harvard University · CVL, ETH Zürich +1