Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
Authors: Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Henghui Ding, Yulun Zhang, Radu Timofte, Luc Van Gool
Organizations: Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University · ETH Zürich · University of Würzburg
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. Experiments show that DAUHST significantly surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models will be released at https://github.com/caiyuanhao1998/MST . An online Gradio demo is available at https://huggingface.co/spaces/CaiYuanhao/MST
Figures & tables
Figure 1: PSNR-FLOPS comparisons of DAUHST and SOTA unfolding methods.
Figure 2: The architecture of our DAUF with K stages (iterations). E estimates informative parameters from the compressed measurement y and sensing matrix Φ . The estimated parameters α and β are fed into each stage of subsequent iterative learning. P and D denote the linear projection and denoising network in each stage.
Figure 3: Diagram of HST. (a) HST adopts a U-shaped structure. (b) HSAB consists of an FFN, an HS-MSA, and two layer normalization. (c) Components of FFN. (d) HS-MSA contains localbranch and non - localbranch .
Algorithms
Params
GFLOPS
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
Avg
TwIST [ 90 ]
-
-
25.16 0.700
23.02 0.604
21.40 0.711
30.19 0.851
21.41 0.635
20.95 0.644
22.20 0.643
21.82 0.650
22.42 0.690
22.67 0.569
23.12 0.669
GAP-TV [ 30 ]
-
-
26.82 0.754
22.89 0.610
26.31 0.802
30.65 0.852
23.64 0.703
21.85 0.663
23.76 0.688
21.98 0.655
22.63 0.682
23.10 0.584
24.36 0.669
DeSCI [ 27 ]
-
-
27.13 0.748
23.04 0.620
26.62 0.818
34.96 0.897
23.94 0.706
22.38 0.683
24.45 0.743
22.03 0.673
24.56 0.732
23.59 0.587
25.27 0.721
λ -Net [ 41 ]
62.64M
117.98
30.10 0.849
28.49 0.805
27.73 0.870
37.01 0.934
26.19 0.817
28.64 0.853
26.47 0.806
26.09 0.831
27.50 0.826
27.13 0.816
28.53 0.841
HSSP [ 44 ]
-
-
31.48 0.858
31.09 0.842
28.96 0.823
34.56 0.902
28.53 0.808
30.83 0.877
28.71 0.824
30.09 0.881
30.43 0.868
28.78 0.842
30.35 0.852
DNU [ 43 ]
1.19M
163.48
31.72 0.863
31.13 0.846
29.99 0.845
35.34 0.908
29.03 0.833
30.87 0.887
28.99 0.839
30.13 0.885
31.03 0.876
29.14 0.849
30.74 0.863
Table 1: Comparisons between DAUHST and SOTA methods on 10 simulation scenes (S1 ∼ S10). Params, FLOPS, PSNR (upper entry in each cell), and SSIM (lower entry in each cell) are reported.
Figure 4: Simulation HSI reconstruction comparisons of Scene 2 with 4 (out of 28) spectral channels. The top-middle shows the spectral curves corresponding to the two green boxes of the RGB image. The top-right depicts the enlarged patches corresponding to the yellow boxes in the bottom HSIs. Zoom in for a better view.
Figure 5: Real HSI reconstruction results of DAUHST-3stg and 9 SOTA methods on Scene 1 with 4 (out of 28) spectra. Only our method can clearly reconstruct the picked flower at all wavelengths. Zoom in for a better view.
Figure 6: Visualization of zk and xk with 4 (out of 28) spectral channels on Scene 1 in different iterations. The bottom-left corner plots the curves of α and β changing with the iteration. Please zoom in for a better view.
Table 2: Ablation studies on simulation datasets [ 92 , 93 ] . PSNR, SSIM, Params, and FLOPS are reported.
Image reconstruction recovers images, volumes, or scenes from incomplete, noisy, indirect, or compressed measurements using physical models and learned priors.
Many algorithms have been developed to solve the inverse problem of coded aperture snapshot spectral imaging (CASSI), i.e., recovering the 3D hyperspectral images (HSIs) from a 2D compressive measurement. In recent years, learning-based methods have demonstrated promising performance and dominated the mainstream research direction. However, existing CNN-based methods show limitations in capturing long-range dependencies and non-local self-similarity. Previous Transformer-based methods densely sample tokens, some of which are uninformative, and calculate the multi-head self-attention (MSA) between some tokens that are unrelated in content. This does not fit the spatially sparse nature of HSI signals and limits the model scalability. In this paper, we propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST), firstly embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing. Comprehensive experiments show that our CST significantly outperforms state-of-the-art methods while requiring cheaper computational costs. The code and models will be released at https://github.com/caiyuanhao1998/MST . An online Gradio demo is available at https://huggingface.co/spaces/CaiYuanhao/MST
Yuanhao Cai, Jing Lin, Xiaowan Hu +5
Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University +1
Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement in the coded aperture snapshot spectral imaging (CASSI) system. The HSI representations are highly similar and correlated across the spectral dimension. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. However, existing CNN-based methods show limitations in capturing spectral-wise similarity and long-range dependencies. Besides, the HSI information is modulated by a coded aperture (physical mask) in CASSI. Nonetheless, current algorithms have not fully explored the guidance effect of the mask for HSI restoration. In this paper, we propose a novel framework, Mask-guided Spectral-wise Transformer (MST), for HSI reconstruction. Specifically, we present a Spectral-wise Multi-head Self-Attention (S-MSA) that treats each spectral feature as a token and calculates self-attention along the spectral dimension. In addition, we customize a Mask-guided Mechanism (MM) that directs S-MSA to pay attention to spatial regions with high-fidelity spectral representations. Extensive experiments show that our MST significantly outperforms state-of-the-art (SOTA) methods on simulation and real HSI datasets while requiring dramatically cheaper computational and memory costs. Code and pre-trained models are available at https://github.com/caiyuanhao1998/MST/ . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST
Yuanhao Cai, Jing Lin, Xiaowan Hu +5
Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University +1
Hyperspectral Image (HSI) reconstruction has made substantial progress with the deep unfolding framework by decomposing the problem into a projection module and a denoiser module. Nevertheless, existing methods still exhibit limitations of insufficient matching with HSI data. The issues lie in two aspects: 1) the data-consistency projection module applies a fixed gradient descent step while ignoring the spectral content adaptivity of HSI; 2) the denoiser module either suffers from high computational complexity or is limited to the local receptive field, failing to efficiently capture the global spatial-spectral dependencies. In this paper, we propose a deep unfolding framework centered on a two-stage manifold learning strategy to obtain a degradation-free structural prior, which is then strategically injected into both the projection and denoiser modules. In the data-consistency projection, a prior-guided spectral weight mechanism is introduced to adaptively modulate the weight of spectral band. In the denoiser module, a manifold-guided Mamba is proposed to achieve the structure-aware extraction of spatial-spectral features through prior modulation. Quantitative and visual comparisons on synthetic and real-world datasets show that the proposed method significantly outperforms existing state-of-the-art approaches, while maintaining the structural interpretability of the unfolding framework. The code is available at https://github.com/SqLiu67/MPG-DU.
Shuqi Liu, Yingkai Zhang, Lin Gu +2
Beijing Institute of Technology · Tohoku University · Hangzhou Dianzi University