Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
Organizations: Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University · ETH Zürich
Abstract
Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement in the coded aperture snapshot spectral imaging (CASSI) system. The HSI representations are highly similar and correlated across the spectral dimension. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. However, existing CNN-based methods show limitations in capturing spectral-wise similarity and long-range dependencies. Besides, the HSI information is modulated by a coded aperture (physical mask) in CASSI. Nonetheless, current algorithms have not fully explored the guidance effect of the mask for HSI restoration. In this paper, we propose a novel framework, Mask-guided Spectral-wise Transformer (MST), for HSI reconstruction. Specifically, we present a Spectral-wise Multi-head Self-Attention (S-MSA) that treats each spectral feature as a token and calculates self-attention along the spectral dimension. In addition, we customize a Mask-guided Mechanism (MM) that directs S-MSA to pay attention to spatial regions with high-fidelity spectral representations. Extensive experiments show that our MST significantly outperforms state-of-the-art (SOTA) methods on simulation and real HSI datasets while requiring dramatically cheaper computational and memory costs. Code and pre-trained models are available at https://github.com/caiyuanhao1998/MST/ . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST
Figures & tables
| TwIST [ 4 ] | GAP-TV [ 57 ] | DeSCI [ 27 ] | -net [ 36 ] | HSSP [ 49 ] | DNU [ 50 ] | DIP-HSI [ 35 ] | TSA-Net [ 33 ] | DGSMP [ 18 ] | MST-S | MST-M | MST-L | |||||||||||||
| Scene | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM |
| 1 | 25.16 | 0.700 | 26.82 | 0.754 | 27.13 | 0.748 | 30.10 | 0.849 | 31.48 | 0.858 | 31.72 | 0.863 | 32.68 | 0.890 | 32.03 | 0.892 | 33.26 | 0.915 | 34.71 | 0.930 | 35.15 | 0.937 | 35.40 | 0.941 |
| 2 | 23.02 | 0.604 | 22.89 | 0.610 | 23.04 | 0.620 | 28.49 | 0.805 | 31.09 | 0.842 | 31.13 | 0.846 | 27.26 | 0.833 | 31.00 | 0.858 | 32.09 | 0.898 | 34.45 | 0.925 | 35.19 | 0.935 | 35.87 | 0.944 |
| 3 | 21.40 | 0.711 | 26.31 | 0.802 | 26.62 | 0.818 | 27.73 | 0.870 | 28.96 | 0.823 | 29.99 | 0.845 | 31.30 | 0.914 | 32.25 | 0.915 | 33.06 | 0.925 | 35.32 | 0.943 | 36.26 | 0.950 | 36.51 | 0.953 |
| 4 | 30.19 | 0.851 | 30.65 | 0.852 | 34.96 | 0.897 | 37.01 | 0.934 | 34.56 | 0.902 | 35.34 | 0.908 | 40.54 | 0.962 | 39.19 | 0.953 | 40.54 | 0.964 | 41.50 | 0.967 | 42.48 | 0.973 | 42.27 | 0.973 |
| 5 | 21.41 | 0.635 | 23.64 | 0.703 | 23.94 | 0.706 | 26.19 | 0.817 | 28.53 | 0.808 | 29.03 | 0.833 | 29.79 | 0.900 | 29.39 | 0.884 | 28.86 | 0.882 | 31.90 | 0.933 | 32.49 | 0.943 | 32.77 | 0.947 |