eess.IVNov 15, 2021

Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction

Authors: Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, Luc Van Gool

Organizations: Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University · ETH Zürich

Abstract

Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement in the coded aperture snapshot spectral imaging (CASSI) system. The HSI representations are highly similar and correlated across the spectral dimension. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. However, existing CNN-based methods show limitations in capturing spectral-wise similarity and long-range dependencies. Besides, the HSI information is modulated by a coded aperture (physical mask) in CASSI. Nonetheless, current algorithms have not fully explored the guidance effect of the mask for HSI restoration. In this paper, we propose a novel framework, Mask-guided Spectral-wise Transformer (MST), for HSI reconstruction. Specifically, we present a Spectral-wise Multi-head Self-Attention (S-MSA) that treats each spectral feature as a token and calculates self-attention along the spectral dimension. In addition, we customize a Mask-guided Mechanism (MM) that directs S-MSA to pay attention to spatial regions with high-fidelity spectral representations. Extensive experiments show that our MST significantly outperforms state-of-the-art (SOTA) methods on simulation and real HSI datasets while requiring dramatically cheaper computational and memory costs. Code and pre-trained models are available at https://github.com/caiyuanhao1998/MST/ . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST

Figures & tables

Explore similar work

CardsList
  1. Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

    Mar 9, 2022Yuanhao Cai, Jing Lin, Xiaowan Hu +5Hyperspectral Image ReconstructionImage Reconstruction

  2. MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction

    Apr 17, 2022Yuanhao Cai, Jing Lin, Zudi Lin +5Hyperspectral Image ReconstructionHyperspectral Imaging

  3. Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging

    May 20, 2022Yuanhao Cai, Jing Lin, Haoqian Wang +5Image ReconstructionHyperspectral Image Reconstruction