cs.CVMar 9, 2022

Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

Authors: Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, Luc Van Gool

Organizations: Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University · ETH Zürich

Abstract

Many algorithms have been developed to solve the inverse problem of coded aperture snapshot spectral imaging (CASSI), i.e., recovering the 3D hyperspectral images (HSIs) from a 2D compressive measurement. In recent years, learning-based methods have demonstrated promising performance and dominated the mainstream research direction. However, existing CNN-based methods show limitations in capturing long-range dependencies and non-local self-similarity. Previous Transformer-based methods densely sample tokens, some of which are uninformative, and calculate the multi-head self-attention (MSA) between some tokens that are unrelated in content. This does not fit the spatially sparse nature of HSI signals and limits the model scalability. In this paper, we propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST), firstly embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing. Comprehensive experiments show that our CST significantly outperforms state-of-the-art methods while requiring cheaper computational costs. The code and models will be released at https://github.com/caiyuanhao1998/MST . An online Gradio demo is available at https://huggingface.co/spaces/CaiYuanhao/MST

Figures & tables

Explore similar work

CardsList
  1. Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction

    Nov 15, 2021Yuanhao Cai, Jing Lin, Xiaowan Hu +5Hyperspectral Image ReconstructionImage Reconstruction

  2. Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging

    May 20, 2022Yuanhao Cai, Jing Lin, Haoqian Wang +5Image ReconstructionHyperspectral Image Reconstruction

  3. MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction

    Apr 17, 2022Yuanhao Cai, Jing Lin, Zudi Lin +5Hyperspectral Image ReconstructionHyperspectral Imaging