cs.CVApr 17, 2022

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction

Authors: Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, Radu Timofte, Luc Van Gool

Organizations: Shenzhen International Graduate School, Tsinghua University · Harvard University · CVL, ETH Zürich · CAIDAS, JMU Würzburg

Abstract

Existing leading methods for spectral reconstruction (SR) focus on designing deeper or wider convolutional neural networks (CNNs) to learn the end-to-end mapping from the RGB image to its hyperspectral image (HSI). These CNN-based methods achieve impressive restoration performance while showing limitations in capturing the long-range dependencies and self-similarity prior. To cope with this problem, we propose a novel Transformer-based method, Multi-stage Spectral-wise Transformer (MST++), for efficient spectral reconstruction. In particular, we employ Spectral-wise Multi-head Self-attention (S-MSA) that is based on the HSI spatially sparse while spectrally self-similar nature to compose the basic unit, Spectral-wise Attention Block (SAB). Then SABs build up Single-stage Spectral-wise Transformer (SST) that exploits a U-shaped structure to extract multi-resolution contextual information. Finally, our MST++, cascaded by several SSTs, progressively improves the reconstruction quality from coarse to fine. Comprehensive experiments show that our MST++ significantly outperforms other state-of-the-art methods. In the NTIRE 2022 Spectral Reconstruction Challenge, our approach won the First place. Code and pre-trained models are publicly available at https://github.com/caiyuanhao1998/MST-plus-plus . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST-plus-plus

Figures & tables

Explore similar work

CardsList
  1. Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction

    Nov 15, 2021Yuanhao Cai, Jing Lin, Xiaowan Hu +5Hyperspectral Image ReconstructionImage Reconstruction

  2. Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

    Mar 9, 2022Yuanhao Cai, Jing Lin, Xiaowan Hu +5Hyperspectral Image ReconstructionImage Reconstruction

  3. Correcting Spectra Outside the Backbone: A Model-Agnostic Rectifier for Hyperspectral Image Super-Resolution

    Jan 29, 2026Ji-Xuan He, Guohang Zhuang, Bo Junge +5Hyperspectral Image Super-ResolutionHyperspectral Imaging