MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction
Organizations: Shenzhen International Graduate School, Tsinghua University · Harvard University · CVL, ETH Zürich · CAIDAS, JMU Würzburg
Abstract
Existing leading methods for spectral reconstruction (SR) focus on designing deeper or wider convolutional neural networks (CNNs) to learn the end-to-end mapping from the RGB image to its hyperspectral image (HSI). These CNN-based methods achieve impressive restoration performance while showing limitations in capturing the long-range dependencies and self-similarity prior. To cope with this problem, we propose a novel Transformer-based method, Multi-stage Spectral-wise Transformer (MST++), for efficient spectral reconstruction. In particular, we employ Spectral-wise Multi-head Self-attention (S-MSA) that is based on the HSI spatially sparse while spectrally self-similar nature to compose the basic unit, Spectral-wise Attention Block (SAB). Then SABs build up Single-stage Spectral-wise Transformer (SST) that exploits a U-shaped structure to extract multi-resolution contextual information. Finally, our MST++, cascaded by several SSTs, progressively improves the reconstruction quality from coarse to fine. Comprehensive experiments show that our MST++ significantly outperforms other state-of-the-art methods. In the NTIRE 2022 Spectral Reconstruction Challenge, our approach won the First place. Code and pre-trained models are publicly available at https://github.com/caiyuanhao1998/MST-plus-plus . An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/MST-plus-plus
Figures & tables
| MSA Scheme | Global MSA | Local W-MSA | S-MSA |
| Receptive Field | Global | Local | Global |
| Complexity to | Quadratic | Linear | Linear |
| Calculating Wise | Spatial | Spatial | Spectral |
| NTIRE 2022 HSI Dataset - Valid | NTIRE 2022 HSI Dataset - Test | |||||||
| Method | Params (M) | FLOPS (G) | MRAE | RMSE | PSNR | Username | MRAE | RMSE |
| HSCNN+ [ 67 ] | 4.65 | 304.45 | 0.3814 | 0.0588 | 26.36 | pipixia | 0.2434 | 0.0411 |
| HRNet [ 88 ] | 31.70 | 163.81 | 0.3476 | 0.0550 | 26.89 | uslab | 0.2377 | 0.0391 |
| EDSR [ 45 ] | 2.42 | 158.32 | 0.3277 | 0.0437 | 28.29 | orange_dog | 0.2377 | 0.0376 |
| AWAN [ 36 ] | 4.04 | 270.61 | 0.2500 | 0.0367 | 31.22 | askldklasfj | 0.2345 | 0.0361 |
| HDNet [ 29 ] | 2.66 | 173.81 | 0.2048 | 0.0317 | 32.13 | HSHAJii | 0.2308 | 0.0364 |