Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
Organizations: Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University · ETH Zürich ·
Abstract
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. Experiments show that DAUHST significantly surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models will be released at https://github.com/caiyuanhao1998/MST . An online Gradio demo is available at https://huggingface.co/spaces/CaiYuanhao/MST
Figures & tables
| Algorithms | Params | GFLOPS | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | S10 | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TwIST [ 90 ] | - | - | 25.16 0.700 | 23.02 0.604 | 21.40 0.711 | 30.19 0.851 | 21.41 0.635 | 20.95 0.644 | 22.20 0.643 | 21.82 0.650 | 22.42 0.690 | 22.67 0.569 | 23.12 0.669 |
| GAP-TV [ 30 ] | - | - | 26.82 0.754 | 22.89 0.610 | 26.31 0.802 | 30.65 0.852 | 23.64 0.703 | 21.85 0.663 | 23.76 0.688 | 21.98 0.655 | 22.63 0.682 | 23.10 0.584 | 24.36 0.669 |
| DeSCI [ 27 ] | - | - | 27.13 0.748 | 23.04 0.620 | 26.62 0.818 | 34.96 0.897 | 23.94 0.706 | 22.38 0.683 | 24.45 0.743 | 22.03 0.673 | 24.56 0.732 | 23.59 0.587 | 25.27 0.721 |
| -Net [ 41 ] | 62.64M | 117.98 | 30.10 0.849 | 28.49 0.805 | 27.73 0.870 | 37.01 0.934 | 26.19 0.817 | 28.64 0.853 | 26.47 0.806 | 26.09 0.831 | 27.50 0.826 | 27.13 0.816 | 28.53 0.841 |
| HSSP [ 44 ] | - | - | 31.48 0.858 | 31.09 0.842 | 28.96 0.823 | 34.56 0.902 | 28.53 0.808 | 30.83 0.877 | 28.71 0.824 | 30.09 0.881 | 30.43 0.868 | 28.78 0.842 | 30.35 0.852 |
| DNU [ 43 ] | 1.19M | 163.48 | 31.72 0.863 | 31.13 0.846 | 29.99 0.845 | 35.34 0.908 | 29.03 0.833 | 30.87 0.887 | 28.99 0.839 | 30.13 0.885 | 31.03 0.876 | 29.14 0.849 | 30.74 0.863 |
Image Reconstruction
Image reconstruction recovers images, volumes, or scenes from incomplete, noisy, indirect, or compressed measurements using physical models and learned priors.
See all 323 papers