Filter-Enhanced Smartphone Hyperspectral Imaging
Organizations: JMU, Würzburg · CRSI, Moscow · MIRIAI, Moscow · AXXX, Moscow
Abstract
Hyperspectral reconstruction from RGB imagery offers an affordable alternative to costly hyperspectral acquisition devices. However, its accuracy is fundamentally constrained by the limited spectral resolution of standard RGB cameras. In this paper, we propose a new approach that leverages the multi-camera systems integrated into most modern smartphones and enhances their capabilities by modulating their spectral response functions using carefully selected spectral filters, enabling a smartphone to capture richer spectral information. We propose a neural network able to exploit auxiliary RGB images with diverse spectral characteristics, enabling more accurate hyperspectral reconstruction compared to conventional single-image methods. Training is supported by a newly collected dataset, Doomer, which consists of multiple misaligned RGB images alongside corresponding hyperspectral data. We also propose a device-adaptation strategy that conditions the multi-image hyperspectral reconstruction network on camera color correction matrices available in standard DNG metadata, requiring no real images from the target device. We evaluate it on three physical smartphones, two of them unseen during training.
Figures & tables
| Method | PSNR, dB ↑ | SAM, ° ↓ | NSE, % ↓ | Params, K | GMACs |
|---|---|---|---|---|---|
| Single-image input | |||||
| AWAN | 33.91 | 4.19 | 7.31 | 436 | 84.82 |
| NAFNet | 33.84 | 4.14 | 7.36 | 376 | 7.74 |
| SPECAT | 33.63 | 4.17 | 7.34 | 351 | 53.32 |
| MST++ | 33.98 | 4.06 | 7.12 | 546 | 31.20 |
| GMST++ (ours) | 33.99 | 4.15 | 7.25 | 546 | 31.30 |
| Method | PSNR, dB ↑ | SAM, ° ↓ | NSE, % ↓ | Params, K | GMACs |
|---|---|---|---|---|---|
| Single-image input | |||||
| AWAN | 30.48 ± 0.70 | 6.41 ± 1.09 | 13.68 ± 1.79 | 525 | 25.82 |
| NAFNet | 30.63 ± 0.66 | 6.00 ± 0.72 | 13.29 ± 0.98 | 376 | 1.50 |
| SPECAT | 31.08 ± 0.86 | 5.47 ± 0.68 | 12.28 ± 1.25 | 394 | 11.21 |
| MST++ | 31.33 ± 0.62 | 5.24 ± 0.42 | 12.03 ± 0.82 | 655 | 7.35 |
| GMST++ (ours) | 31.55 ± 0.51 | 5.18 ± 0.29 | 11.71 ± 0.73 | 654 | 7.37 |
| DCAM | GDFN | PSNR, dB ↑ | SAM, ° ↓ | NSE, % ↓ |
|---|---|---|---|---|
| 33.19 ± 0.86 | 3.99 ± 0.12 | 9.66 ± 0.80 | ||
| + | 33.47 ± 0.59 | 4.03 ± 0.24 | 9.45 ± 0.63 | |
| + | 33.82 ± 0.30 | 3.95 ± 0.25 | 9.09 ± 0.44 | |
| + | + | 34.47 ± 0.77 | 3.85 ± 0.25 | 8.45 ± 0.82 |
| Huawei Mate 40 Pro | Huawei Mate 80 Pro | iPhone 16 Pro | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | PSNR, dB ↑ | SAM, ° ↓ | NSE, % ↓ | PSNR, dB ↑ | SAM, ° ↓ | NSE, % ↓ | PSNR, dB ↑ | SAM, ° ↓ | NSE, % ↓ |
| Single-image | 20.02 | 18.55 | 32.31 | 11.86 | 12.96 | 25.05 | 17.81 | 5.86 | 18.97 |
| Multi-image | 26.33 | 9.56 | 17.64 | 20.38 | 5.33 | 13.69 | 23.23 | 9.12 | 20.29 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | # scenes | Spectral data | Color reference | Spectral sampling Range (step), nm | Corresponding RGBs |
|---|---|---|---|---|---|
| CAVE ( Yasuma et al., 2010 ) | 32 | Reflectance | Color chart or no | 400–700 (10) | simulated BMP |
| Harvard ( Chakrabarti and Zickler, 2011 ) | 79 | Radiance | No | 400–720 (10) | No |
| ICVL ( Arad and Ben-Shahar, 2016 ) | 200 | Radiance | Color chart or no | 400–1000 (10) | simulated JPEG |
| KAUST ( Li et al., 2021 ) | 409 | Reflectance | White patch | 400–730 (10) | No |
| ARAD 1K ( Arad et al., 2022 ) | 949 | Radiance | No | 400–700 (10) | simulated JPEG |
| Beyond RGB ( Glatt et al., 2024 ) | 1680 | Radiance | Color chart | 380–730 (20) | 2× real RAW |
| Dataset version | Smartphone images | HSIs spectral range (step) | FoV matching | |
|---|---|---|---|---|
| RAW | Main, Tele, Wide | Original 400–1000 nm (3 nm) | – | |
| 1-camera | Preprocessed Main camera image | Preprocessed 400–730 nm (10 nm) | Preprocessed 400–730 nm (3 nm) | ✓ |
| 3-camera | Preprocessed and concatenated Main, Tele, Wide images | Preprocessed 400–730 nm (10 nm) | Preprocessed 400–730 nm (3 nm) | ✓ |
| Light conditions | PSNR, dB ↑ | SAM, ° ↓ | NSE, % ↓ |
|---|---|---|---|
| Bright scenario (outdoor sunny) | 41.60 ± 3.41 | 0.71 ± 0.11 | 3.00 ± 0.71 |
| Medium brightness (indoor LED) | 37.82 ± 3.51 | 1.02 ± 0.13 | 4.45 ± 1.28 |
| Inside DCAM | GT warping | PSNR, dB ↑ | SAM, ° ↓ | NSE, % ↓ |
|---|---|---|---|---|
| PWC-Net | PWC-Net | 34.39 ± 0.77 | 3.86 ± 0.24 | 8.55 ± 0.71 |
| PWC-Net | RAFT-Small | 34.14 ± 0.75 | 3.85 ± 0.21 | 8.73 ± 0.82 |
| RAFT-Small | PWC-Net | 34.22 ± 0.60 | 3.96 ± 0.41 | 8.51 ± 0.57 |
| RAFT-Small | RAFT-Small | 34.34 ± 0.53 | 3.97 ± 0.23 | 8.43 ± 0.53 |
Hyperspectral Imaging
Hyperspectral imaging captures spatial images across many narrow wavelength bands to identify materials and support detection, mapping, and analysis.
See all 144 papers