TANGO: Real-World Unregistered RGB-Guided Hyperspectral Super-Resolution with Two-Camera Acquisitions
Organizations: University of Milano-Bicocca
Abstract
Unregistered hyperspectral image super-resolution aims to recover a high-resolution HSI from a low-resolution hyperspectral observation and an unregistered high-resolution RGB reference. Although recent methods have begun to address the geometric misalignment between the two modalities, their evaluation is still constrained by the lack of a large-scale public benchmark based on real two-camera RGB--HSI acquisitions with a high-resolution HSI reference. To address this gap, we introduce DUET, the first large-scale public dataset for unregistered RGB-guided hyperspectral super-resolution, comprising 403 indoor and outdoor scenes captured with two separate RGB and hyperspectral cameras. We further propose TANGO (Two-camera AligNment with Gated Occlusions), which first compensates for the dominant cross-camera displacement through end-to-end global alignment and then refines the remaining inconsistencies with a coarse-to-fine local feature alignment. A learned gating mechanism further attenuates the RGB guidance where the two views cannot be consistently matched. Experimental results show that TANGO achieves state-of-the-art performance on both DUET and REAL across all evaluated scale factors, outperforming the strongest competitor on DUET by up to 2.29 dB in PSNR while requiring 2.5 fewer FLOPs. The code and dataset will be available at https://github.com/FagiolottoConSombrero/TANGO-DUET
Figures & tables
| Method | Scale Factor | Scale Factor | Scale Factor | Params [M] | FLOPs [G] | Time [s] | ||||||
| PSNR | SSIM | SAM | PSNR | SSIM | SAM | PSNR | SSIM | SAM | ||||
| Bicubic | 31.140 | 0.8599 | 0.04541 | 26.603 | 0.7462 | 0.07357 | 23.140 | 0.6616 | 0.11375 | – | – | 0.01 |
| SSPSR [ 15 ] | 38.475 | 0.9515 | 0.02812 | 31.274 | 0.8512 | 0.04908 | 25.879 | 0.7375 | 0.07890 | 26.08 | 21066.1 | 1.72 |
| Bi-3DQRNN [ 9 ] | 37.662 | 0.9441 | 0.03324 | 30.712 | 0.8369 | 0.05477 | 25.882 | 0.7247 | 0.08839 | 1.288 | 3749.4 | 1.11 |
| ESSAformer [ 40 ] | 37.936 | 0.9478 | 0.03115 | 30.752 | 0.8406 | 0.05381 | 25.729 | 0.7327 | 0.08317 | 11.194 | 23015.6 | 2.14 |
| HSIFN [ 21 ] | 35.931 | 0.9264 | 0.04682 | 32.103 | 0.8753 | 0.06406 | 28.585 | 0.8139 | 0.09318 | 69.922 | 10707.3 | 2.54 |
| Method | PSNR | SSIM | SAM |
|---|---|---|---|
| Bicubic | 28.675 | 0.7810 | 0.05743 |
| SSPSR [ 15 ] | 32.404 | 0.8496 | 0.04912 |
| Bi-3DQRNN [ 9 ] | 31.926 | 0.8379 | 0.05433 |
| ESSAformer [ 40 ] | 31.443 | 0.8304 | 0.06384 |
| HSIFN [ 21 ] | 33.181 | 0.8795 | 0.07013 |
| M2DTN [ 7 ] | 34.027 | 0.9121 | 0.06966 |
| Global | Local | PSNR | SSIM | SAM | Params [M] |
|---|---|---|---|---|---|
| 33.292 | 0.9004 | 0.05001 | 3.551 | ||
| ✓ | 33.644 | 0.9059 | 0.04914 | 3.630 | |
| 35.339 | 0.9330 | 0.04221 | 3.551 | ||
| ✓ | 35.676 | 0.9367 | 0.04060 | 3.630 | |
| ✓ | 37.888 | 0.9534 | 0.03942 | 8.809 | |
| ✓ | 38.116 | 0.9560 | 0.03762 | 8.887 |
| Local alignment | PSNR | SSIM | SAM | Params [M] |
|---|---|---|---|---|
| w/o gate | 37.867 | 0.9528 | 0.03886 | 8.824 |
| w/ gate | 38.116 | 0.9560 | 0.03762 | 8.887 |
| [m] | 1 | 3 | 10 | 30 | 100 |
|---|---|---|---|---|---|
| Disparity [px] | 138 | 46 | 14 | 5 | 1.4 |
| Statistic | Value |
|---|---|
| P95 residual, median scene [px] | 11.1 |
| P95 residual, 90th-percentile scene [px] | 26.4 |
| Scenes with P95 residual 4 px [%] | 72.7 |
| Pixels with residual 4 px, median scene [%] | 22.0 |
| Pixels without correspondence, median / mean [%] | 5.2 / 9.6 |
| Method | Scale Factor | Scale Factor | Scale Factor | Params [M] | FLOPs [G] | ||||||
| PSNR | SSIM | SAM | PSNR | SSIM | SAM | PSNR | SSIM | SAM | |||
| Bicubic | 33.678 | 0.8868 | 0.04003 | 28.675 | 0.7810 | 0.05743 | 24.848 | 0.7024 | 0.08372 | – | – |
| SSPSR [ 15 ] | 39.521 | 0.9498 | 0.03468 | 32.404 | 0.8496 | 0.04912 | 27.317 | 0.7517 | 0.06652 | 26.08 | 21066.1 |
| Bi-3DQRNN [ 9 ] | 38.366 | 0.9398 | 0.04044 | 31.926 | 0.8379 | 0.05433 | 27.078 | 0.7414 | 0.08212 | 1.288 | 4100.9 |
| ESSAformer [ 40 ] | 37.906 | 0.9368 | 0.04445 | 31.443 | 0.8304 | 0.06384 | 26.942 | 0.7379 | 0.08295 | 11.194 | 21795.5 |
| HSIFN [ 21 ] | 36.928 | 0.9321 | 0.05550 | 33.181 | 0.8795 | 0.07013 | 29.161 | 0.8317 | 0.09586 | 69.922 | 14276.4 |
| Global | Local | Scale Factor | Scale Factor | Scale Factor | Params [M] | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| PSNR | SSIM | SAM | PSNR | SSIM | SAM | PSNR | SSIM | SAM | |||
| 39.046 | 0.9597 | 0.03127 | 33.292 | 0.9004 | 0.05001 | 29.058 | 0.8329 | 0.07061 | 3.551 | ||
| ✓ | 39.449 | 0.9624 | 0.03027 | 33.644 | 0.9059 | 0.04914 | 29.774 | 0.8485 | 0.06819 | 3.630 | |
| 42.570 | 0.9807 | 0.02653 | 35.339 | 0.9330 | 0.04221 | 25.587 | 0.7250 | 0.08254 | 3.551 | ||
| ✓ | 42.739 | 0.9813 | 0.02624 | 35.676 | 0.9367 | 0.04060 | 25.517 | 0.7239 | 0.08206 | 3.630 | |
| ✓ | 43.325 | 0.9824 | 0.02593 | 37.888 | 0.9534 | 0.03942 | 31.863 | 0.8967 | 0.06075 | 8.809 | |
| Local alignment | Scale Factor | Scale Factor | Scale Factor | Params [M] | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PSNR | SSIM | SAM | PSNR | SSIM | SAM | PSNR | SSIM | SAM | ||
| w/o gate | 43.249 | 0.9820 | 0.02577 | 37.867 | 0.9528 | 0.03886 | 32.139 | 0.8987 | 0.06025 | 8.824 |
| w/ gate | 43.672 | 0.9833 | 0.02467 | 38.116 | 0.9560 | 0.03762 | 32.418 | 0.9047 | 0.05725 | 8.887 |
| Region | Pixels [%] | Global | w/o gate | TANGO | |
|---|---|---|---|---|---|
| Planar ( 1 px) | 51.6 | 6.04 | 6.01 | 5.73 | |
| 1–4 px | 22.2 | 7.73 | 7.87 | 7.50 | |
| 4 px | 15.8 | 9.87 | 10.07 | 9.81 | |
| No correspondence | 10.4 | 12.28 | 12.06 | 11.53 |
Hyperspectral Image Super-Resolution
Hyperspectral image super-resolution reconstructs finer spatial detail while preserving the rich spectral signatures measured across many wavelengths.
See all 23 papers