SpinSCI: Mask-Conditioned Spatial-Temporal Spin Transformer for Video Snapshot Compressive Imaging
Organizations: ETH AI Center, ETH Zurich, Switzerland
Abstract
Video snapshot compressive imaging (SCI) recovers multiple frames from one spatiotemporally coded measurement. This paper proposes SpinSCI, a mask-conditioned Spatial-Temporal Spin Transformer whose Mask-Conditioned Ising Refinement Block (MC-IRB) combines attention logits and sensing information to form signed spin couplings. Finite mean-field updates gate residual feature refinement. Physics-guided initialization and final data consistency incorporate the measurement model. Compared with HiSViT, SpinSCI achieves average PSNR gains of 0.51 dB, 0.23 dB, and 0.04 dB on small-scale gray, mid-scale color, and large-scale color benchmarks, respectively, with 16.9% fewer parameters, 21.0-21.6% fewer FLOPs, 17.1-49.9% lower latency, and 62.3-63.3% lower peak memory. Relative to DU-9stg, average PSNR differs by at most 0.13 dB across these benchmarks, with 28.2-28.7% fewer FLOPs, 64.6-71.7% lower inference latency, and 30.1-35.0% lower peak memory. Qualitative comparisons on the real benchmark further show recovery of fine details and textures. Code and reference checkpoints are available at https://github.com/skethz/SpinSCI.
Figures & tables
| Spatial MC-IRB | Temporal MC-IRB | PSNR (dB) | Params (M) | FLOPs (G) | Latency (ms) | Memory (MiB) |
| - | - | 36.68 | 6.86 | 4528 | 117 | 4280 |
| - | ✓ | 36.91 | 7.16 | 4690 | 134 | 4282 |
| ✓ | - | 36.84 | 7.16 | 4694 | 135 | 4282 |
| ✓ | ✓ | 37.07 | 7.47 | 4856 | 150 | 4284 |
| Variant | PSNR (dB) | Params (M) | FLOPs (G) | |
| Full MC-IRB, | 37.07 | – | 7.47 | 4856 |
| Without MC-IRB | 36.68 | 6.86 | 4528 | |
| Without MC-IRB, | 36.87 | 7.53 | 4880 | |
| Residual MLP, fixed | 36.78 | 7.46 | 4851 | |
| Conditioned gate without mean-field inference | 36.87 | 7.47 | 4856 | |
| Sigmoid gate on only | 36.84 | 7.47 | 4856 |
| Configuration | Benchmark binary | Unseen binary | Gaussian |
| Full model | 37.07 | 37.07 | 18.08 |
| removed from | 36.94 | 36.77 | – |
| Also removed from the stem input | 36.88 | 36.69 | – |
| Training with random binary masks | 37.10 | – | – |
| Training with per-sample Gaussian masks | 27.12 | 27.13 | 35.96 |
| Method | Params | FLOPs | Latency | Memory | Aerial | Crash | Drop | Kobe | Runner | Traffic | Average | |||||||
| (M) | (G) | (ms) | (MiB) | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | |
| PnP-FastDVDNet | 2.48 | 29605 | 2834 | 842 | 27.98 | 0.897 | 27.32 | 0.925 | 41.82 | 0.989 | 32.73 | 0.947 | 36.29 | 0.962 | 27.95 | 0.932 | 32.35 | 0.942 |
| EfficientSCI-B | 8.82 | 1424 | 50 | 1382 | 31.50 | 0.953 | 31.13 | 0.971 | 45.18 | 0.995 | 35.76 | 0.974 | 43.05 | 0.988 | 32.30 | 0.968 | 36.48 | 0.975 |
| STFormer-B | 19.49 | 3060 | 83 | 1883 | 31.56 | 0.953 | 31.06 | 0.970 | 45.08 | 0.995 | 35.53 | 0.973 | 42.64 | 0.988 | 32.15 | 0.967 | 36.34 | 0.974 |
| HiSViT 9 | 8.99 | 1536 | 178 | 4904 | 31.67 | 0.957 | 31.62 | 0.976 | 45.55 | 0.995 | 36.24 | 0.976 | 43.84 | 0.991 | 33.06 | 0.973 | 37.00 | 0.978 |
| DU-9stg | 6.67 | 1690 | 437 | 2647 | 32.27 | 0.961 | 32.47 | 0.978 | 45.99 | 0.995 | 36.51 | 0.978 | 44.75 | 0.991 | 33.45 | 0.974 | 37.57 | 0.980 |
| Method | Params | FLOPs | Latency | Memory | Beauty | Bosphorus | Jockey | Runner | ShakeNDry | Traffic | Average | |||||||
| (M) | (G) | (ms) | (MiB) | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | |
| PnP-FastDVDNet | 2.48 | 120077 | 45907 | 2664 | 35.27 | 0.972 | 37.24 | 0.978 | 35.63 | 0.950 | 38.22 | 0.965 | 33.71 | 0.969 | 27.49 | 0.915 | 34.60 | 0.955 |
| EfficientSCI-B | 8.83 | 5702 | 213 | 5570 | 37.51 | 0.979 | 40.89 | 0.988 | 38.49 | 0.969 | 42.73 | 0.985 | 35.19 | 0.953 | 30.13 | 0.943 | 37.49 | 0.970 |
| MambaSCI-B | 8.59 | 557 | 990 | 4206 | 36.95 | 0.979 | 38.62 | 0.982 | 36.54 | 0.960 | 40.02 | 0.977 | 34.55 | 0.950 | 27.52 | 0.904 | 35.70 | 0.959 |
| STFormer-B | 19.49 | 12156 | 371 | 7081 | 37.37 | 0.981 | 40.39 | 0.988 | 38.32 | 0.968 | 42.45 | 0.985 | 35.15 | 0.956 | 30.24 | 0.939 | 37.32 | 0.970 |
| HiSViT 9 | 8.99 | 6146 | 760 | 19462 | 37.75 | 0.981 | 41.50 | 0.990 | 39.29 | 0.972 | 43.27 | 0.986 | 35.49 | 0.958 | 30.76 | 0.946 | 38.01 | 0.972 |
| Method | Params | FLOPs | Latency | Memory | Football | Hummingbird | Messi | Swinger | Average | |||||
| (M) | (G) | (ms) | (MiB) | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | |
| PnP-FastDVDNet | 2.48 | 956866 | 369528 | 891 | 37.42 | 0.982 | 38.51 | 0.979 | 33.54 | 0.979 | 28.26 | 0.930 | 34.43 | 0.967 |
| EfficientSCI-B | 8.83 | 45100 | 1634 | 1841 | 40.33 | 0.989 | 40.41 | 0.974 | 34.85 | 0.974 | 29.45 | 0.940 | 36.26 | 0.969 |
| MambaSCI-B | 8.59 | 4405 | 7673 | 1309 | 35.91 | 0.974 | 31.67 | 0.641 | 29.09 | 0.923 | 27.01 | 0.900 | 30.92 | 0.859 |
| STFormer-B | 19.49 | 96071 | 2616 | 2416 | 40.64 | 0.990 | 40.43 | 0.974 | 34.87 | 0.976 | 29.59 | 0.943 | 36.38 | 0.971 |
| HiSViT 9 | 8.99 | 48977 | 8377 | 6503 | 42.25 | 0.993 | 41.48 | 0.989 | 36.39 | 0.986 | 30.43 | 0.954 | 37.64 | 0.980 |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Training epochs | Full PSNR | No-MC-IRB PSNR | |
| 35.27 | 34.82 | ||
| 37.07 | 36.68 | ||
| 38.11 | 37.74 |
| Variant | PSNR (dB) mean sample SD | Paired (dB) | interval for |
| Full MC-IRB, | – | – | |
| Without MC-IRB | |||
| Without MC-IRB, | |||
| Residual MLP, fixed | |||
| Conditioned gate without mean-field inference | |||
| Sigmoid gate on only |
| Internal MC-IRB statistics | ||
| Measurement | Spatial | Temporal |
| Effective (range for each) | ||
| Effective (range for each) | ||
| Negative-coupling fraction (%) | 51.95 | 55.46 |
| 4.540 | 1.650 | |
| 24.720 | 1.103 | |
| Benchmark | HiSViT 9 reported | HiSViT 9 measured | SpinSCI-B |
| Small-scale gray | 37.00/0.978 | 37.00/0.977 | 37.51 / 0.980 |
| Mid-scale color | 38.01/0.972 | 38.01/0.972 | 38.24 / 0.973 |
| Benchmark | Reported | Measured | Difference |
| Small gray | 37.57 | 37.523 | |
| Mid color | 38.30 | 38.117 | |
| Large color | — | 37.799 | — |
| Method | PSNR (dB) | SSIM | Latency (ms) | Memory (MiB) |
| PnP-FastDVDNet | 35.28 | 0.958 | 38,732.4 | 1,139 |
| EfficientSCI-B | 37.17 | 0.968 | 130.2 | 2,143 |
| MambaSCI-B | 35.62 | 0.958 | 581.7 | 1,639 |
| STFormer-B | 37.46 | 0.970 | 120.7 | 3,188 |
| HiSViT 9 | 37.91 | 0.972 | 213.8 | 6,368 |
| DU-9stg | 38.06 | 0.972 | 1,315.6 | 3,726 |
Compressed Sensing
Compressed sensing recovers signals from fewer measurements by exploiting sparsity, structure, and learned prior information.
See all 39 papers