Graphene: A Lightweight Transformer with Ising Mean-Field Refinement for Image Super-Resolution
Abstract
This paper proposes Graphene, a lightweight image super-resolution architecture with two complementary feature refinement mechanisms: Ising-model-inspired local feature refinement using an Ising Refinement Block (IRB) and stage-history-based feature refinement using a History Attention (HistAttn) module. The IRB reuses attention output features and pre-softmax attention logits to construct signed pairwise token couplings within each local window, applies unrolled mean-field updates, and refines features through a gated residual correction. These signed couplings allow alignment and opposition between latent spins to modulate local feature refinement. HistAttn attends over the current stage output and earlier representations at the same token position and adaptively blends the current stage output with a history-weighted feature aggregate without adding spatial pairwise attention. On Urban100 at , Graphene and Graphene-light improve over IET and IET-light by 0.24 dB and 0.38 dB, respectively, with 34.3%/13.7% fewer parameters, 44.8%/44.4% fewer FLOPs, 83.9%/95.9% lower latency, and 63.7%/92.0% lower peak memory. Real-world SR comparisons further show that Graphene recovers fine details and preserves textures across diverse degradations. Code and reference checkpoints are available at https://github.com/skethz/Graphene
Figures & tables
| (a) Component baselines | ||||||
| Window | IRB | HistAttn | Params (M) | FLOPs (G) | PSNR (dB) | (dB) |
| 16 | – | – | 10.3 | 91.0 | 30.14 | |
| 32 | – | – | 10.5 | 111.4 | 30.33 | |
| 32 | – | Enabled | 10.5 | 111.6 | 30.53 | |
| 32 | Enabled | – | 12.9 | 131.8 | 30.88 | |
| 32 | Enabled | Enabled | 12.9 | 132.0 | 30.93 | |
| History configuration | Params (M) | FLOPs (G) | PSNR (dB) | (dB) |
|---|---|---|---|---|
| No history fusion | 12.9 | 131.8 | 30.88 | |
| Learned shallow skip | 12.9 | 131.8 | 30.89 | |
| Static history weights | 12.9 | 131.8 | 30.90 | |
| HistAttn | 12.9 | 132.0 | 30.93 |
| Classical SR | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Params | FLOPs | Latency | Memory | Set5 | Set14 | B100 | U100 | M109 |
| (M) | (G) | (ms) | (MiB) | ||||||
| SwinIR ( Liang et al., 2021 ) | 11.9 | 107.7 | 2.93 | 86.3 | 32.92 | 29.09 | 27.92 | 27.45 | 32.03 |
| CAT-A ( Chen et al., 2022 ) | 16.6 | 160.0 | 6.79 | 216.4 | 33.08 | 29.18 | 27.99 | 27.89 | 32.39 |
| ART ( Zhang et al., 2023 ) | 16.5 | 156.1 | 4.43 | 97.3 | 33.04 | 29.16 | 27.97 | 27.77 | 32.31 |
| HAT ( Chen et al., 2023 ) | 20.8 | 208.4 | 7.16 | 163.3 | 33.04 | 29.23 | 28.00 | 27.97 | 32.48 |
| Method | Latency (ms) | Memory (GiB) | Latency (s) | Memory (GiB) |
|---|---|---|---|---|
| Classical SR | ||||
| SwinIR ( Liang et al., 2021 ) | 19.6 | 0.37 | 0.254 | 4.65 |
| CAT-A ( Chen et al., 2022 ) | 86.6 | 2.22 | 5.268 | 109.51 |
| ART ( Zhang et al., 2023 ) | 225.2 | 3.17 | OOM | OOM |
| HAT ( Chen et al., 2023 ) | 49.3 | 0.84 | 0.676 | 11.32 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Lean module | Checked properties |
|---|---|
| Lifting | Group-sum preservation for a nonempty replication group and a general key-cell fiber. |
| Couplings | Centered row sums, symmetry, zero diagonal, commuting transformations, constant-row degeneracy, and the admissible lifted counterexample. |
| Ising | Binary-state probabilities and mean; single-coordinate entropy and free-energy derivatives; stationarity equivalence; symmetric field identity. |
| Bounds | Magnetization and gate bounds, the neutral gate, and gated residual norm identities. |
| Stability | Euclidean Lipschitz bounds, bounded finite iterates, conditional unique fixed point and convergence, and the finite-iteration error bound. |
| History | Positive normalized weights, convex-hull membership, effective fusion coefficients, endpoint identities, and an extrapolation example. |
| Setting | Graphene | Graphene-light |
|---|---|---|
| Stages | 6 | 4 |
| Blocks per stage | 6 | 6 |
| Feature channels | 180 | 60 |
| Attention heads per block | 6 | 6 |
| Window side length | 32 | 16 |
| PSA reduction factor | 2 | 2 |
| Signed | Symmetric | PSNR | SSIM | |
|---|---|---|---|---|
| 1 | Enabled | Enabled | 30.42 | 0.915 |
| 2 | Enabled | Enabled | 30.84 | 0.915 |
| 3 | Enabled | Enabled | 30.93 | 0.917 |
| 10 | Enabled | Enabled | 30.85 | 0.915 |
| 100 | Enabled | Enabled | 30.85 | 0.915 |
| 3 | Disabled | Enabled | 30.90 | 0.916 |
| Metric | 4128 | 4129 | 4130 | 4127 | 4131 | 4132 | Mean | SD |
|---|---|---|---|---|---|---|---|---|
| PSNR | 31.02 | 30.90 | 30.81 | 30.94 | 30.86 | 30.99 | 30.92 | 0.08 |
| SSIM | 0.916 | 0.915 | 0.913 | 0.915 | 0.914 | 0.916 | 0.915 | 0.001 |
| Scale | Model | Params | FLOPs | Set5 | Set14 | B100 | U100 | M109 |
|---|---|---|---|---|---|---|---|---|
| (M) | (G) | |||||||
| HAT | 20.6 | 203.4 | 37.97 | 33.48 | 32.25 | 32.21 | 38.77 | |
| HAT+IRB | 22.8 | 226.1 | 38.16 | 33.82 | 32.41 | 33.15 | 39.36 | |
| HAT | 20.8 | 208.4 | 32.40 | 28.54 | 27.78 | 26.75 | 31.36 | |
| HAT+IRB | 22.9 | 231.0 | 32.38 | 28.55 | 27.86 | 26.96 | 31.55 |
| Step | All pixels ( ) | Boundary ( ) | Boundary minus interior ( ) |
|---|---|---|---|
| 0 | |||
| 250 | |||
| 500 | |||
| 750 | |||
| 1000 |
| Destination | |||||
|---|---|---|---|---|---|
| 2 | 0.593 | 0.407 | 0.500 | 0.297 | 0.703 |
| 3 | 0.969 | 0.030 | 0.832 | 0.806 | 0.193 |
| 4 | 0.988 | 0.009 | 0.815 | 0.805 | 0.192 |
| 5 | 0.555 | 0.029 | 0.858 | 0.477 | 0.167 |
| 6 | 0.178 | 0.408 | 0.667 | 0.119 | 0.605 |
| Method | HAT | SRFormer | MambaIRv2 | PFT | IET | Graphene |
|---|---|---|---|---|---|---|
| DI | 19.7563 | 12.5254 | 18.6200 | 14.0510 | 15.6248 | 24.9676 |
| Scale | Method | Params | FLOPs | Latency | Memory | Set5 | Set14 | B100 | U100 | M109 | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (M) | (G) | (ms) | (MiB) | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | ||
| SwinIR | 11.8 | 102.7 | 2.89 | 85.2 | 38.42 | 0.9623 | 34.46 | 0.9250 | 32.53 | 0.9041 | 33.81 | 0.9427 | 39.92 | 0.9797 | |
| CAT-A | 16.5 | 155.0 | 6.77 | 215.3 | 38.51 | 0.9626 | 34.78 | 0.9265 | 32.59 | 0.9047 | 34.26 | 0.9440 | 40.10 | 0.9805 | |
| ART | 16.4 | 151.1 | 4.39 | 96.1 | 38.56 | 0.9629 | 34.59 | 0.9267 | 32.58 | 0.9048 | 34.30 | 0.9452 | 40.24 | 0.9808 | |
| HAT | 20.6 | 203.4 | 7.10 | 162.2 | 38.63 | 0.9630 | 34.86 | 0.9274 | 32.62 | 0.9053 | 34.45 | 0.9466 | 40.26 | 0.9809 | |
| SRFormer | 10.4 | 103.2 | 4.48 | 117.0 | 38.51 | 0.9627 | 34.44 | 0.9253 | 32.57 | 0.9046 | 34.09 | 0.9449 | 40.07 | 0.9802 | |
| Scale | Method | Params | FLOPs | Latency | Memory | Set5 | Set14 | B100 | U100 | M109 | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (K) | (G) | (ms) | (MiB) | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | ||
| SwinIR-light | 910.0 | 8.7 | 1.13 | 26.1 | 38.14 | 0.9611 | 33.86 | 0.9206 | 32.31 | 0.9012 | 32.76 | 0.9340 | 39.12 | 0.9783 | |
| SRFormer-light | 853.0 | 8.2 | 1.55 | 28.2 | 38.23 | 0.9613 | 33.94 | 0.9209 | 32.36 | 0.9019 | 32.91 | 0.9353 | 39.28 | 0.9785 | |
| ATD-light | 761.0 | 13.6 | 3.73 | 19.1 | 38.29 | 0.9616 | 34.10 | 0.9217 | 32.39 | 0.9023 | 33.27 | 0.9375 | 39.52 | 0.9789 | |
| MambaIR-light | 905.0 | 11.9 | 28.96 | 24.8 | 38.16 | 0.9610 | 34.00 | 0.9212 | 32.34 | 0.9017 | 32.92 | 0.9356 | 39.31 | 0.9779 | |
| MambaIRv2-light | 774.0 | 10.0 | 18.23 | 19.3 | 38.26 | 0.9615 | 34.09 | 0.9221 | 32.36 | 0.9019 | 33.26 | 0.9378 | 39.35 | 0.9785 | |
| Method | Latency | Memory | Latency | Memory | Latency | Memory |
|---|---|---|---|---|---|---|
| (ms) | (MiB) | (ms) | (MiB) | (ms) | (MiB) | |
| Classical SR | ||||||
| SwinIR ( Liang et al., 2021 ) | 25.6 | 138.5 | 25.8 | 138.9 | 25.4 | 139.6 |
| CAT-A ( Chen et al., 2022 ) | 92.5 | 281.5 | 82.3 | 282.8 | 86.3 | 282.1 |
| ART ( Zhang et al., 2023 ) | 42.9 | 170.9 | 43.4 | 170.9 | 46.1 | 172.1 |
| Method | Latency (ms) | Memory (GiB) | Latency (s) | Memory (GiB) |
|---|---|---|---|---|
| Classical SR | ||||
| SwinIR ( Liang et al., 2021 ) | 108.2 | 0.69 | 1.67 | 9.31 |
| CAT-A ( Chen et al., 2022 ) | 300.8 | 2.86 | OOM | OOM |
| ART ( Zhang et al., 2023 ) | 377.6 | 10.72 | OOM | OOM |
| HAT ( Chen et al., 2023 ) | 187.4 | 2.41 | 3.24 | 34.48 |
Image Super-Resolution
Image super-resolution reconstructs high-resolution images from lower-resolution observations, balancing detail, fidelity, and computational cost.
See all 274 papers