Fast Training-Free Low-Light Video Enhancement with a Frozen Image Diffusion Prior
Organizations: City University of Hong Kong
Abstract
Pretrained image diffusion models can enhance low-light video without low-light training data and without retraining for a new camera. ZVRD, the only published training-free method for low-light video, is too slow to use: our implementation needs over a minute per frame. We present a causal, training-free pipeline that adds two temporal components to a frozen MR. Illuminate image prior. Temporal input fusion recursively averages motion-compensated low-light frames before diffusion inversion. Late-start offset propagation runs the full sampling trajectory only on key frames. Each non-key frame inverts the low-noise part of its own trajectory and adds an enhancement offset warped from the latest key frame. At ZVRD's own resolution the pipeline is faster than ZVRD and scores better on all five metrics we report. On 13 held-out SDSD clips at , it improves on framewise eight-step processing by 0.55 dB in PSNR and 0.034 in LPIPS and halves the warping error, in 10% less time. Measurements inside the model show why, and a linear-Gaussian model of inversion explains them. Fusion removes frame-to-frame noise before inversion amplifies it. A flow error perturbs the transported offset ten times less than it perturbs a transported state. The perceptual gain does not transfer to low resolution: at , LPIPS is worse by 0.028. At 0.65 frames per second, the pipeline is an offline method.
Figures & tables
| Method | PSNR | SSIM | LPIPS | Warp | Flicker | NFE | s/frame | Speedup |
|---|---|---|---|---|---|---|---|---|
| ZVRD [ 2 ] (our impl.) | 15.81 ∘ | 0.512 | 0.312 | 1.51 | 0.280 | 950 | 90.78 | 1 |
| MR E15 | 19.08 ∘ | 0.681 ∘ | 0.156 | 1.00 | 0.107 | 30 | 0.67 | 135 |
| MR E8 | 18.91 ∘ | 0.672 | 0.163 | 0.94 | 0.103 | 16 | 0.38 | 239 |
| Ours | 19.18 | 0.688 | 0.178 | 0.51 | 0.087 | 12 | 0.35 | 261 |
| Method | Low-light training data | PSNR | SSIM | LPIPS | Warp | Flicker | NFE | s/frame |
| ReDDiT-2 [ 23 ] | LOLv2 pairs | 17.25 ∘ | 0.617 | 0.286 | 1.99 | 0.166 | 2 | 0.47 |
| ReDDiT-4 [ 23 ] | LOLv2 pairs | 18.04 ∘ | 0.582 | 0.302 | 2.48 | 0.184 | 4 | 0.87 |
| ReDDiT-8 [ 23 ] | LOLv2 pairs | 18.51 ∘ | 0.609 | 0.295 | 2.33 | 0.187 | 8 | 1.67 |
| Zero-TIG [ 24 ] | BVI-RLV videos | 19.09 ∘ | 0.739 | 0.296 | 1.10 | 0.131 | n/a | 0.12 |
| AGLLDiff [ 27 ] | none | 13.35 | 0.589 | 0.378 | 1.39 | 0.134 | 49.9 | 67.45 |
| MR A25 [ 5 ] | none | 17.47 | 0.672 | 0.233 ∘ | 1.23 | 0.136 | 50 | 3.81 |
| Trajectory | Calib. | PSNR | SSIM | LPIPS | Warp | Flicker | Time | |
|---|---|---|---|---|---|---|---|---|
| – | E8 | yes | 17.21 | 0.674 | 0.252 | 1.01 | 0.126 | 1.00 |
| – | E15 | yes | 17.35 | 0.673 | 0.235 ∘ | 1.15 | 0.132 | 1.45 ‡ |
| 0.8 | E8 | yes | 17.49 ∘ | 0.707 ∘ | 0.228 | 0.50 | 0.109 | 1.03 |
| 0.8 | E15 | yes | 17.71 ∘ | 0.710 ∘ | 0.213 | 0.54 | 0.110 | 1.49 ∗ |
| – | late start, offset | yes | 17.34 | 0.673 | 0.235 ∘ | 1.12 | 0.131 | 0.87 |
| 0.8 | late start, key state | yes | 17.57 | 0.696 | 0.249 | 0.56 | 0.112 | 0.91 ∗ |
| Set | Method | PSNR | SSIM | LPIPS | Warp | Flicker |
|---|---|---|---|---|---|---|
| SDE | MR E15 | 16.44 | 0.530 ∘ | 0.353 | 2.52 | 0.202 |
| MR E8 | 15.98 | 0.537 ∘ | 0.344 | 2.27 | 0.185 | |
| MR state reuse | 14.85 | 0.481 | 0.416 | 3.76 | 0.278 | |
| Fusion only | 16.01 | 0.537 ∘ | 0.371 ∘ | 1.50 | 0.153 | |
| Late start only | 16.55 | 0.536 ∘ | 0.351 | 2.46 | 0.199 | |
| Ours, | 16.61 ∘ | 0.541 ∘ | 0.353 | 2.14 | 0.188 |
| Split | SDSD scenes | Used to choose |
|---|---|---|
| Design (12) | indoor 13, 15, 21, 23, 31, 33; outdoor 1, 5, 14, 36, 46, 48 | , , ; calibration |
| Held-out (6) | indoor 50, 52, 58; outdoor 49, 60, 62 | calibration |
| Held-out (7) | indoor 60, 68, 70; outdoor 63, 66, 75, 76 | neither |
| Method | s/frame | P95 (s) | NFE | Memory (GiB) |
|---|---|---|---|---|
| MR E8 | 1.69 | 1.70 | 16 | 6.1 |
| MR E15 | 2.44 | 2.46 | 30 | 6.4 |
| Ours | 1.54 | 2.49 | 12 | 6.4 |
| Method | PSNR | SSIM | LPIPS | Warp | Flicker | NFE | s/frame |
|---|---|---|---|---|---|---|---|
| MR E8 | 18.91 ∘ | 0.672 | 0.163 | 0.94 | 0.103 | 16 | 0.38 |
| Fusion only | 19.10 ∘ | 0.683 ∘ | 0.186 | 0.50 | 0.086 | 16 | 0.41 |
| Late start only | 18.97 | 0.678 | 0.156 | 0.96 | 0.105 | 12 | 0.32 |
| Ours, | 19.08 | 0.688 ∘ | 0.156 | 0.65 | 0.091 | 12 | 0.35 |
| Ours | 19.18 | 0.688 | 0.178 | 0.51 | 0.087 | 12 | 0.35 |
| Set | Clips | Mean | Reliable | |
|---|---|---|---|---|
| SDSD, , 30 frames | 25 | 0.0055 | 0.755 | 0.992 |
| SDSD, , 300 frames ∗ | 6 | 0.0057 | 0.760 | 0.992 |
| SDSD, | 6 | 0.0059 | 0.770 | 0.989 |
| SDE, | 15 | 0.0022 | 0.708 | 0.990 |
| State | Raw | Fused | Reduction | Raw | Fused | Reduction |
|---|---|---|---|---|---|---|
| Input luminance | 0.011 | 0.002 | 78% | 0.005 | 0.001 | 74% |
| VAE latent | 0.477 | 0.171 | 65% | 0.168 | 0.066 | 59% |
| Inversion latent, | 0.607 | 0.234 | 62% | 0.222 | 0.084 | 62% |
| Inversion latent, | 0.852 | 0.364 | 58% | 0.355 | 0.134 | 64% |
| Inversion endpoint, | 1.092 | 0.506 | 55% | 0.491 | 0.192 | 63% |
| Key age | Mismatch / offset | Mismatch / content | LPIPS gap |
|---|---|---|---|
| 1 | 0.69 | 0.59 | 0.023 |
| 2 | 0.85 | 0.72 | 0.037 |
| 3 | 0.94 | 0.80 | 0.043 |
| 4 | 1.00 | 0.84 | 0.049 |
| 5 | 1.03 | 0.87 | 0.050 |
| 6 | 1.06 | 0.89 | 0.051 |
| Resolution | Input | and (median / max) | Band power (median / max) | (median / max) |
|---|---|---|---|---|
| Raw | 4.2% / 14.0% | 3.5% / 11.5% | 0.001 / 0.027 | |
| Fused | 3.5% / 14.6% | 3.3% / 11.5% | 0.002 / 0.027 | |
| Raw | 1.6% / 8.3% | 1.7% / 4.1% | 0.001 / 0.008 | |
| Fused | 2.1% / 5.4% | 2.1% / 4.8% | 0.001 / 0.006 |
| State | Energy ratio | Relative response |
|---|---|---|
| VAE skip, level 0 (first convolution) | 4.00 | 0.005 |
| VAE skip, levels 1–3 | 2.20–2.49 | 0.14–0.18 |
| VAE latent | 2.21 | 0.10 |
| Inversion latent, | 2.11 | 0.13 |
| Inversion latent, | 1.88 | 0.21 |
| Inversion endpoint, | 1.75 | 0.42 |
Low-Light Image Enhancement
Low-light image enhancement restores visibility, color, and detail in underexposed images while limiting noise, blur, and color distortion.
See all 106 papers