What Does LOL Measure? Hidden Label Shifts in Low-Light Enhancement Benchmarks
Organizations: University of Wisconsin–Madison
Abstract
LOL-v1 and LOLv2-Real are the standard benchmarks for supervised low-light image enhancement, and several leading methods ship hand-set test-time adjustments with their evaluation scripts. We audit both benchmarks with 18 public checkpoints of eight methods under one metric implementation that reproduces every published number to within 0.09 dB, and find that their test labels differ from their training labels. (i) The 15 LOL-v1 test labels are rendered differently: the same captures appear in LOLv2 with labels of identical luminance but 1.48x lower saturation, and LOL-v1's own training labels match the LOLv2 rendering; we confirm this on the official release. (ii) The LOLv2-Real test set is the contiguous block of images 690-789, whose labels are 14% darker than the training labels (Cliff's delta = -0.45, p < 1e-12) with no comparable shift in the input statistics we examined; methods trained without any labels score up to 5.5 dB higher on test than on train. (iii) All 15 LOL-v1 test inputs reappear in LOLv2-Real, 12 in its training set. The released test-time settings are consistent with compensating for these shifts. Separating what can be calibrated on training data from what only test-set tuning finds, the test-only residual has median 0.00 dB on LOLv2-Syn, which has no shift, but 0.37 dB on LOL-v1 and 0.92 dB on LOLv2-Real (up to 3.9 dB). Under the shift, test-set checkpoint selection inflates a from-scratch model by 0.9-2.0 dB, and removing the released settings changes which method ranks first. We give concrete reporting recommendations.
Figures & tables
| Finding or analysis | Pan [20] | Ours |
|---|---|---|
| LOLv2-Real test labels identical to LOL-v1 training labels (91/100) | ✓ | ✓ |
| LOL-v1 test scenes with same-scene training twins | ✓(14/15) | (✓) (5/15) |
| Nearest-neighbour lookup baseline; leakage-free split lists | ✓ | – |
| LOL-v1 test inputs inside LOLv2-Real, with re-rendered labels | – | ✓ |
| LOL-v1 test labels more saturated than the training rendering | – | ✓ |
| LOLv2-Real contiguous split with darker test labels | – | ✓ |
| Method | Benchmark | Checkpoint | Metric | Paper | Ours | |
| HVI-CIDNet | LOL-v1 | w/o perc. | GT-mean | 28.14 | 28.23 | |
| HVI-CIDNet | LOL-v1 | w/ perc. | GT-mean | 27.71 | 27.73 | |
| Retinexformer | LOL-v1 | – | PSNR | 25.16 | 25.15 | |
| IAT | LOL-v1 | – | PSNR | 23.38 | 23.38 | |
| HVI-CIDNet | LOLv2-Real | bestPSNR | PSNR | 23.90 | 23.89 | |
| HVI-CIDNet | LOLv2-Real | bestSSIM | GT-mean | 28.39 | 28.39 |
| PSNR | GT-mean PSNR | Grid optimum | |||||
|---|---|---|---|---|---|---|---|
| Checkpoint | Benchmark | Released setting | removed | released | removed | released | train / test |
| CIDNet w/o perc. | LOL-v1 | saturation | 23.08 | 23.61 | 26.95 | 28.23 | 1.00 / 1.45 |
| CIDNet w/ perc. | LOL-v1 | saturation | 23.22 | 23.84 | 26.65 | 27.73 | 1.00 / 1.50 |
| CIDNet bestPSNR | LOLv2-Real | brightness | 20.56 | 23.89 | 27.96 | 27.95 | 1.02 / 0.80 |
| CIDNet bestSSIM | LOLv2-Real | brightness | 20.78 | 23.89 | 28.39 | 28.39 | 1.02 / 0.82 |
| PairLIE | LOL-v1 | exponent 0.14 (vs. 0.2) | 18.50 | 19.52 | 23.51 | 23.16 | – |
| LOL-v1 (485 / 15) | LOLv2-Real (689 / 100) | LOLv2-Syn (900 / 100) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Statistic | ratio | ratio | ratio | ||||||
| Input luminance | 1.15 | 0.54 | 0.09 | 0.84 | 0.29 | 0.85 | 0.048 | ||
| Input saturation | 0.94 | 0.62 | 0.93 | 0.012 | 1.02 | 0.98 | 0.00 | ||
| Input noise (MAD of Laplacian) | 1.02 | 0.53 | 0.10 | 1.11 | 0.22 | 0.08 | 0.95 | 0.53 | |
| Required gain (label/input lum.) | 0.84 | 0.48 | 0.90 | 0.039 | 1.07 | 0.09 | 0.10 | ||
| Label luminance | 1.08 | 0.95 | 0.01 | 0.86 | 0.96 | 0.18 | |||
| Set | Found in | Identical | Note |
|---|---|---|---|
| LOLv2-Real test labels (100) | LOL-v1 train | 91 | Pan [20] report 91 |
| LOLv2-Real test inputs (100) | LOL-v1 train | 86 | |
| LOLv2-Real train inputs (689) | LOL-v1 train | 399 | |
| LOL-v1 test inputs (15) | LOLv2-Real | 15 | 12 in LOLv2 train, 3 in LOLv2 test |
| LOL-v1 test labels (15) | LOLv2-Real | 0 | same captures, re-rendered labels (§ 5 ) |
| Selected iteration | Calibration gain on test (dB) | |||||
|---|---|---|---|---|---|---|
| Benchmark | Seed | by val | by test | Inflation (dB) | honest (val-fitted) | oracle |
| LOL-v1 | 0 | 9k | 6k | 0.12 | 0.10 | 0.58 |
| LOL-v1 | 1 | 10k | 14k | 0.08 | 0.10 | 0.68 |
| LOL-v1 | 2 | 11k | 8k | 0.16 | 0.09 | 0.69 |
| LOLv2-Real | 0 | 10k | 2k | 0.92 | 0.00 | 1.53 |
| LOLv2-Real | 1 | 11k | 2k | 2.03 | 0.39 | 2.21 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| PSNR | Saturation ratio | Brightness ratio | ||||||
| Method | Benchmark | Ckpt | train | test | train | test | train | test |
| HVI-CIDNet | LOL-v1 | w/o perc. | 31.90 | 23.08 | 0.98 | 0.68 | 1.00 | 1.05 |
| HVI-CIDNet | LOL-v1 | w/ perc. | 28.06 | 23.22 | 0.94 | 0.64 | 1.00 | 1.07 |
| Retinexformer | LOL-v1 | – | 27.07 | 25.15 | 0.98 | 0.71 | 1.00 | 0.97 |
| IAT | LOL-v1 | – | 25.80 | 23.38 | 0.97 | 0.59 | 0.98 | 0.99 |
| HVI-CIDNet | LOLv2-Real | bestPSNR | 22.47 | 20.56 | 0.87 | 0.68 | 0.94 | 1.26 |
| LOL-v1 | LOLv2-Real | LOLv2-Syn | |||||||
| Checkpoint | train | test | sat. ratio | train | test | bright. ratio | train | test | bright. ratio |
| PSNR | PSNR | tr te | PSNR | PSNR | tr te | PSNR | PSNR | tr te | |
| Zero-DCE | 14.14 | 14.86 | 2.48 1.40 | 12.74 | 18.06 | 0.61 0.73 | 17.59 | 17.76 | 0.86 0.83 |
| Zero-DCE++ | 14.56 | 15.36 | 2.49 1.44 | 13.01 | 18.49 | 0.65 0.75 | 17.29 | 17.58 | 0.88 0.86 |
| SCI-easy | 9.50 | 9.63 | 2.65 1.51 | 9.63 | 12.04 | 0.31 0.32 | 14.34 | 13.96 | 0.73 0.68 |
| SCI-medium | 14.00 | 14.85 | 2.61 1.49 | 12.40 | 17.36 | 0.64 0.70 | 15.12 | 15.43 | 1.06 1.02 |
Label Shift
Label shift studies how to adapt predictions when class proportions change between training and target data.
See all 23 papers