Only Means Only: what a harmonization benchmark score averages over
Organizations: Polytechnic Faculty, University of Zenica
Abstract
Image harmonization methods adjust the colour and lighting of an object pasted into a photograph so that it looks native to its new scene, and they are ranked by average error on one benchmark whose test images are real photographs with one object recoloured. One photograph can yield several such test images, and the benchmark promises that no photograph appears in both the training and the test split. We find that 1,399 photographs do, and the composites built on them are a quarter of the test set. Duplicate checks over file names or bytes miss this, because every composite is a different file. On that quarter the three methods we score end to end read two to five times lower error than on the rest, but most of the gap is composition rather than the sharing: the shared composites come almost entirely from one of the benchmark's four source collections, and inside that collection the gap falls to about one and a half times. An official score is therefore an average over two populations, mixed in a proportion the benchmark does not report. We release a test list of 5,581 composites whose construction identities do not appear in training, with four confirmed near-duplicate pairs disclosed rather than removed, adoptable without retraining at the price of one rescore per checkpoint. We leave open whether the sharing itself changes what a model learns.
Figures & tables
| Method (MSE ) | ret. | excl. | offic. | |
|---|---|---|---|---|
| 5,581 | 1,823 | 7,404 | excl. ret. | |
| DoveNet | ||||
| Harmonizer | ||||
| Editor (ours) |
| Method | Rows | Step (MSE ) | 95% CI | Detection limit |
|---|---|---|---|---|
| DoveNet | pooled | |||
| HCOCO only | ||||
| Harmonizer | pooled | |||
| HCOCO only | ||||
| Editor (ours) | pooled | |||
| HCOCO only |
| Method | Estimand | fMSE contrast | 95% joint-bootstrap CI | MDE |
|---|---|---|---|---|
| DoveNet | Change between arms in the shared-composite advantage | |||
| Harmonizer | Change between arms in the shared-composite advantage | |||
| DoveNet | Training check on the arms’ own composites | n/a | ||
| Harmonizer | Training check on the arms’ own composites | n/a |
Image Editing Evaluation
Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.
See all 92 papers