VL-Dedup: Graded Multimodal Entity Resolution for Image-Text Data Deduplication
Organizations: Zhejiang University China · The Chinese University of Hong Kong China · University of Science and Technology Liaoning China · Institute of Automation Chinese Academy of Sciences China ·
Abstract
Image-text corpora contain copies of one image, captions that describe one image in different ways, captions reused for different images, and captions that do not match their images. Each case calls for a different curation action. Deduplication and multimodal entity resolution (ER) only decide whether records are the same, and they are evaluated on labeled pairs. Yet pair scores do not tell which records end up in the same cluster or which useful captions get deleted. We therefore formulate image-text deduplication as graded multimodal ER, which decides visual identity, caption handling, and mismatches for each pair, and we evaluate it at three levels: pairs, clusters, and the retained set. Our method, VL-Dedup, groups records of the same visual entity into clusters and caps the size of each cluster. Within each cluster, a vision-language model (VLM) and CLIPScore decide which captions to keep and which to flag. We also build VL-DedupBench, with 50,000 COCO and SOP records and 66,788 labeled pairs. On it, a classifier that selects no false edge among the labeled pairs still joins millions of record pairs from different entities. Spending VLM calls on the pairs inside clusters, rather than on pairs with uncertain scores, gives better caption decisions from the same model. Even when almost all clusters are correct, keeping one record per cluster loses facts in many deletions, while the deletions of VL-Dedup lose much less information.
Figures & tables
| Relation | Full | Test |
|---|---|---|
| L0/L1/L3 | 52,618 | 7,941 |
| L4 | 5,037 | 760 |
| M | 3,599 | 543 |
| L2 | 4,481 | 634 |
| HN | 1,053 | 159 |
| All | 66,788 | 10,037 |
| Grade | Constructed relation | Same visual entity | Action reference |
|---|---|---|---|
| L0/L1/L3 | duplicate | yes | none |
| L4 | keep_augment_candidate | yes | caption labels |
| L2 | text_duplicate_only | no | none |
| HN | hard_negative | no | none |
| M | mismatch | not scored | flag_review |
| Test surface | Records | Record pairs | Same-entity | Labeled |
|---|---|---|---|---|
| Original | 5,998 | 17,985,003 | 8,701 | 9,494 |
| Mixed, Mild, Strong | 5,998 | 17,985,003 | 8,701 | 9,494 |
| MV | 8,044 | 32,348,946 | 11,685 | 15,462 |
| Larger entities | 11,498 | 66,096,253 | 107,201 | 9,494 |
| Method | Modality | Pairwise F1 | Labeled test pairs in one cluster | ||||||||||||
| Labeled | Original | Mixed | Mild | Strong | MV | Products | L0/L1/L3 | L4 | Same product | L2 | HN | Two products | |||
| Data-Juicer | Image | – | 1.0000 | 1.0000 | 0.9800 | 0.9054 | 0.5997 | 0.8572 | 0.0420 | 7,941 | 760 | 64 | 0 | 0 | 0 |
| SemDeDup | Image | – | 1.0000 | 1.0000 | 0.6672 | 0.6910 | 0.1568 | 0.8555 | 0.0219 | 7,941 | 760 | 33 | 0 | 0 | 0 |
| B2 SSCD | Image | 1.0000 | 0.9998 | 0.9998 | 0.9998 | 0.9979 | 0.8611 | 0.0900 | 7,941 | 760 | 141 | 0 | 0 | 0 | |
| Caption MinHash | Text | – | 0.0679 | 0.0722 | 0.0722 | 0.0722 | 0.0722 | 0.0614 | 0.0290 | 327 | 1 | 44 | 634 | 0 | 0 |
| SemHash | Image+text | – | 0.5164 | 0.3753 | 0.3494 | 0.4100 | 0.3599 | 0.3360 | 0.1495 | 2,791 | 243 | 259 | 15 | 0 | 0 |
| Edges of VL-Dedup | Edges of HGB | |||||||
|---|---|---|---|---|---|---|---|---|
| Algorithm | Orig. | Strong | MV | Larger | Orig. | Strong | MV | Larger |
| No cap (components) | 0.9827 | 0.9771 | 0.7386 | 0.9981 | 0.0023 | 0.0022 | 0.0017 | 0.0061 |
| Center | 0.5818 | 0.5382 | 0.5116 | 0.1328 | 0.5989 | 0.4862 | 0.4984 | 0.1388 |
| Merge-Center | 0.9827 | 0.9782 | 0.7446 | 0.9986 | 0.0024 | 0.0024 | 0.0018 | 0.0063 |
| KwikCluster | 0.9914 | 0.9817 | 0.8672 | 0.9657 | 0.7241 | 0.4708 | 0.5877 | 0.5563 |
| Louvain | 0.9827 | 0.9771 | 0.7920 | 0.9981 | 0.3045 | 0.2361 | 0.1996 | 0.7450 |
| Features | Original | Strong | MV | ||
|---|---|---|---|---|---|
| All 11 | 0.667 | 5 | 0.9937 | 0.9947 | 0.8691 |
| No copy detection | 0.250 | 5 | 0.9551 | 0.8965 | 0.8102 |
| No perceptual hash | 0.286 | 5 | 0.9605 | 0.9627 | 0.8593 |
| No image embedding | 0.500 | 5 | 0.9908 | 0.9928 | 0.8634 |
| No caption | 0.727 | 5 | 0.9969 | 0.9953 | 0.8703 |
| No cross-modal | 0.250 | 5 | 0.9953 | 0.9951 | 0.8697 |
| Reference and captions | Action of VL-Dedup |
|---|---|
| Complementary. “Birds are flying past a steeple with a clock on it.” / “A green and white church tower under a partly cloudy sky.” | keep_augment : keep both |
| Redundant. “a large plane is sitting in the dirt” / “A large plane is parked in a dirt field.” | duplicate : keep one |
| Swapped caption on a photo of a bride and groom: “A man is riding a skateboard down a road.” | : flag for review |
| Wrong object , confirmed by a human: “a couple zebra drinking water out in the desert” / “Some giraffes and some big elephants by the water.” | mismatch with : review |
| Lost under keep-one. Deleted: “a long hair cat laying on a white bed”. Kept: “A cat sleeping on a bed with a small TV in a bedroom.” | keep_augment : keep both |
| Rule | TP | FP | FN | P | R | F1 |
|---|---|---|---|---|---|---|
| CLIPScore | 324 | 38 | 48 | 0.8950 | 0.8710 | 0.8828 |
| VLM probability | 340 | 22 | 32 | 0.9392 | 0.9140 | 0.9264 |
| Combined | 347 | 17 | 25 | 0.9533 | 0.9328 | 0.9429 |
| Pair verdict | 334 | 314 | 38 | 0.5154 | 0.8978 | 0.6549 |
| , then verdict | 361 | 323 | 11 | 0.5278 | 0.9704 | 0.6837 |
| Records | Candidates | B (s) | C (s) | D (s) | , cap | , no cap |
|---|---|---|---|---|---|---|
| 10K | 289,200 | 6.2 | 3.4 | 0.05 | 3,063 | 7,127 |
| 20K | 595,089 | 15.4 | 12.9 | 0.11 | 11,970 | 33,901 |
| 30K | 900,756 | 28.6 | 18.1 | 0.18 | 26,356 | 100,387 |
| 50K | 1,502,683 | 66.7 | 18.8 | 0.41 | 71,930 | 457,215 |
| COCO | SOP | |
| Records | 580,767 | 97,503 |
| Entities | 116,091 | 21,227 |
| Same-entity pairs | 1,162,473 | 266,739 |
| 5 | 8 | |
| Stage A (min) | 60 | 36 |
| Stages B–D (min) | 120 | 5.1 |