VL-Dedup: Graded Multimodal Entity Resolution for Image-Text Data Deduplication
Authors: Zhenxi Liu, Guanze Chen, Dehui Yu, Xingyu Wang, Bingxu Hou, Chengwei Li, Tao Ma, Ke Wang, Gongsheng Yuan
Organizations: Zhejiang University China · The Chinese University of Hong Kong China · University of Science and Technology Liaoning China · Institute of Automation Chinese Academy of Sciences China · Institute of Computing Technology Chinese Academy of Sciences China
Image-text corpora contain copies of one image, captions that describe one image in different ways, captions reused for different images, and captions that do not match their images. Each case calls for a different curation action. Deduplication and multimodal entity resolution (ER) only decide whether records are the same, and they are evaluated on labeled pairs. Yet pair scores do not tell which records end up in the same cluster or which useful captions get deleted. We therefore formulate image-text deduplication as graded multimodal ER, which decides visual identity, caption handling, and mismatches for each pair, and we evaluate it at three levels: pairs, clusters, and the retained set. Our method, VL-Dedup, groups records of the same visual entity into clusters and caps the size of each cluster. Within each cluster, a vision-language model (VLM) and CLIPScore decide which captions to keep and which to flag. We also build VL-DedupBench, with 50,000 COCO and SOP records and 66,788 labeled pairs. On it, a classifier that selects no false edge among the labeled pairs still joins millions of record pairs from different entities. Spending VLM calls on the pairs inside clusters, rather than on pairs with uncertain scores, gives better caption decisions from the same model. Even when almost all clusters are correct, keeping one record per cluster loses facts in many deletions, while the deletions of VL-Dedup lose much less information.
Figures & tables
Figure 1. (a) Three relations in an image–text corpus and the action that each needs. (b) Four test records at the pair, graph, and retained-set levels.
Figure 2. Overview of VL-Dedup. Stages A–D build visual-entity clusters of at most K records. Stages E–F set caption and mismatch actions inside the clusters and export the retained set.
Relation
Full
Test
L0/L1/L3
52,618
7,941
L4
5,037
760
M
3,599
543
L2
4,481
634
HN
1,053
159
All
66,788
10,037
Table 1. Relation counts of VL-DedupBench over all partitions (Full) and in the test partition (Test).
Grade
Constructed relation
Same visual entity
Action reference
L0/L1/L3
duplicate
yes
none
L4
keep_augment_candidate
yes
caption labels
L2
text_duplicate_only
no
none
HN
hard_negative
no
none
M
mismatch
not scored
flag_review
Table 2. Constructed pair relations and their action references.
Figure 3. Edited copies of one test image and three photos of one test product in MV.
Test surface
Records
Record pairs
Same-entity
Labeled
Original
5,998
17,985,003
8,701
9,494
Mixed, Mild, Strong
5,998
17,985,003
8,701
9,494
MV
8,044
32,348,946
11,685
15,462
Larger entities
11,498
66,096,253
107,201
9,494
Table 3. Test populations of the graph level. Records exclude swapped captions.
Method
Modality
K
Pairwise F1
Labeled test pairs in one cluster
Labeled
Original
Mixed
Mild
Strong
MV
Products
L0/L1/L3 ↑
L4 ↑
Same product ↑
L2 ↓
HN ↓
Two products ↓
Data-Juicer
Image
–
1.0000
1.0000
0.9800
0.9054
0.5997
0.8572
0.0420
7,941
760
64
0
0
0
SemDeDup
Image
–
1.0000
1.0000
0.6672
0.6910
0.1568
0.8555
0.0219
7,941
760
33
0
0
0
B2 SSCD
Image
∞
1.0000
0.9998
0.9998
0.9998
0.9979
0.8611
0.0900
7,941
760
141
0
0
0
Caption MinHash
Text
–
0.0679
0.0722
0.0722
0.0722
0.0722
0.0614
0.0290
327
1
44
634
0
0
SemHash
Image+text
–
0.5164
0.3753
0.3494
0.4100
0.3599
0.3360
0.1495
2,791
243
259
15
0
0
Table 4. Visual identity on the test surfaces. The left block gives pairwise F1 (Equation 3 ) over the 9,494 labeled test pairs (Labeled) and over all test record pairs. The right block counts the labeled test pairs of each relation whose two records share a cluster, on the original records and, for the product columns, on MV. ↑ marks pairs that should share a cluster, and ↓ pairs that should not. Bold marks the best F1 in each column.
Figure 4. One component of the VL-Dedup graph (a) without the cap and (b) with K=5 .
Figure 5. Pairwise F1 as the size cap K changes. Dotted lines mark the values of K chosen on validation.
Edges of VL-Dedup
Edges of HGB
Algorithm
Orig.
Strong
MV
Larger
Orig.
Strong
MV
Larger
No cap (components)
0.9827
0.9771
0.7386
0.9981
0.0023
0.0022
0.0017
0.0061
Center
0.5818
0.5382
0.5116
0.1328
0.5989
0.4862
0.4984
0.1388
Merge-Center
0.9827
0.9782
0.7446
0.9986
0.0024
0.0024
0.0018
0.0063
KwikCluster
0.9914
0.9817
0.8672
0.9657
0.7241
0.4708
0.5877
0.5563
Louvain
0.9827
0.9771
0.7920
0.9981
0.3045
0.2361
0.1996
0.7450
Table 5. Pairwise test F1 of clustering algorithms on the same selected edges. Larger denotes the larger entities. Bold marks the best value in each column.
Features
τ
K
Original
Strong
MV
All 11
0.667
5
0.9937
0.9947
0.8691
No copy detection
0.250
5
0.9551
0.8965
0.8102
No perceptual hash
0.286
5
0.9605
0.9627
0.8593
No image embedding
0.500
5
0.9908
0.9928
0.8634
No caption
0.727
5
0.9969
0.9953
0.8703
No cross-modal
0.250
5
0.9953
0.9951
0.8697
Table 6. Pairwise test F1 when the classifier of Stage C is retrained without one group of features. Each row selects τ and K on validation.
Reference and captions
Action of VL-Dedup
Complementary. “Birds are flying past a steeple with a clock on it.” / “A green and white church tower under a partly cloudy sky.”
keep_augment : keep both
Redundant. “a large plane is sitting in the dirt” / “A large plane is parked in a dirt field.”
duplicate : keep one
Swapped caption on a photo of a bride and groom: “A man is riding a skateboard down a road.”
β=0.020<t : flag for review
Wrong object , confirmed by a human: “a couple zebra drinking water out in the desert” / “Some giraffes and some big elephants by the water.”
mismatch with β≥t : review
Lost under keep-one. Deleted: “a long hair cat laying on a white bed”. Kept: “A cat sleeping on a bed with a small TV in a bedroom.”
keep_augment : keep both
Table 7. Test records inside clusters of VL-Dedup and the actions it takes on them.
Rule
TP
FP
FN
P
R
F1
CLIPScore α
324
38
48
0.8950
0.8710
0.8828
VLM probability q
340
22
32
0.9392
0.9140
0.9264
Combined β
347
17
25
0.9533
0.9328
0.9429
Pair verdict
334
314
38
0.5154
0.8978
0.6549
β , then verdict
361
323
11
0.5278
0.9704
0.6837
Table 8. Mismatch detection on the labeled test pairs inside clusters, with 372 M pairs as positives and 8,696 same-entity pairs as negatives. Bold marks the best precision, recall, and F1.
Figure 6. Retained-set audit of 50 test clusters. (a) Deleted records by auditor outcome. (b) What the VL-Dedup export does with the records that keep-one loses.
Records
Candidates
B (s)
C (s)
D (s)
∣U∣ , cap
∣U∣ , no cap
10K
289,200
6.2
3.4
0.05
3,063
7,127
20K
595,089
15.4
12.9
0.11
11,970
33,901
30K
900,756
28.6
18.1
0.18
26,356
100,387
50K
1,502,683
66.7
18.8
0.41
71,930
457,215
Table 9. Measured cost of Stages B–D (median of three runs) and the pairs inside clusters, ∣U∣ , with K=5 and without a cap.
COCO
SOP
Records
580,767
97,503
Entities
116,091
21,227
Same-entity pairs
1,162,473
266,739
K
5
8
Stage A (min)
60
36
Stages B–D (min)
120
5.1
Table 10. Stages A–D on two corpora outside VL-DedupBench. False joins and pair GPU hours refer to the clusters of VL-Dedup. F1 is over all record pairs of COCO and over the record pairs of the SOP test products.
Vision-language compositionality is the ability to identify which properties, relations, quantities, and states bind to which entities in both an image and a caption, and to match those bindings across modalities. Existing benchmarks evaluate this ability through image-text matching: a model ranks candidate image-caption match similarities and succeeds when the correct match comes on top. In such datasets, paired images differ by more than a single binding: scene, layout, and content all differ, with 97% of pixels changing on average. Such pairs do not isolate compositionality: results are confounded by global image differences. We introduce Compositional Single-EdiT (CompoSET), a benchmark dataset in which each base image-caption pair has an edited counterpart, or foil, that preserves the scene but differs by a single controlled edit, such as color, count, spatial relation, or object state. Base and foil images preserve naturalistic scene complexity yet are visually near-identical: only 8% of pixels change on average. Our dataset has many foils per base scene (~22) in order to separate scene difficulty from edit difficulty, a distinction one-foil-per-base benchmarks cannot make. CompoSET contains 1,776 instances across 80 naturalistic scenes, spanning 16 edit types in 4 semantic categories, with captions at 3 verbosity levels. We test 22 dual-encoder VLMs and find that all models perform at or below chance level on CompoSET, despite scoring well on prior benchmarks. This suggests that apparent progress under prior image-text matching protocols may be an artifact arising from confounds in benchmark design, and that compositional understanding in dual-encoder VLMs is far from solved. CompoSET addresses the confounds and offers a better controlled benchmark to measure future progress.
An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still judge every image-caption pair in isolation. We ask what these neighboring captions contribute when a frozen encoder consults them at inference time. Fusing image similarity with similarity to an image's own captions improves text-to-image R@1 by 3.4 to 16.8 points across nine frozen encoders when the captions come from the same annotators as the queries. The gain falls to 1.9 to 7.2 points when an off-the-shelf captioner writes them, and to at most one point when they come from a different annotation source; a paraphrase-trained text encoder strengthens the text side further. A natural refinement, scoring the part of the image that the captions leave unexplained, appears to add much more under a leave-one-out evaluation. We show that this advantage arises because the correct candidate's supports are built differently from those of the negative candidates, with set size and query removal each contributing about half, and that under a fixed support index the residual reduces to a reweighting of what the captions already provide. On compositional benchmarks, the captions help by matching candidates to human descriptions of the image, without improving image-text grounding. The code is available at: https://github.com/nhatkhangcs/clip_consensus
Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from "visual shortcuts", as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone. To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.e. related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose Multi-Image MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document. Our experiments validate the limitations of existing benchmarks and demonstrate the effectiveness of RETINA and MIMIR. Our project is available at: Project Page.