RePair: Turning Retrieval Failures into Counterfactual Hard Pairs

Siyi Liu; Xiaorong Zhu; Enjun Du; Xinyu Zuo; Lisheng Duan; Haijin Liang; Jin Ma; Junfu Pu

cs.IRAug 30, 2026

RePair: Turning Retrieval Failures into Counterfactual Hard Pairs

Authors: Siyi Liu, Xiaorong Zhu, Enjun Du, Xinyu Zuo, Lisheng Duan, Haijin Liang, Jin Ma, Junfu Pu,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong · ARC Lab, Tencent

Abstract

Vision-language retrieval with CLIP-style dual encoders achieves strong cross-modal performance, yet practical accuracy often hinges on localized semantic distinctions where top-ranked near misses differ from the true match by a single critical detail. Hard-sample mining can select confusable candidates but cannot construct corrected counterparts; synthetic augmentation can generate novel samples but, without conditioning on actual model failures, targets irrelevant dimensions of hardness. We observe that a top-ranked false positive is a counterfactual scaffold---sharing most of the query's semantics while differing in a localized failure-causing residual. Minimally correcting this residual yields a hard positive of the ground truth in the same modality; the corrected and unedited versions form a hard negative pair that straddles the decision boundary, producing complementary pull--push supervision. We introduce RePair, guided by three principles---Validity, Minimality, and Locality---which mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective. On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples---26%--75% fewer than comparable methods---confirming failure-conditioned repair is more data-efficient than error-agnostic augmentation.

Figures & tables

Figure 1: The RePair operator. Top: For a text query $q$ , the retrieval system returns a near-miss false positive $p^{\mathrm{fp}}$ that shares the query’s scaffold (green: “a girl in a red shirt”) but differs on the failure-causing residual (red: wrong swing type). RePair minimally edits only the residual to produce $p^{\mathrm{edit}}$ , which carries the same semantics as the ground truth $p^{\mathrm{gt}}$ . Bottom: Before repair, $p^{\mathrm{fp}}$ falls in the FP zone between the ideal boundary and the model’s learned boundary. The repair signal pulls $p^{\mathrm{edit}}$ toward $p^{\mathrm{gt}}$ and pushes $p^{\mathrm{fp}}$ away, shifting the model boundary to eliminate the FP zone.

Figure 2: RePair pipeline overview. Validity (Mining & Validation) : A retriever mines top- $K_{\mathrm{mine}}$ false positives ( $p^{\mathrm{fp}}_{I}$ / $p^{\mathrm{fp}}_{T}$ ) in both T2I and I2T directions; a pre-edit verifier filters annotation noise and benign mismatches, ensuring only genuine failures serve as editing seeds. Minimality (Counterfactual Editing) : An instruction generator produces a minimal edit instruction $\tau_{\mathrm{edit}}$ conditioned on $(q,p^{\mathrm{fp}})$ ; a modality-specific editor applies $\tau_{\mathrm{edit}}$ to $p^{\mathrm{fp}}$ , and post-edit selection picks the candidate closest to $p^{\mathrm{fp}}$ , yielding a hard positive $p^{\mathrm{edit}}$ (edited) and a hard negative $p^{\mathrm{fp}}$ (original). The resulting pairs are organized into local contrastive grids for training ( Locality ; see Figure 3 ).

Figure 3: The $3\times 3$ local confusion grid. Diagonal entries are positive pairs; off-diagonal entries serve as hard negatives.

Method	Samples (k)	Aug. Type				Flickr30K				COCO30K
Method	Samples (k)	$I^{+}$	$I^{-}$	$T^{+}$	$T^{-}$	I2T		T2I		I2T		T2I
						R@1	R@5	R@1	R@5	R@1	R@5	R@1	R@5
Vanilla	–					87.87	98.22	74.60	92.87	60.54	83.86	44.15	71.97
NegCLIP	205				✓	88.07	98.13	74.38	92.78	61.06	84.00	44.37	72.29
SugarCrepe	145				✓	88.95	98.62	74.64	92.98	61.92	85.02	45.09	72.94
LaCLIP	145			✓		86.69	97.83	74.46	92.60	60.96	84.30	44.99	72.70

Table 1: Retrieval performance on Image-to-Text (I2T) and Text-to-Image (T2I) on Flickr30K and COCO30K. The best and second-best results are marked in bold and underlined , respectively. Augmentation modality and polarity are marked by

I^{+}

I^{-}

(image hard positive/negative) and

T^{+}

T^{-}

(text hard positive/negative).

Category	Ablation setting	$\mathrm{R@1}_{\mathrm{I2T}}$	$\mathrm{R@1}_{\mathrm{T2I}}$
RePair	-	62.64	46.11
Failure Cond. ( Val. )	w/o Failure	$60.54_{\text{{\color[rgb]{0.72,0,0}-2.10}}}$	$43.69_{\text{{\color[rgb]{0.72,0,0}-2.42}}}$
Repair Op. ( Min. $+$ Loc. )	w/o Repair (FP-as-HN)	$60.83_{\text{{\color[rgb]{0.72,0,0}-1.81}}}$	$44.20_{\text{{\color[rgb]{0.72,0,0}-1.91}}}$
Seed Valid. ( Val. )	w/o QC	$61.98_{\text{{\color[rgb]{0.72,0,0}-0.66}}}$	$44.87_{\text{{\color[rgb]{0.72,0,0}-1.24}}}$
Min.-Change ( Min. )	w/o PQC	$62.12_{\text{{\color[rgb]{0.72,0,0}-0.52}}}$	$45.33_{\text{{\color[rgb]{0.72,0,0}-0.78}}}$
Local Training ( Loc. )	Full Matrix	$61.42_{\text{{\color[rgb]{0.72,0,0}-1.22}}}$	$44.36_{\text{{\color[rgb]{0.72,0,0}-1.75}}}$

Table 2: Ablation and sensitivity analysis on COCO30K. Top: core design ablations (one design choice toggled at a time), each defined where it is discussed in Section 4.3 . Bottom: external component sensitivity. Subscripts show

\Delta

R@1 vs. the full method.

Figure 4: Case study of RePair’s synthesis on T2I (left) and I2T (right). In T2I, the query $q$ ( “playing chess alone” ) retrieves a near-miss image $I^{\mathrm{fp}}$ depicting multiple players; RePair edits only the failure-causing residual to produce $I^{\mathrm{edit}}$ , which matches the query and serves as a hard positive alongside $I^{\mathrm{gt}}$ , while the unedited $I^{\mathrm{fp}}$ becomes the paired hard negative. Symmetrically in I2T, a false-positive caption $T^{\mathrm{fp}}$ sharing the query’s scaffold ( “people watch” ) but with a mismatched residual ( “skateboard” ) is corrected to a faithful description $T^{\mathrm{edit}}$ . $I^{\mathrm{gt}}$ and $T^{\mathrm{gt}}$ denote the ground truth.

Method	Accuracy		P@1
Method	VG-Attr	VG-Rel	COCO-Ord	Flk-Ord
Vanilla	62.6	51.6	25.4	31.1
NegCLIP	62.0	49.3	26.7	31.5
SugarCrepe	62.1	51.3	26.6	32.1
LaCLIP	62.8	49.7	27.1	32.0
SimCLR	62.5	53.7	27.0	32.8

Table 3: Compositional evaluation on ARO Yuksekgonul et al. (2023) . All values in %; best and second-best in bold and underlined .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# Train	# Val	# Test	Caps/Img	Resolution
Flickr30K	29,000	1,014	1,000	5	$224^{2}$
COCO30K	29,000	5,000	5,000	5	$224^{2}$

Appendix

Table 4: Summary of dataset statistics and preprocessing details.

Optimization
Hyperparameter	Value / Search Range
Optimizer	AdamW
Learning Rate	{1e-5, 5e-6, 2e-6, 1e-6}
Weight Decay	0.1
LR Scheduler	Cosine Annealing
Warmup Ratio	0.1

Appendix

Table 5: Hyperparameter Settings and Grid Search Ranges.

Symbol	Meaning
$\mathcal{D}=\{(I_{i},T_{i})\}_{i=1}^{N}$	Paired image–text dataset
$I,T$	An image / a text (generic)
$z_{I}(I),z_{T}(T)$	$\ell_{2}$ -normalized embeddings
$s(I,T)=z_{I}(I)^{\top}z_{T}(T)$	Cosine similarity
$\ell(I,T)=\alpha\,s(I,T)$	Training/retrieval logit
$\alpha$ (or $\tau=1/\alpha$ )	Logit scale (or temperature)

Appendix

Table 6: Notation used in failure-driven counterfactual synthesis and

3\times 3

grid training.

Method	Samples (k)	Aug. Type				I2T				T2I
Method	Samples (k)	$I^{+}$	$I^{-}$	$T^{+}$	$T^{-}$	R@1	R@5	R@10	MRR	R@1	R@5	R@10	MRR
Vanilla	–					87.87	98.22	99.30	0.9247	74.60	92.87	96.21	0.8262
NegCLIP	205				✓	88.07	98.13	99.51	0.9235	74.38	92.78	96.19	0.8264
SugarCrepe	145				✓	88.95	98.62	99.51	0.9273	74.64	92.98	96.25	0.8284
LaCLIP	145			✓		86.69	97.83	99.21	0.9157	74.46	92.60	96.13	0.8257
SimCLR	145	✓				88.26	98.32	99.41	0.9245	74.30	92.78	96.11	0.8258

Appendix

Table 7: Full retrieval performance on Image-to-Text (I2T) and Text-to-Image (T2I) on Flickr30K . The best and second-best results are marked in bold and underlined , respectively. Augmentation modality and polarity are marked by

I^{+}

I^{-}

(image hard positive/negative) and

T^{+}

T^{-}

(text hard positive/negative). Combined baseline abbreviations: La = LaCLIP, SC = SugarCrepe, CF = COCO-CF, AL = ALiA.

Method	Samples (k)	Aug. Type				I2T				T2I
Method	Samples (k)	$I^{+}$	$I^{-}$	$T^{+}$	$T^{-}$	R@1	R@5	R@10	MRR	R@1	R@5	R@10	MRR
Vanilla	–					60.54	83.86	90.94	0.7070	44.15	71.97	81.74	0.5679
NegCLIP	205				✓	61.06	84.00	91.24	0.7134	44.37	72.29	81.95	0.5710
SugarCrepe	145				✓	61.92	85.02	91.14	0.7167	45.09	72.94	82.69	0.5778
LaCLIP	145			✓		60.96	84.30	91.00	0.7267	44.99	72.70	82.32	0.5778
SimCLR	145	✓				61.34	84.14	90.86	0.7190	45.13	72.77	82.24	0.5775

Appendix

Table 8: Full retrieval performance on Image-to-Text (I2T) and Text-to-Image (T2I) on COCO30K . The best and second-best results are marked in bold and underlined , respectively. Augmentation modality and polarity are marked by

I^{+}

I^{-}

(image hard positive/negative) and

T^{+}

T^{-}

(text hard positive/negative). Combined baseline abbreviations: La = LaCLIP, SC = SugarCrepe, CF = COCO-CF, AL = ALiA.

Figure 5: A representative $3\times 3$ grid from Flickr30K centered on an outdoor sports scenario. The anchor pair $(I^{\mathrm{a}},T^{\mathrm{a}})$ depicts “a boy and a girl play cricket,” but the model retrieves a hard-negative text $T^{\mathrm{hn}}$ describing “two kids play soccer.” RePair synthesizes a corrected counterpart $T^{+}(I^{\mathrm{hn}})$ (“Two boys play cricket”) and retrieves the matched image $I^{+}(T^{\mathrm{hn}})$ , creating a grid that forces the model to distinguish between fine-grained activities (cricket vs. soccer) and subjects (boy/girl vs. two boys).

Figure 6: A $3\times 3$ grid from COCO30K capturing a kitchen scene. The anchor $(I^{\mathrm{a}},T^{\mathrm{a}})$ features a “grey counter top” with “red stools,” while the hard negative $T^{\mathrm{hn}}$ describes a “modern kitchen” with “red cabinets” and “stainless appliances.” The synthesized positive $T^{+}(I^{\mathrm{hn}})$ corrects the attributes to match the hard-negative image $I^{\mathrm{hn}}$ (“wooden counter top,” “black chairs”), ensuring that the model learns to disentangle compositional attributes like color and material within the shared kitchen domain.

Figure 7: Distribution of synthesized and mined samples for COCO30K. The four panels show: (Top-Left) Number of hard-positive captions generated per image anchor; (Top-Right) Number of hard-positive images generated per caption anchor; (Bottom-Left) Distribution of hard-negative images linked to each image anchor; (Bottom-Right) Distribution of hard-negative captions linked to each caption anchor. Note the long-tail distribution in hard negatives (log scale) versus the controlled sparsity of hard positives.

Figure 8: Distribution of synthesized and mined samples for Flickr30K. The layout follows Figure 7 . While the overall patterns are similar to COCO30K, the tail of the hard-negative distribution is shorter (max

\sim

250 vs.

\sim

400).

Figure 9: Effect of synthetic sample budget on Recall@1 for Flickr30K. (a) Image-to-Text and (b) Text-to-Image retrieval across 20k–100k synthesized samples. RePair consistently outperforms all baselines at every budget level, while error-agnostic methods saturate or regress as sample count grows.

Dataset	Dir	In	Out	Tot	In	Out	Tot	In	Out	Tot	In	Out	Tot
		Quality Control			Instruction Generation			Editing			Total
Flickr30K	I2T	$707.3$	$80.3$	$787.6$	$981.4$	$61.7$	$1,043.1$	$597.9$	$45.9$	$643.8$	$2,286.6$	$187.9$	$2,474.5$
	T2I	$1,064.6$	$101.0$	$1,165.6$	$279.0$	$25.8$	$304.8$	-	-	-	$1,343.6$	$126.8$	$1,470.4$
	Combined	$1,771.9$	$181.3$	$1,953.2$	$1,260.3$	$87.5$	$1,347.8$	$597.9$	$45.9$	$643.8$	$3,630.2$	$314.7$	$3,944.9$
COCO30K	I2T	$701.9$	$69.5$	$771.4$	$706.7$	$31.1$	$737.8$	$382.2$	$25.2$	$407.4$	$1,790.8$	$125.8$	$1,916.6$
	T2I	$1,063.3$	$95.3$	$1,158.6$	$214.9$	$20.6$	$235.5$	-	-	-	$1,278.2$	$115.9$	$1,394.1$

Appendix

Table 9: Token usage (K tokens) by dataset, direction, and stage. “Total” sums over Stage 1–3; “Combined” sums I2T and T2I.

Method	QC?	Samples (K)	LLM (K)	Image (K)	LLM ratio	Image ratio
NegCLIP	No	205	0	0	0 $\times$	0 $\times$
LaCLIP	No	145	145	0	0.41 $\times$	0 $\times$
TripletCLIP	No	145	145	145	0.41 $\times$	0.81 $\times$
SugarCrepe	Yes	145	435–1,305	0	1.23–3.68 $\times$	0 $\times$
ALiA	Yes	145	145	435	0.41 $\times$	2.42 $\times$
COCO-CF	Yes	145	435	435	1.23 $\times$	2.42 $\times$

Appendix

Table 10: Unified generation overhead comparison. “LLM (K)” and “Image (K)” denote total LLM API calls and image-generation invocations in thousands. Baselines with native QC are aligned at 3-candidates-to-1-selected to match RePair’s

M{=}3

protocol. Ratios are relative to RePair (Full).

Variant	Samples	LLM/sample	Image/sample	Total LLM / Image
RePair-I2T	47K	5	0	235K / 0
RePair-T2I	60K	2	3	120K / 180K
Full	107K	—	—	355K / 180K

Appendix

Table 11: RePair per-sample call breakdown.

Dataset	Direction	Criterion 1 (%)	Criterion 2 (%)
Flickr30K	T2I	32.39	67.61
Flickr30K	I2T	57.61	42.39
COCO30K	T2I	25.85	74.15
COCO30K	I2T	27.93	72.07

Appendix

Table 12: Distribution of QC-filtered samples by criterion. Each row sums to 100% of total filtered samples for that setting.

Metric	Value
Precision	90.2%
Recall	83.9%
Accuracy	82.0%
Cohen’s $\kappa$	0.58

Appendix

Table 13: Agreement between QC decisions and human annotators (

N{=}400

, COCO30K).

Criterion	Overall	I2T	T2I
Original FP genuinely wrong	99.5	99.0	100.0
Edited sample matches query	95.8	96.5	95.0
Scaffold preserved	97.0	99.3	94.8
Edit is minimal	93.3	96.8	89.8
Valid hard contrast	92.5	95.5	89.5

Appendix

Table 14: Verification of retained post-edit pairs by human annotators (%;

N{=}800

, 200 per dataset

\times

direction split).

	COCO30K	Flickr30K
I2T (text edit)	98.5	92.5
T2I (image edit)	92.0	87.0

Appendix

Table 15: Valid hard-contrast rate (%) by dataset and direction.

Error Type	I2T (%)	T2I (%)
Object Error	44.0	48.3
Hallucination	36.9	0.5
Action Error	34.1	26.6
Attribute Error	33.7	10.7
Count Error	18.8	12.4
Relation Error	12.1	15.8

Appendix

Table 16: Error type distribution on COCO30K (multi-label; percentages sum to

>

100%). Column maxima and the I2T Hallucination rate are shown in bold .

Error Type	Mean	Std	Median
Attribute Error	0.912	0.063	0.932
Relation Error	0.882	0.086	0.904
Action Error	0.875	0.093	0.901
Count Error	0.873	0.100	0.899
Object Error	0.860	0.104	0.891

Appendix

Table 17: Editing quality by error type (COCO30K T2I, cosine similarity between original FP and edited counterpart). Hallucination is omitted due to insufficient T2I samples (

n{=}3

Failure Mode	COCO30K		Flickr30K		Total
Failure Mode	I2T	T2I	I2T	T2I	Total
Semantic mismatch	2	8	12	12	34
Scaffold drift	0	8	0	13	21
Artifact	0	0	0	1	1
Ambiguous FP	1	0	3	0	4
Total	3	16	15	26	60

Appendix

Table 18: Editing failure modes among the 60 shortfall records from the post-edit verification (

N{=}800

Iteration	$N_{\mathrm{I2T}}$	$N_{\mathrm{T2I}}$	$\mathrm{R@1}_{\mathrm{I2T}}$	$\mathrm{R@1}_{\mathrm{T2I}}$
Iter0	$47.0\mathrm{k}$	$60.0\mathrm{k}$	$62.64$	$46.11$
Iter1	$58.1\mathrm{k}_{\text{+11.1k}}$	$81.7\mathrm{k}_{\text{+21.7k}}$	$63.02_{\text{{\color[rgb]{0,0.88,0}+0.38}}}$	$46.32_{\text{{\color[rgb]{0,0.88,0}+0.21}}}$
Iter2	$62.3\mathrm{k}_{\text{+4.2k}}$	$92.3\mathrm{k}_{\text{+10.6k}}$	$63.11_{\text{{\color[rgb]{0,0.88,0}+0.09}}}$	$46.40_{\text{{\color[rgb]{0,0.88,0}+0.08}}}$

Appendix

Table 19: Multi-round synthesis: Recall@1 evolution on I2T and T2I retrieval.

Cross-Modal Retrieval

Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.

See all 179 papers

Half-Truths Break Similarity-Based Retrieval
Feb 27, 2026Bora Kargi, Arnas Uselis, Seong Joon OhContrastive Learning for VLMs Vision-Language Models
University of Tübingen, Tübingen AI Center · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)
What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?
Oct 7, 2026Nikos Giakoumoglou, Paschalis Giakoumoglou, Andreas Floros +2Vision-Language Pretraining Contrastive Learning
Imperial College London, London, UK · Centre for Research and Technology Hellas, Thessaloniki, Greece
Detecting and Repairing Hallucinations in Retrieval-Augmented Generation
Aug 29, 2026Sai Krishna Reddy Mulakkayala, Niki van Stein, Aske PlaatHallucination in RAG LLM Hallucination Mitigation
LIACS, Leiden University, Leiden, The Netherlands

Feb 27, 2026J/K move · Enter open · S save

Half-Truths Break Similarity-Based Retrieval

Bora Kargi, Arnas Uselis, Seong Joon Oh

University of Tübingen, Tübingen AI Center · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)

Contrastive Learning for VLMs Vision-Language Models Vision-Language Retrieval Vision-Language Alignment Compositional Representation Learning Counterfactual Evaluation of VLMs

Open paper AskPDF arXiv

When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP

Abstract

Cross-Modal Retrieval

Explore similar work

Half-Truths Break Similarity-Based Retrieval

What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?

Detecting and Repairing Hallucinations in Retrieval-Augmented Generation