Can the Parts Fool the Test? Counterfactual Pairing Cycles for Relational OOD

Yibo Zhou; Yirong Xiang

cs.CVOct 16, 2026

Can the Parts Fool the Test? Counterfactual Pairing Cycles for Relational OOD

Authors: Yibo Zhou, Yirong Xiang

Organizations: Beihang University · Air Force Medical Center

Abstract

Relational out-of-distribution errors are easy to describe and hard to evaluate cleanly. An image may be natural, a caption may be fluent, and all mentioned objects may be common, yet the caption may describe the wrong relation in the image. Many evaluations intended to test this behavior can be solved without checking the pairing at all, because negatives differ in caption style, template, length, object frequency, or image statistics. We propose counterfactual pairing cycles, a blocked evaluation primitive that reuses the same two images and the same two captions on both sides of the comparison, changing only which image is paired with which caption. This makes image-only, text-only, and additive component shortcuts cancel exactly in finite samples. The same contrast also gives a training loss for relational outlier exposure and a diagnostic for bilinear vision-language scores. Across synthetic controls, GQA, and COCO, cycles separate coarse image-caption matching from directional role-binding failures, while also identifying geometric size relations that remain strongly visible in global features. In a shortcut-contaminated COCO outlier-exposure stress test, a model selected by a near-perfect shortcut validation AUROC retains substantial relational signal but remains below cycle training on exact cycles; cycle training improves CycleAcc from 0.882 to 0.927. The main lesson is simple: to test whether a model understands a pairing, the benchmark should keep the parts fixed, change the pairing, and ask only about coupling. The official implementation is available online.

Figures & tables

Panel of Table 1 — Table 1: Synthetic controls. AUROC is reported for these unblocked two-sample diagnostics. The vertical tick in each mini-bar marks chance.

Split group	Additive audit	Interaction scores	Reading
GQA scene-graph cycles
Attribute / caption mismatch	$0.500\pm 0.000$	$0.594$ – $0.954$	Attribute coupling varies across families and can be strong.
Relation / object swaps	$0.500\pm 0.000$	$0.500$ – $0.835$	Performance ranges from chance to substantial relation signal.
COCO raw-image cycles
Coarse coupling	$0.500\pm 0.000$	$0.997$ – $1.000$	Coarse image-caption matching is easy.
Matched / neighbor hard	$0.500\pm 0.000$	$0.941$ – $0.987$	Coarse object signal remains strong.

Table 2: Exact-cycle diagnostics on GQA and COCO. Values are exact CycleAcc. Additive probes are implementation consistency checks. For COCO, the interaction ranges are zero-shot CLIP cosine ranges only; trained bilinear and low-rank bilinear probes are reported separately. For GQA, the interaction ranges summarize the evaluated diagonal, full-bilinear, and rank-64 probes. The GQA binding family is omitted because it yields only 18–19 validation cycles per seed, below the prespecified minimum of 30.

Panel of Figure 1 — Figure 1: Strict COCO relation cycles separate directional role binding from recoverable geometric cues. Left/right remains near chance, while size is strongly exposed and above/below and same-pair relation changes retain moderate trained-probe signal.

Split	Rank 1	Rank 4	Rank 16	Rank 64
Object-matched COCO	$0.784\pm 0.018$	$0.905\pm 0.011$	$0.936\pm 0.009$	$0.959\pm 0.001$
Text-neighbor hard	$0.720\pm 0.020$	$0.797\pm 0.026$	$0.858\pm 0.018$	$0.888\pm 0.006$
Box left/right	$0.707\pm 0.029$	$0.812\pm 0.005$	$0.843\pm 0.005$	$0.858\pm 0.019$
Strict present left/right	$0.514\pm 0.014$	$0.518\pm 0.017$	$0.489\pm 0.025$	$0.505\pm 0.036$
Strict same-pair relation diff.	$0.567\pm 0.019$	$0.614\pm 0.020$	$0.587\pm 0.023$	$0.599\pm 0.016$
Strict size	$0.687\pm 0.023$	$0.745\pm 0.014$	$0.796\pm 0.014$	$0.787\pm 0.010$

Table 3: Low-rank bilinear probes. Values are

\mathrm{CycleAcc}

. Rank helps for recoverable interactions but does not rescue strict present left/right.

Panel of Figure 2 — Figure 2: Rank sweeps characterize recoverability within the evaluated bilinear probe class. Some interactions improve with rank, while strict present left/right remains near chance.

Method / audit	Shortcut val AUROC	Exact $\mathrm{CycleAcc}$
Cycle OE	–	$\mathbf{0.9270\pm 0.0075}$
Shortcut-contaminated OE	$0.9998\pm 0.0002$	$0.8817\pm 0.0021$
Text-only shortcut probe	$1.0000\pm 0.0000$	–
Image-only shortcut probe	$0.5000\pm 0.0000$	–

Table 4: Shortcut-contaminated COCO OE stress test. Shortcut validation uses AUROC; relational transfer uses exact-cycle

\mathrm{CycleAcc}

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Cycle family	Marginal TV	Validity violations
COCO coarse object	0	0
COCO matched object	0	0
COCO box relation	0	0
COCO strict spatial	0	0
COCO strict size	0	0
GQA attribute	0	0

Appendix

Table 5: Exact cycle-construction audits. All accepted validation cycles have exact marginal balance and no validity violations. GQA binding is not reported because it does not satisfy the minimum-cycle requirement.

Dataset	Family	Train	Validation	Test
COCO	All reported families except strict same-pair	3000	500	1000
COCO	Strict same-pair	3000	466	1000
GQA	AD, CD, SR, RS, OP, OO, SS	3000	500	1000
GQA	BS binding	732–772	18–19	155–174

Appendix

Table 6: Cycle counts and image pools. Counts are reported as train/validation/test cycles. COCO train and validation cycles are constructed from image-disjoint subsets of the COCO train2017 pool, while test cycles use the separate COCO val2017 pool. GQA uses an image-disjoint 60/10/30 split.

Seed	Cycle OE	Shortcut OE	Difference
0	0.919	0.884	+0.035
1	0.934	0.880	+0.054
2	0.928	0.881	+0.047
Mean	0.9270	0.8817	+0.0453
Std.	0.0075	0.0021	0.0096

Appendix

Table 7: Per-seed Shortcut OE results. Differences are computed as Cycle OE minus Shortcut OE. Means and standard deviations are computed across seeds

0

1

, and

2

Control score	Exact CycleAcc
Text-only score	0.5000
Image-only score	0.5000
Additive image-plus-text score	0.5000

Appendix

Table 8: Additive and unimodal controls under exact CycleAcc. Every control produces tied cycle contrasts and therefore receives exactly half credit on every cycle.

Counterfactual Evaluation of VLMs

Counterfactual evaluation of vision-language models tests whether their answers change appropriately when relevant visual evidence is minimally altered.

See all 54 papers

What Paired Evaluations Reveal under Visual Perturbations
Sep 28, 2026Yongda Wei, Chen Zhang, Yifei Wang +4Image Corruption Robustness Adversarial Robustness
RePair: Turning Retrieval Failures into Counterfactual Hard Pairs
Aug 30, 2026Siyi Liu, Xiaorong Zhu, Enjun Du +6Cross-Modal Retrieval Counterfactual Data Augmentation
The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong +1
Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models
Sep 6, 2026Bayar Menzat, Maximilian Süss, Ruizhi Wang +3CoT Faithfulness Faithfulness in VLMs
Vienna University of Technology, Austria · University of Oxford, UK · Imperial College London, UK +1

Sep 28, 2026J/K move · Enter open · S save

What Paired Evaluations Reveal under Visual Perturbations

Yongda Wei, Chen Zhang, Yifei Wang, Xinyu Wang+3

Image Corruption Robustness Adversarial Robustness

Open paper AskPDF arXiv

Robustness evaluation must examine diverse visual perturbations, while benchmarks cover only some real-world conditions and physical testing is costly. Paired evaluations link clean and perturbed predictions for the same image, capturing changes in correctness, confidence, and acceptance beyond aggregate accuracy. We investigate how this image correspondence supports two needs in robustness evaluation: interpreting paired evaluation results and prioritizing samples for physical testing. To interpret paired evaluation results, we fix both sets of prediction records and vary their correspondence within each class. We prove that classwise correct-correct counts give the same sharp bounds on lost acceptance and mean true-class probability decrease among retained-correct inputs as any feasible five-state refinement. Distinguishing persistent from changed wrong answers can further constrain accepted-error transitions, while shared correspondence can establish policy orderings left unresolved by separate cost intervals. To prioritize samples for physical testing, we retain each image's synthetic responses and rank clean-correct images by their mean true-class probability under corruption. Across 44 classifiers, testing the highest-risk 20% finds 67% and 45% of failures under mild screen and print recaptures, versus 58% and 36% for clean confidence and 60% and 37% for an equal-size natural-transformation average. With both probability averaging and an A3Rank scoring adaptation, the tested corruption set yields higher mean failure recall than the natural-transform set; differences between scores depend on the source and budget. Together, these findings show that the value of correspondence depends on the evaluation objective: classwise counts suffice for specified reliability bounds, while image-specific synthetic responses improve the allocation of physical tests within the evaluated pool.

Abstract

Counterfactual Evaluation of VLMs

Explore similar work

What Paired Evaluations Reveal under Visual Perturbations

RePair: Turning Retrieval Failures into Counterfactual Hard Pairs

Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models