Retrieving Counterfactuals Improves Visual In-Context Learning

Guangzhi Xiong; Sanchit Sinha; Zhenghao He; Aidong Zhang

cs.CVMar 17, 2026

Retrieving Counterfactuals Improves Visual In-Context Learning

Authors: Guangzhi Xiong, Sanchit Sinha, Zhenghao He, Aidong Zhang

Organizations: University of Virginia

Abstract

Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal reasoning tasks, but they often struggle to disentangle fine-grained visual attributes and reason about underlying causal relationships. In-context learning (ICL) offers a promising avenue for VLMs to adapt to new tasks, but its effectiveness critically depends on the selection of demonstration examples. Existing retrieval-augmented approaches typically rely on passive similarity-based retrieval, which tends to select correlated but non-causal examples, amplifying spurious associations and limiting model robustness. We introduce CIRCLES (Composed Image Retrieval for Causal Learning Example Selection), a novel framework that actively constructs demonstration sets by retrieving counterfactual-style examples through targeted, attribute-guided composed image retrieval. By incorporating counterfactual-style examples, CIRCLES enables VLMs to implicitly reason about the causal relations between attributes and outcomes, moving beyond superficial correlations and fostering more robust and grounded reasoning. Comprehensive experiments on four diverse datasets demonstrate that CIRCLES consistently outperforms existing methods across multiple architectures, especially on small-scale models, with pronounced gains under information scarcity. Furthermore, CIRCLES retrieves more diverse and causally informative examples, providing qualitative insights into how models leverage in-context demonstrations for improved reasoning. Our code is available at https://github.com/gzxiong/CIRCLES.

Figures & tables

Figure 1: Illustration of how composed image retrieval provides additional causal understanding.

Figure 2: Overview of the CIRCLES framework. Given a query image $I_{q}$ and question $Q_{q}$ , the top branch illustrates correlational understanding via standard image retrieval, while the bottom branch depicts causal understanding using attribute-guided composed image retrieval with counterfactual captions from the VLM $\Phi$ . Blue/orange rounded rectangles represent image/text embeddings. $\mathcal{R}_{\text{corr}}$ and $\mathcal{R}_{\text{causal}}$ denote the retrieved in-context examples for answer prediction.

Model	Method	CUB		Flowers		OK-VQA		VizWiz		Average
Model	Method	Acc	F1	Acc	F1	EM	F1	EM	F1	EM	F1
Gemma3 -4B	None	10.56	8.19	46.76	46.50	19.12	26.70	39.85	57.26	29.07	34.66
	Random	9.98	8.10	37.01	39.51	24.81	30.74	51.84	67.08	30.91	36.36
	RICES	65.40	67.62	86.70	87.43	26.65	32.72	56.08	70.40	58.71	64.54
	MUIER	65.21	67.42	86.39	87.28	26.87	32.82	56.59	70.75	58.77	64.57
	MMICES	13.95	10.98	38.09	38.20	26.24	32.41	52.72	68.21	32.75	37.45

Table 1: Performance comparison between CIRCLES and baseline methods across different models and datasets. Average scores are computed by treating accuracy (Acc) as EM for classification tasks and averaging EM and F1 across all datasets. The best results for each model and dataset are highlighted in bold.

Figure 3: Qualitative comparison of in-context examples retrieved by RICES and CIRCLES for a CUB test image ( Magnolia Warbler ). Top: standard image retrieval (IR) neighbors used by RICES, leading to incorrect predictions. Bottom: counterfactual examples from composed image retrieval (CIR) in CIRCLES, highlighting key attribute changes and guiding the model to the correct label.

Figure 4: Performance comparison between CIRCLES and RICES on the CUB dataset under varying levels of information scarcity in the training set.

Figure 5: Comparison of CIRCLES performance using CIR implemented by CIReVL and OSrCIR on the CUB dataset with Gemma3 (4B/12B) backbones.

Panel of Figure 6 — Figure 6: Retrieval budget analysis on CUB with Gemma3-4B. (a) CIRCLES accuracy vs. number of composed images retrieved (#CIR) under different standard retrievals (#IR). (b) Accuracy as a function of the number of intervened attributes (#Attributes) and composed images (#CIR).

Model	Setting	OK-VQA		VizWiz
Model	Setting	EM	F1	EM	F1
Gemma -4B	w/o	27.72	33.64	57.40	71.65
Gemma -4B	w/	31.27	36.89	57.61	71.35
Gemma -12B	w/o	33.02	39.07	74.37	83.20
Gemma -12B	w/	37.75	43.23	74.30	82.35
Qwen -3B	w/o	41.12	45.33	73.28	75.94

Table 2: Performance comparison of CIRCLES on OK-VQA and VizWiz with (w/) and without (w/o) the question-question similarity component in CIR.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	CUB	Flowers	OK-VQA	VizWiz
Gemma3 -4B	RICES	11.1k	10.0k	9.4k	10.2k
Gemma3 -4B	CIRCLES	12.2k	11.2k	10.6k	11.4k
Qwen2.5 -VL-3B	RICES	9.9k	15.8k	12.8k	33.3k
Qwen2.5 -VL-3B	CIRCLES	10.9k	17.2k	14.1k	35.9k

Appendix

Table 3: Average token usage per question for RICES and CIRCLES. The additional VLM calls in CIRCLES introduce only a modest overhead (around 10%).

Model	Method	CUB	Flowers	OK-VQA	VizWiz
Gemma3 -4B	RICES*	64.73	86.60	26.34	56.17
Gemma3 -4B	CIRCLES	71.97	93.32	31.27	57.61
Gemma3 -12B	RICES*	75.70	96.02	36.64	74.09
Gemma3 -12B	CIRCLES	77.03	97.77	37.75	74.30

Appendix

Table 4: Compute-matched comparison between CIRCLES and RICES*. RICES* uses multiple generations (

K=3

) with self-consistency under the same inference-time call budget.

Method	CUB	Flowers	OK-VQA	VizWiz
RICES	64.42 $\pm$ 0.32	86.42 $\pm$ 0.24	26.08 $\pm$ 0.23	55.69 $\pm$ 0.07
CIRCLES	70.30 $\pm$ 0.28	92.87 $\pm$ 0.13	30.54 $\pm$ 0.21	57.04 $\pm$ 0.23

Appendix

Table 5: Robustness under stochastic decoding for Gemma3-4B. We set temperature = 1.0, repeat decoding five times, and report mean

\pm

standard deviation.

Figure 7: Qualitative examples of CIRCLES. Top: failure due to a non-discriminative extracted attribute. Middle: failure due to weak composed retrieval. Bottom: success case with better matched composed examples. Red denotes unhelpful retrieved examples or incorrect predictions, and green denotes helpful retrieved examples or the correct prediction.

Attribute Source	Gemma3-4B		Gemma3-12B
Attribute Source	Acc	F1	Acc	F1
LLM	71.97	72.39	77.03	76.90
Dataset	72.44	72.82	77.65	77.46

Appendix

Table 6: Effect of replacing VLM-extracted attributes with ground-truth attribute annotations on CUB.

Setting	CUB		Flowers		OK-VQA		VizWiz
Setting	Acc	F1	Acc	F1	EM	F1	EM	F1
IR	65.40	67.62	86.70	87.43	26.65	32.72	56.08	70.40
IR + Attr	71.99	71.91	92.91	93.53	26.16	32.63	53.83	68.82
IR + CIR	71.97	72.39	93.32	93.49	31.27	36.89	57.61	71.35

Appendix

Table 7: Ablation of CIR versus attribute-only prompting on Gemma3-4B. IR denotes standard image retrieval; Attr denotes adding textual attributes without CIR.

Similarity	OK-VQA		VizWiz
Similarity	EM	F1	EM	F1
image-image	31.27	36.89	57.61	71.35
image-image + image-text	30.14	35.67	57.81	71.63
image-image + text-text	31.37	36.74	59.09	72.64

Appendix

Table 8: CIRCLES with different implementations of the image retrieval component (Gemma3-4B).

Setting	CUB		Flowers		OK-VQA		VizWiz
Setting	Acc	F1	Acc	F1	EM	F1	EM	F1
IR only	65.40	67.62	86.70	87.43	26.65	32.72	56.08	70.40
CIR only	25.16	26.14	60.24	65.85	29.81	35.52	54.06	68.29
IR + CIR	71.97	72.39	93.32	93.49	31.27	36.89	57.61	71.35

Appendix

Table 9: Ablation of IR and CIR in CIRCLES, tested on Gemma3-4B.

Size	Random	RICES	MUIER	CIRCLES
3B	78.33	80.12	80.66	80.71
7B	85.18	88.05	87.90	88.35

Appendix

Table 10: Test accuracy (%) on ScienceQA. In-context examples are retrieved from the validation split. The experiments are run on the Qwen2.5-VL model family.

Method	CUB	Flowers	OKVQA	VizWiz
RICES	69.76	93.98	38.66	67.24
CIRCLES	72.21	97.51	39.14	68.72

Appendix

Table 11: Performance comparison on Gemma3-27B.

Panel of Figure 8 — Figure 8: Prompt template for attribute extraction.

Panel of Figure 9 — Figure 9: Prompt template for generating counterfactual caption based on the query image and identified key attribute.

Panel of Figure 10 — Figure 10: Prompt template for VLM inference used by None (zero-shot learning).

Panel of Figure 11 — Figure 11: Prompt template for VLM inference used by Random, RICES, MUIER, and MMICES.

Panel of Figure 12 — Figure 12: Prompt template for VLM inference used by CIRCLES.

Multimodal ICL

Multimodal in-context learning (ICL) studies how models use examples and other context to solve tasks involving images, text, audio, and video.

See all 46 papers

In-Context Learning Can Help Vision-Language Models Overcome Training Priors
Oct 9, 2026Kun Wang, Xindi Wu, Sanghyuk Chun +2Visual Evidence Sensitivity in VLMs In-Context Learning
Princeton University
The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors
Oct 12, 2026Shu Zhao, Tan YuIn-Context Learning Test-Time Adaptation of VLMs
NVIDIA
Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
May 4, 2026Haoyu Wang, Haonan Wang, Yuyan Chen +5Multimodal ICL In-Context Learning
Tencent QQ · Fudan University · Cornell University

Oct 9, 2026J/K move · Enter open · S save

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky+1

Princeton University

Visual Evidence Sensitivity in VLMs In-Context Learning Vision-Language Grounding Counterfactual Evaluation of VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples (

+27.2\%

for Qwen3-VL 235B and

+15.1\%

for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from

+9.7\%

+30.3\%

. We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.

Abstract

Multimodal ICL

Explore similar work

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors

Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning