Visual Grounding First, Multimodal In-context Learning Follows

Minhyuk Seo; Minjae Lee; Chaeeun Lee; Wei Lin; Muhammad Jehanzeb Mirza; Tinne Tuytelaars; Jonghyun Choi

cs.CVOct 22, 2026

Visual Grounding First, Multimodal In-context Learning Follows

Authors: Minhyuk Seo, Minjae Lee, Chaeeun Lee, Wei Lin, Muhammad Jehanzeb Mirza, Tinne Tuytelaars, Jonghyun Choi

Organizations: UC Berkeley · Seoul National University · Xero · MIT CSAIL ·

Abstract

LLMs have shown strong in-context learning (ICL) capability, but extending it to Multimodal LLMs (MLLMs) remains challenging. Prior multimodal ICL methods often rely on ICL-specific datasets for additional training or task vector extraction, which can improve performance on the same ICL benchmark used for adaptation. However, they do not generalize well to other benchmarks and often induce forgetting in previously well-solved tasks. For this reason, we identify 'visual grounding' as the key bottleneck in multimodal ICL; MLLMs often fail to attend to task-relevant visual evidence in demonstrations, instead over-relying on textual cues and producing hallucinated outputs. To address this, we propose ProCoRe to enhance visual grounding through contrastive reinforcement learning on multimodal contrastive data for improving multimodal ICL. ProCoRe generates captions for contrastive image pairs, decomposes them into propositions, and optimizes contrastive rewards that encourage alignment with corresponding images while discouraging mismatched ones. Despite never seeing ICL-formatted training examples, ProCoRe improves average multimodal ICL classification accuracy by 5.6% and captioning ROUGE-L by 13.8% over the off-the-shelf Qwen-VL-3-8B model, outperforming ICL-data-dependent state of the arts across benchmarks. Finally, we introduce PICL, a personalized multimodal ICL captioning benchmark that tests whether MLLMs genuinely understand visual demonstrations rather than relying on pre-trained semantic priors.

Figures & tables

Figure 1: Visual grounding in multimodal ICL with generic vs. personalized identifiers. Attention maps show visual-token attention when predicting the highlighted token using Qwen-VL-3-8B. (a) When a generic identifier ( e.g. , “ monkey ”) seen during pre-training appears in the ICL demonstrations, both baselines and ProCoRe attend to visual regions relevant to the predicted token ( i.e. , monkey ). (b) In contrast, when the generic identifier is replaced with a personalized identifier ( e.g. , “ Jaden ”), baselines fail to attend to the relevant region and thereby fall back to generic terms ( e.g. , “ character ”, “ monkey ”), whereas ProCoRe correctly predicts the personalized concept ( i.e. , “ Jaden ”) with strong visual grounding. See Sec. A.17 for more examples.

Figure 2: ICL data-dependent methods show limited generalization. Blue-shaded bars denote ICL data-dependent baselines trained on TrueMICL ICL dataset, while red-shaded bars denote those based on ISEKAI. They often outperform the off-the-shelf model when evaluated on the same ICL dataset used for training ( i.e. , train: TrueMICL $\rightarrow$ test: TrueMICL), but degrade significantly when their training and test ICL datasets differ ( i.e. , train: ISEKAI $\rightarrow$ test: TrueMICL). In contrast, even without using any ICL data, ProCoRe outperforms them through strong visual grounding.

Figure 3: Overview of ProCoRe. (1) MLLM $f$ predict captions $\hat{t}_{1},\hat{t}_{2}$ for contrastive images. (2) Decompose them into propositions using ChatGPT: unique to $I_{1}$ ( i.e. , $\hat{P}_{1}$ ), $I_{2}$ ( i.e. , $\hat{P}_{2}$ ), or shared ( i.e. , $\hat{P}_{s}$ ). (3) Compute proposition-level contrastive reward: $R_{I}$ and $R_{T}$ encourage $\hat{t}_{1}$ to reflect discriminative visual details ( i.e. , fire hydrant vs. parking meter), while $R_{s}$ preserves shared details.

Figure 4: Example of 2-shot PICL. (left) Query image alone cannot infer the personalized identity. (right) (1) Identifying the dinosaur as “Charlotte” in the first demo and (2) linking it to the dinosaur in the second enables (3) inferring that the pink beaver is “Emma” in the second, allowing correct prediction of the personalized identifier for the query. See Fig. 10 and Fig. 11 for more examples.

Category	ICL Training Dataset	Method	MS COCO					PICL
Category	ICL Training Dataset	Method	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	1-shot	2-shot	3-shot	4-shot
Off-the-shelf	-	-	15.53	19.02	21.21	23.52	25.65	24.50	25.40	24.78	28.30	29.29
Retrieval and ordering	-	RICES (NeurIPS22)	15.53	19.02	21.08	23.36	25.70	24.50	25.40	26.44	27.14	28.20
		VICL (ACL24)	17.41	21.52	23.44	24.16	27.63	23.12	26.77	27.38	25.96	29.41
		MMICES (WACV25)	15.53	19.02	21.65	22.52	25.16	24.50	25.40	21.70	23.82	24.79
Task vector	MS COCO	MTV (NeurIPS24)	15.84	18.36	18.68	18.53	19.02	12.11	15.02	16.41	15.88	17.63

Table 1: Comparison in multimodal ICL captioning benchmarks. Best and second-best scores are bolded and underlined per dataset and shot. We use Qwen-VL-3-8B and apply random ordering of demonstrations for all baselines except retrieval- and ordering-based methods.

Category	ICL Training Dataset	Method	TrueMICL					ISEKAI
Category	ICL Training Dataset	Method	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	2-shot	4-shot	8-shot	12-shot
Off-the-shelf	-	-	27.11	44.79	48.83	55.03	62.70	0.16	37.98	83.68	93.79	96.42
Retrieval and ordering	-	RICES (NeurIPS22)	27.11	44.79	47.16	51.83	61.86	0.16	36.31	81.21	90.53	92.36
		VICL (ACL24)	28.22	45.44	50.01	55.12	61.80	1.12	37.81	74.64	88.93	88.41
		MMICES (WACV25)	27.11	44.79	48.40	58.07	63.04	0.16	34.55	73.09	85.27	84.87
Task vector	TrueMICL	MTV (NeurIPS24)	23.80	36.45	42.52	45.30	55.70	3.04	32.88	81.98	94.99	90.12

Table 2: Comparison in multimodal ICL classification benchmarks. Best and second-best scores are bolded and underlined per dataset and shot. We use Qwen-VL-3-8B and apply random ordering of demonstrations for all baselines except retrieval- and ordering-based methods.

Figure 5: Effect of ICL training on simple VQA tasks. ICL training-based methods, which perform additional instruction tuning on ICL datasets, degrade performance on previously well-solved simple VQA tasks ( i.e. , IconQA, TextVQA) due to forgetting. In contrast, ProCoRe improves these tasks by enhancing visual grounding, while also achieving strong multimodal ICL performance (Tab. 2 ).

Figure 6: Comparison of target attention score (TAS) on multimodal ICL captioning benchmarks. While hallucination-mitigation baselines sometimes enhance TAS compared to off-the-shelf model, our proposed ProCoRe significantly outperforms both off-the-shelf model and baselines.

Method	MS COCO					PICL
Method	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	1-shot	2-shot	3-shot	4-shot
Off-the-Shelf (Qwen-VL-3-8B)	15.53	19.02	21.21	23.52	25.65	24.50	25.40	24.78	28.30	29.29
SFT	19.48	27.92	30.11	26.84	37.62	22.13	27.88	23.41	30.74	26.81
Sentence-Level Contrastive Reward	36.02	45.73	43.21	44.11	42.58	32.88	34.74	31.92	34.02	33.41
ProCoRe ( Ours )	39.12	46.88	46.41	46.26	46.03	33.54	36.11	34.89	36.71	36.76

Table 3: Ablations of ProCoRe. Gains in ProCoRe stem from proposition-level contrastive rewards, not contrastive-pair training it self, with clear improvements over sentence-level rewards.

Figure 7: Comparison of hallucinations on MS COCO ICL captioning benchmark. ProCoRe consistently reduces hallucinations across varying numbers of demonstrations compared to baselines.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Mean $\pm$ SD	Median
Agreement (%)	$95.91{\pm}2.98$	$96.67$
Cohen’s $\kappa$	$0.92{\pm}0.08$	$0.93$
Precision (%)	$96.97{\pm}4.72$	$100$
Recall (%)	$95.43{\pm}5.00$	$100$
F1-score (%)	$95.97{\pm}3.88$	$96.77$
Raw score (/30)	$28.77{\pm}1.19$	$29$

Appendix

Table 4: Human agreement study on personalized identifier accuracy.

Category	Method	PICL-PororoSV
Category	Method	1-shot	2-shot	3-shot	4-shot
Off-the-shelf	-	38	21	29	31
Retrieval and ordering	RICES (NeurIPS22)	33	27	29	35
	VICL (ACL24)	36	28	30	35
	MMICES (WACV25)	30	24	27	32
Task vector	MTV (NeurIPS24)	34	13	26	25

Appendix

Table 5: Comparison of personalized identifier accuracy. We measure whether predicted captions correctly match personalized identifiers in the ground-truth captions for each query. ProCoRe achieves the highest accuracy across all shots.

Panel of Figure 8 — Figure 8: Accuracy heatmap over reward weights ( $\lambda_{I}$ , $\lambda_{T}$ ) with $\lambda_{s}=4$ . If the text reward weight ( $\lambda_{T}$ ) outweighs the visual contrastive reward weight ( $\lambda_{I}$ ), the model over-focuses on textual demonstrations, weakening visual grounding and degrading performance. Conversely, if $\lambda_{I}$ is too large compared to $\lambda_{T}$ , the model underutilizes useful information from ground-truth captions, also leading to performance drops. Moderate ratios that emphasize visual rewards while retaining text signals ( e.g. , $\lambda_{I}/\lambda_{T}=2$ , red dotted box) consistently show strong performance across shots. We evaluate ICL performance on the MS COCO captioning benchmark, with InpaintCOCO used for contrastive RL training in ProCoRe.

Proposition Decomposer	0-shot	1-shot	2-shot	4-shot
GPT	$33.04{\pm}1.52$	$35.82{\pm}1.44$	$35.31{\pm}1.88$	$35.30{\pm}1.19$
Qwen3-1.7B	$32.55{\pm}1.30$	$32.74{\pm}2.30$	$36.74{\pm}1.29$	$35.20{\pm}1.66$
LLaMA3.2-3B	$32.32{\pm}1.41$	$34.44{\pm}1.72$	$36.87{\pm}1.66$	$37.60{\pm}2.15$
Qwen3-4B	$31.88{\pm}1.14$	$34.97{\pm}1.87$	$37.62{\pm}2.05$	$34.41{\pm}0.96$

Appendix

Table 6: Comparison of different LLM-based proposition decomposers in ProCoRe on PICL.

Proposition Decomposer	0-shot	1-shot	2-shot	4-shot
GPT	$32.44{\pm}1.89$	$50.91{\pm}2.74$	$55.44{\pm}1.59$	$61.37{\pm}1.68$
Qwen3-1.7B	$31.98{\pm}1.79$	$53.90{\pm}1.34$	$54.04{\pm}1.34$	$63.73{\pm}1.45$
LLaMA3.2-3B	$31.32{\pm}1.72$	$50.02{\pm}1.56$	$53.33{\pm}1.60$	$60.90{\pm}1.69$
Qwen3-4B	$33.47{\pm}1.42$	$50.26{\pm}2.18$	$54.91{\pm}0.89$	$62.38{\pm}1.67$

Appendix

Table 7: Comparison of different LLM-based proposition decomposers in ProCoRe on TrueMICL.

Method	1-shot	2-shot	3-shot	4-shot
Off-the-shelf	23.33	27.00	31.67	29.00
SFT	9.67	9.33	15.67	10.33
No-proposition-decomposition	42.67	38.67	40.00	32.00
Image contrastive only ( $\lambda_{T}=0,\lambda_{s}=0$ )	40.00	42.67	37.67	30.33
Text contrastive only ( $\lambda_{I}=0,\lambda_{s}=0$ )	33.00	30.33	31.67	32.00
No shared-content reward ( $\lambda_{s}=0$ )	53.33	38.00	44.00	39.67

Appendix

Table 8: Ablation study of ProCoRe’s individual reward components on PICL using personalized identifier accuracy.

Method	0-shot	1-shot	2-shot	4-shot	8-shot
Off-the-shelf	15.53	19.02	21.21	23.52	25.65
SFT	19.48	27.92	30.11	26.84	37.62
No-proposition-decomposition	36.02	44.73	43.21	44.11	42.58
Image contrastive only ( $\lambda_{T}=0,\lambda_{s}=0$ )	22.91	35.44	29.37	33.88	31.26
Text contrastive only ( $\lambda_{I}=0,\lambda_{s}=0$ )	24.12	36.91	27.54	34.26	28.93
No shared-content reward ( $\lambda_{s}=0$ )	30.06	30.58	30.11	35.49	37.82

Appendix

Table 9: Ablation study of ProCoRe’s individual reward components on TrueMICL using ROUGE-L.

Category	ICL Training Dataset	Method	VL-ICL					MIC
Category	ICL Training Dataset	Method	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	1-shot	2-shot	4-shot	8-shot
Off-the-shelf	-	-	19.29	45.13	59.25	57.42	63.25	16.49	24.90	30.23	35.48	42.46
Retrieval and ordering	-	RICES (NeurIPS22)	19.29	45.13	44.67	53.13	57.88	16.49	24.90	30.70	37.32	45.48
		VICL (ACL24)	15.84	41.72	41.03	53.68	58.21	14.11	27.42	32.88	33.21	45.03
		MMICES (WACV25)	19.29	45.13	44.08	53.33	59.84	16.49	24.90	30.68	36.62	41.94
Task vector	VL-ICL	MTV (NeurIPS24)	20.96	37.79	38.92	44.04	48.88	33.74	35.21	36.02	37.88	38.41

Appendix

Table 10: Quantitative comparison on the VL-ICL benchmark and MIC. Best results for each shot are highlighted in bold, while second-best results are underlined. All methods use Qwen-VL-3-8B as the base MLLM. Demonstrations are randomly ordered for all methods except retrieval- and ordering-based approaches.

Category	ICL Training Dataset	Method	MS COCO					PICL
Category	ICL Training Dataset	Method	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	1-shot	2-shot	3-shot	4-shot
Off-the-shelf	-	Random	26.92	31.22	33.25	35.83	36.26	18.44	21.86	24.56	25.84	28.01
Retrieval and ordering	-	RICES (NeurIPS22)	26.92	31.22	31.84	33.70	35.41	18.44	21.86	25.01	26.31	27.08
		VICL (ACL24)	27.14	32.81	35.02	37.06	36.41	18.72	23.11	23.56	24.84	27.01
		MMICES (WACV25)	26.92	31.22	29.62	33.71	36.65	18.44	21.86	20.01	20.99	23.31
Task vector	MS COCO	MTV (NeurIPS24)	22.56	22.75	24.40	25.00	26.63	9.84	13.72	12.06	12.88	11.91

Appendix

Table 11: Quantitative comparison in multimodal ICL captioning benchmarks. For each dataset and shot, the top scores are highlighted in bold, and the second-best scores are underlined. We use LLaVA-OneVision-1.5-8B as the MLLM and apply random ordering of demonstrations for all baselines except retrieval- and ordering-based methods.

Category	ICL Training Dataset	Method	MS COCO					PICL
Category	ICL Training Dataset	Method	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	1-shot	2-shot	3-shot	4-shot
Off-the-shelf	-	Random	27.57	31.78	33.47	34.11	33.99	15.68	14.58	17.37	16.82	19.95
Retrieval and ordering	-	RICES (NeurIPS22)	27.57	31.78	33.29	33.05	34.06	15.68	14.58	18.27	22.19	25.02
		VICL (ACL24)	26.01	33.02	34.56	34.73	35.21	16.41	15.92	15.36	21.88	21.47
		MMICES (WACV25)	27.57	31.78	33.48	33.83	33.74	15.68	14.58	14.85	20.54	20.56
Task vector	MS COCO	MTV (NeurIPS24)	28.41	32.67	34.21	34.11	35.72	13.02	13.74	15.26	13.18	16.72

Appendix

Table 12: Quantitative comparison in multimodal ICL captioning benchmarks. For each dataset and shot, the top scores are highlighted in bold, and the second-best scores are underlined. We use InternVL-3.5-8B as the MLLM and apply random ordering of demonstrations for all baselines except retrieval- and ordering-based methods.

Figure 9: Examples of multimodal contrastive datasets. Each sample consists of two image-text pairs, $(I_{1},t_{1})$ and $(I_{2},t_{2})$ , with captions $t_{1}$ and $t_{2}$ describe images $I_{1}$ and $I_{2}$ , respectively. $I_{1}$ and $I_{2}$ are visually contrastive ( e.g. , differing in objects (left) or object attributes (right)).

Category	ICL Training Dataset	Method	MS COCO					PICL
Category	ICL Training Dataset	Method	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	1-shot	2-shot	3-shot	4-shot
Off-the-shelf	-	-	11.02	14.87	16.55	21.11	22.34	22.31	26.94	22.17	30.88	31.42
Retrieval and ordering	-	RICES (NeurIPS22)	13.21	17.44	20.02	21.08	24.91	26.77	24.01	27.83	25.16	29.94
		VICL (ACL24)	12.88	19.13	20.76	22.05	24.77	20.42	29.63	28.91	28.04	32.11
		MMICES (WACV25)	10.74	15.62	18.03	20.14	23.09	21.55	27.48	18.92	25.77	27.61
Task vector	MS COCO	MTV (NeurIPS24)	6.11	10.84	14.73	17.02	20.88	24.41	22.88	18.72	28.11	30.92

Appendix

Table 13: Quantitative comparison in multimodal ICL captioning benchmarks. For each dataset and shot, the top scores are highlighted in bold, and the second-best scores are underlined. We use Qwen-VL-3-4B as the MLLM and apply random ordering of demonstrations for all baselines except retrieval- and ordering-based methods.

Category	ICL Training Dataset	Method	MS COCO					PICL
Category	ICL Training Dataset	Method	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	1-shot	2-shot	3-shot	4-shot
Off-the-shelf	-	Random	26.41	35.02	35.11	32.84	35.91	24.82	17.77	28.66	22.48	29.14
Retrieval and ordering	-	RICES (NeurIPS22)	24.91	32.66	36.58	34.11	31.94	26.04	28.85	25.71	34.72	30.63
		VICL (ACL24)	23.84	36.02	36.44	35.97	34.82	22.51	19.47	27.18	31.11	33.62
		MMICES (WACV25)	28.73	30.42	34.95	36.48	37.28	19.18	15.77	21.42	29.83	34.66
Task vector	MS COCO	MTV (NeurIPS24)	22.74	34.18	29.41	33.02	35.88	21.92	18.74	30.11	23.84	31.27

Appendix

Table 14: Quantitative comparison in multimodal ICL captioning benchmarks. For each dataset and shot, the top scores are highlighted in bold, and the second-best scores are underlined. We use InternVL-3.5-14B as the MLLM and apply random ordering of demonstrations for all baselines except retrieval- and ordering-based methods.

Figure 10: Examples comparing PICL and MS COCO. While PICL requires visual understanding all demonstrations to answer correctly ( e.g. , resolving “Mason” in the first demo is necessary to correctly identify “Isabella” for Pororo in the second, enabling the query prediction), MS COCO ICL task can often be solved using pre-trained priors without understanding demonstrations.

Contrastive Benchmark	TrueMICL					MS COCO
Contrastive Benchmark	0-shot	1-shot	2-shot	4-shot	8-shot	0-shot	1-shot	2-shot	4-shot	8-shot
Winoground	28.74	48.12	52.03	64.91	66.77	36.84	41.72	43.03	42.11	43.88
VisMin	28.74	49.72	54.18	60.05	65.11	40.07	44.82	44.44	45.11	43.96
InpaintCOCO	31.11	49.63	54.62	63.80	68.24	38.21	45.02	45.18	43.77	46.91
EqBen	27.41	52.88	53.03	59.44	63.02	38.72	47.03	41.92	48.21	41.33

Appendix

Table 15: Quantitaive comparison of ProCoRe across different contrastive benchmarks. For each dataset and shot, the top scores are highlighted in bold, and the second-best scores are underlined. For a fair comparison, we randomly sample 1,000 contrastive pairs from each benchmark. We use Qwen-VL-3-8B as the base MLLM for all contrastive benchmarks.

Figure 11: Comparison across ICL-shot settings. Higher-shot settings require reasoning over increasingly complex relationships among demonstrations, rather than simply providing more context. For example, in the 3-shot case, the first demo lets the model infer that the dinosaur is “Henry”, while the second remains ambiguous since “Ava” and “Mason” perform the same action; only after the third demo (where the fox is identified as “Henry”) can the model resolve the ambiguity and infer that the pink character in the second demo is “Ava”. In contrast, lower-shot settings can be easier due to fewer required inferences (e.g., in 1-shot, the pink character can be directly identified as “Amella”).

Category	Method	PICL			TrueMICL
		2-shot	3-shot	4-shot	2-shot	4-shot	8-shot
Off-the-shelf	-	$24.97{\pm}1.55$	$29.18{\pm}1.50$	$29.37{\pm}1.52$	$49.37{\pm}1.29$	$55.58{\pm}2.60$	$62.47{\pm}2.40$
Retrieval & Ordering	RICES	$26.44{\pm}0.00$	$27.14{\pm}0.00$	$28.20{\pm}0.00$	$47.16{\pm}0.00$	$51.83{\pm}0.00$	$61.86{\pm}0.00$
	VICL	$28.11{\pm}1.39$	$26.10{\pm}1.75$	$28.48{\pm}1.89$	$50.06{\pm}1.59$	$55.11{\pm}1.74$	$61.66{\pm}0.53$
	MMICES	$21.70{\pm}0.00$	$23.82{\pm}0.00$	$24.79{\pm}0.00$	$48.40{\pm}0.00$	$58.07{\pm}0.00$	$63.04{\pm}0.00$
Task Vector	MTV	$13.85{\pm}2.10$	$15.09{\pm}1.84$	$13.24{\pm}1.25$	$42.94{\pm}1.78$	$45.39{\pm}0.32$	$55.55{\pm}0.81$

Appendix

Table 16: Comparison on PICL and TrueMICL across 3 seeds . Task Vector and ICL-Data Training baselines use the training and evaluation splits from the same dataset.

\hat{t}_{1}\leftarrow f_{\theta}(I_{1}),\quad\hat{t}_{2}\leftarrow f_{\theta}(I_{2})

(8)

Appendix

Algorithm 1 ProCoRe

Multimodal ICL

Multimodal in-context learning (ICL) studies how models use examples and other context to solve tasks involving images, text, audio, and video.

See all 49 papers

Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learning
Sep 9, 2026Mingbo Yang, Wenqiang Wang, Zhaolu Kang +4Contrastive Learning Multimodal ICL
Sun Yat-Sen University Shenzhen, Guangdong, China · Peking University Beijing, Beijing, China · Sun Yat-Sen University Shenzhen, Guangdong, China Pengcheng Laboratory +1
GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models
Jun 10, 2026Garvita Allabadi, Matteo Sodano, Roberto Estevão +4Multimodal ICL Multimodal Large Language Models
University of Illinois Urbana Champaign · University of Bonn · Microsoft
Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
Apr 15, 2026Yu Wang, Sharon LiMultimodal ICL In-Context Learning
Department of Computer Sciences, University of Wisconsin-Madison

Sep 9, 2026J/K move · Enter open · S save

Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learning

Mingbo Yang, Wenqiang Wang, Zhaolu Kang, Peng Chen+3

Sun Yat-Sen University Shenzhen, Guangdong, China · Peking University Beijing, Beijing, China · Sun Yat-Sen University Shenzhen, Guangdong, China Pengcheng Laboratory+1

Contrastive Learning Multimodal ICL Multimodal Reasoning Retrieval-Augmented ICL

Open paper AskPDF arXiv

In-context learning (ICL) is widely used in multimodal large language models (MLLMs) and achieves strong performance across a wide range of multimodal tasks. However, existing multimodal ICL methods often rely on surface level imitation of in-context demonstrations, making it difficult for MLLMs to align their responses with the reasoning path required by the given multimodal input. This limitation becomes more pronounced in complex multimodal tasks, thereby restricting further improvements in MLLM performance. To address this issue, we propose a new multimodal ICL framework that combines contrastive demonstration modeling with the self-refinement capability of MLLMs. Specifically, our framework reformulates each demonstration by explicitly contrasting a suboptimal response with a better response under the same input, together with a reasoning path that reveals how the response should be refined. This contrastive formulation makes the reasoning path toward the desired response more explicit and guides the MLLM beyond superficial imitation. Furthermore, because effective refinement depends on the current response, we introduce a response-conditioned retrieval mechanism to select demonstrations whose reasoning paths are more relevant to the current response. In addition, we use a lightweight alignment controller to predict response quality and determine whether further refinement is needed. Experiments on three types of multimodal tasks show that the proposed framework consistently improves MLLM performance, with particularly notable gains on visual question answering (VQA).

Abstract

Multimodal ICL

Explore similar work

Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learning

GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks