A central challenge in multimodal in-context learning (MICL) arises from tasks that are ill-defined without context and require uncovering the underlying rules from demonstrations. Existing methods predominantly treat MICL as direct input-output mapping, overlooking the importance of explicitly inducing and reasoning over such rules, limiting both performance and interpretability. We propose ERIM, Explicit Rule-Induction for Multimodal In-Context Learning, a unified framework that enhances MICL by explicitly inducing the latent rules governing demonstration samples. Leveraging the reasoning ability of multi-modal large language model (MLLM), ERIM constructs rule-centric MICL datasets, introduces explicit supervision for rule induction, and fine-tunes MLLM to perform explicit rule-induction via a two-stage training strategy that combines mixed cold-start supervised fine-tuning with reinforcement learning. Extensive experiments show consistent performance improvements across diverse MICL settings and demonstrate that correctly inducing underlying rules directly leads to stronger and more stable performance.
Figures & tables
Figure 1: Comparison of MICL paradigms. (a) Existing methods treat MICL as a direct input-output mapping. (b) Our approach explicitly induces a rule from the support set and applies it to the query.
Figure 2: Overview of ERIM for Case II multimodal in-context learning. ERIM organizes data into rule-centric MICL episodes and generates explicit rule-induction supervision by transforming multimodal episodes into text-based proxies with rule-preserving captions and faithfully filtered reasoning chains. The model is trained with a two-stage strategy that combines mixed supervised fine-tuning and reinforcement learning, enabling robust induction and application of underlying rules from context.
Method
Mode
CLEVR
Operator Induction
2
3
4
5
6
Avg
2
3
4
5
6
Avg
OpenFlamingo-9B
answer
24.00
24.67
24.17
26.67
24.83
24.87
8.33
7.78
6.11
5.00
8.33
7.11
IDEFICS3-8B-Llama3
answer
57.00
54.00
54.50
57.83
60.83
56.83
0.00
0.00
1.11
1.67
1.67
0.89
Phi-3.5-Vision
answer
32.17
26.83
28.33
26.00
28.33
28.33
10.56
15.56
11.11
6.11
7.78
10.22
LLaVA-OneVision-7B
answer
46.33
48.83
51.17
50.50
49.83
49.33
43.89
50.56
54.44
49.44
55.00
50.67
OpenVLThinker-3B
reasoning
56.33
57.17
53.00
55.83
53.33
55.13
73.33
82.22
80.00
79.44
82.22
79.44
Table 1: Main results on Case II MICL benchmarks. We report accuracy (%) of different methods across benchmark subsets under varying n-shot settings, along with the average performance over n-shots. For subsets with benchmark-specific additional information, results are averaged over all such settings. Due to space constraints, only 5 out of the 8 selected subsets are shown; the final block reports subset-level averages to assess overall performance on Case II MICL benchmarks.
Figure 3: Statistical overview of the two core datasets used in our method.
Method
Strategy
CLEVR
Interleaved Operator Induction
SFT
RL
2
3
4
5
6
Avg
2
3
4
5
6
Avg
Qwen2.5VL
✗
✗
51.00
52.67
53.33
54.67
55.17
53.37
54.44
60.56
62.22
58.89
66.11
60.44
+Direct-SFT
Direct
✗
60.17
61.17
61.17
62.83
59.83
61.03
67.22
73.33
75.56
76.11
78.89
74.22
+Think-RL
✗
✓
49.00
51.17
51.33
50.67
51.00
50.63
68.89
73.89
76.67
81.11
84.44
77.00
+ERIM
Mixed
✓
64.00
64.33
65.33
66.83
63.83
64.87
84.44
82.78
87.22
79.44
83.89
83.56
Method
Strategy
Clock
Palindrome
Table 2: Ablation results of training strategies on Qwen2.5-VL-3B-Instruct. We report accuracy (%) with the same metrics and settings as Table 1 . "+Direct-SFT" denotes supervised fine-tuning on the Rule-Centric MICL Dataset using the direct-answer prompt; "+Think-RL" denotes DAPO training with the think-then-answer prompt on the same dataset; "+ERIM" corresponds to the full proposed training strategy with mixed SFT and DAPO. Due to space constraints, only 5 of the 8 selected subsets are shown. Full results are provided in the Appendix C.4 .
benchmark
zero-shot
few-shot
ERI prompt
ERIM-3B
VL-ICL
20.34
31.65
40.30
78.52
TrueMICL
26.60
30.93
40.33
61.87
Table 3: Effect of prompting strategies. Zero-shot uses the basic thinking template, few-shot additionally provides rule-induction chains as demonstrations, and ERI prompt explicitly instructs the model to induce and apply the underlying rule. We report average accuracy (%) on VL-ICL and TrueMICL, averaged over all other configurations.
Figure 4: Qualitative case study. The blue region indicates the support set, and the green region indicates the query sample. We report the outputs of Qwen2.5-VL-7B-Instruct, VLAA-Thinker-3B, OpenVLThinker-3B, and our method for comparison. Incorrect reasoning steps are highlighted in red, while correct ones are highlighted in green.
Figure 5: Comparison of answer correctness and rule consistency across four models. Rule consistency is evaluated using Qwen2.5-72B-Instruct as the LLM judge. We report average accuracy (%) on VL-ICL and TrueMICL, averaged over all other configurations.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
temperature
0.0
min p
0.0
top p
1.0
top k
-1
repetition penalty
1.0
frequency penalty
0.0
Appendix
Table 4: Inference hyperparameters.
Hyperparameter
Value
batch size
128
learning rate
1e-5
train epochs
6
lr scheduler type
cosine
warmup ratio
0.1
training precision
bfloat16
Appendix
Table 5: Supervised fine-tuning hyperparameters.
Hyperparameter
Value
max prompt length
3072
max response length
2048
rollout batch size
64
global batch size
16
min pixels
3136
max pixels
313600
Appendix
Table 6: DAPO hyperparameters.
Method
Mode
Open MI
CLEVR
0
1
2
4
5
Avg
2
3
4
5
6
Avg
OpenFlamingo-9B
answer
0.00
5.17
49.67
80.17
80.50
43.10
24.00
24.67
24.17
26.67
24.83
24.87
IDEFICS3-8B-Llama3
answer
0.00
0.00
6.83
33.00
39.33
15.83
57.00
54.00
54.50
57.83
60.83
56.83
Phi-3.5-Vision
answer
0.00
2.50
20.33
29.33
29.83
16.40
32.17
26.83
28.33
26.00
28.33
28.33
LLaVA-OneVision-7B
answer
0.00
23.83
78.00
85.50
87.33
54.93
46.33
48.83
51.17
50.50
49.83
49.33
OpenVLThinker-3B
reasoning
0.00
4.17
27.17
30.00
30.67
18.40
56.33
57.17
53.00
55.83
53.33
55.13
Appendix
Table 7: Complete main experimental results comparing ERIM with baseline methods across all evaluation benchmarks.
Method
Strategy
Open MI
CLEVR
SFT
RL
0
1
2
4
5
Avg
2
3
4
5
6
Avg
Qwen2.5VL
✗
✗
0.00
12.17
81.50
90.33
89.50
54.70
51.00
52.67
53.33
54.67
55.17
53.37
+Direct-SFT
Direct
✗
0.00
98.50
99.67
100.00
99.67
79.57
60.17
61.17
61.17
62.83
59.83
61.03
+Think-RL
✗
✓
0.00
91.83
98.83
99.33
99.33
77.87
49.00
51.17
51.33
50.67
51.00
50.63
+ERIM
Mixed
✓
0.00
83.67
96.67
97.67
96.17
74.83
64.00
64.33
65.33
66.83
63.83
64.87
Method
Strategy
Operator Induction
Interleaved Operator Induction
Appendix
Table 8: Complete ablation results analyzing the impact of different training strategies.
Method
Mode
ShapeWorld
2
3
4
5
6
Avg
OpenFlamingo-9B
answer
36.67
23.33
21.11
35.56
30.00
29.33
Phi-3.5-Vision
answer
41.11
25.56
26.67
28.89
30.00
30.45
OpenVLThinker-3B
reasoning
37.78
26.67
44.44
37.78
40.00
37.33
VLAA-Thinker-3B
reasoning
40.00
33.33
46.67
42.22
36.67
39.78
Qwen2.5-VL-3B
answer
30.00
33.33
35.56
33.33
31.11
32.67
Appendix
Table 9: Results on out-of-domain evaluation benchmark. We report accuracy (%) of different methods under varying n-shot settings, along with the average performance over n-shots.
We introduce MMCL-Bench, a benchmark for multimodal context learning: learning task-local rules, procedures, and empirical patterns from visual or mixed-modality teaching context and applying them to new visual instances. Unlike text-only context learning or standard multimodal question answering, this setting requires models to recover and localize relevant evidence from images, screenshots, manuals, videos, and frame sequences before they can reason over the learned context. MMCL-Bench contains 102 tasks spanning three categories: rule system application, procedural task execution, and empirical discovery and induction. We evaluate frontier multimodal models with strict rubric-based scoring and find that current systems remain far from robust multimodal context learning, with even the strongest model solving fewer than one-third of tasks under strict evaluation. Diagnostic ablations and error analysis show that failures arise throughout the context-to-answer pipeline, including context anchoring, visual evidence extraction, context reasoning, and response construction. MMCL-Bench thus highlights multimodal context learning as an important unsolved capability bottleneck for current multimodal models.
Yifan Chen, Fei Yin, Qingyan Bai +2
University of Cambridge · HKUST · Tsinghua University
In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile. Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning, while struggling to extract consistent rules across demonstrations. This gap is further exacerbated by two visual-level obstacles: an overwhelming proportion of redundant visual tokens that obscure textual cues, and a skewed attention distribution that favors the initial image at the expense of subsequent context. To address these issues, we introduce a framework that restructures multimodal ICL as a principled inductive-deductive process. The framework incorporates a similarity-based visual token compression module to filter out redundant patches, a dynamic attention rebalancing mechanism to distribute focus equitably across all images, and a chain-of-thought paradigm that explicitly guides the model to analyze individual examples, derive a generalizable rule, and then apply it to the query. An auxiliary learning pipeline combines supervised fine-tuning with reinforcement learning using verifiable rewards to reinforce faithful citation and noise filtering. Evaluations across eight benchmarks covering visual perception, logical reasoning, STEM problems, and sarcasm detection demonstrate consistent and significant improvements over standard ICL baselines for multiple open-source VLMs, highlighting the potential of equipping models with genuine inductive capabilities in multimodal settings.
Haoyu Wang, Haonan Wang, Yuyan Chen +5
Tencent QQ · Fudan University · Cornell University
In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available here.
Yu Wang, Sharon Li
Department of Computer Sciences, University of Wisconsin-Madison