cs.AIOct 30, 2026
SaveMultimodal In-Context Learning with Explicit Rule Induction
Organizations: Tongji University · Microsoft Research Asia
Abstract
A central challenge in multimodal in-context learning (MICL) arises from tasks that are ill-defined without context and require uncovering the underlying rules from demonstrations. Existing methods predominantly treat MICL as direct input-output mapping, overlooking the importance of explicitly inducing and reasoning over such rules, limiting both performance and interpretability. We propose ERIM, Explicit Rule-Induction for Multimodal In-Context Learning, a unified framework that enhances MICL by explicitly inducing the latent rules governing demonstration samples. Leveraging the reasoning ability of multi-modal large language model (MLLM), ERIM constructs rule-centric MICL datasets, introduces explicit supervision for rule induction, and fine-tunes MLLM to perform explicit rule-induction via a two-stage training strategy that combines mixed cold-start supervised fine-tuning with reinforcement learning. Extensive experiments show consistent performance improvements across diverse MICL settings and demonstrate that correctly inducing underlying rules directly leads to stronger and more stable performance.
Figures & tables
Figure 1: Comparison of MICL paradigms. (a) Existing methods treat MICL as a direct input-output mapping. (b) Our approach explicitly induces a rule from the support set and applies it to the query.
Figure 2: Overview of ERIM for Case II multimodal in-context learning. ERIM organizes data into rule-centric MICL episodes and generates explicit rule-induction supervision by transforming multimodal episodes into text-based proxies with rule-preserving captions and faithfully filtered reasoning chains. The model is trained with a two-stage strategy that combines mixed supervised fine-tuning and reinforcement learning, enabling robust induction and application of underlying rules from context.
| Method | Mode | CLEVR | Operator Induction | ||||||||||
| 2 | 3 | 4 | 5 | 6 | Avg | 2 | 3 | 4 | 5 | 6 | Avg | ||
| OpenFlamingo-9B | answer | 24.00 | 24.67 | 24.17 | 26.67 | 24.83 | 24.87 | 8.33 | 7.78 | 6.11 | 5.00 | 8.33 | 7.11 |
| IDEFICS3-8B-Llama3 | answer | 57.00 | 54.00 | 54.50 | 57.83 | 60.83 | 56.83 | 0.00 | 0.00 | 1.11 | 1.67 | 1.67 | 0.89 |
| Phi-3.5-Vision | answer | 32.17 | 26.83 | 28.33 | 26.00 | 28.33 | 28.33 | 10.56 | 15.56 | 11.11 | 6.11 | 7.78 | 10.22 |
| LLaVA-OneVision-7B | answer | 46.33 | 48.83 | 51.17 | 50.50 | 49.83 | 49.33 | 43.89 | 50.56 | 54.44 | 49.44 | 55.00 | 50.67 |
| OpenVLThinker-3B | reasoning | 56.33 | 57.17 | 53.00 | 55.83 | 53.33 | 55.13 | 73.33 | 82.22 | 80.00 | 79.44 | 82.22 | 79.44 |
Table 1: Main results on Case II MICL benchmarks. We report accuracy (%) of different methods across benchmark subsets under varying n-shot settings, along with the average performance over n-shots. For subsets with benchmark-specific additional information, results are averaged over all such settings. Due to space constraints, only 5 out of the 8 selected subsets are shown; the final block reports subset-level averages to assess overall performance on Case II MICL benchmarks.
Figure 3: Statistical overview of the two core datasets used in our method.
| Method | Strategy | CLEVR | Interleaved Operator Induction | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SFT | RL | 2 | 3 | 4 | 5 | 6 | Avg | 2 | 3 | 4 | 5 | 6 | Avg | |
| Qwen2.5VL | ✗ | ✗ | 51.00 | 52.67 | 53.33 | 54.67 | 55.17 | 53.37 | 54.44 | 60.56 | 62.22 | 58.89 | 66.11 | 60.44 |
| +Direct-SFT | Direct | ✗ | 60.17 | 61.17 | 61.17 | 62.83 | 59.83 | 61.03 | 67.22 | 73.33 | 75.56 | 76.11 | 78.89 | 74.22 |
| +Think-RL | ✗ | ✓ | 49.00 | 51.17 | 51.33 | 50.67 | 51.00 | 50.63 | 68.89 | 73.89 | 76.67 | 81.11 | 84.44 | 77.00 |
| +ERIM | Mixed | ✓ | 64.00 | 64.33 | 65.33 | 66.83 | 63.83 | 64.87 | 84.44 | 82.78 | 87.22 | 79.44 | 83.89 | 83.56 |
| Method | Strategy | Clock | Palindrome | |||||||||||
Table 2: Ablation results of training strategies on Qwen2.5-VL-3B-Instruct. We report accuracy (%) with the same metrics and settings as Table 1 . "+Direct-SFT" denotes supervised fine-tuning on the Rule-Centric MICL Dataset using the direct-answer prompt; "+Think-RL" denotes DAPO training with the think-then-answer prompt on the same dataset; "+ERIM" corresponds to the full proposed training strategy with mixed SFT and DAPO. Due to space constraints, only 5 of the 8 selected subsets are shown. Full results are provided in the Appendix C.4 .
| benchmark | zero-shot | few-shot | ERI prompt | ERIM-3B |
|---|---|---|---|---|
| VL-ICL | 20.34 | 31.65 | 40.30 | 78.52 |
| TrueMICL | 26.60 | 30.93 | 40.33 | 61.87 |
Table 3: Effect of prompting strategies. Zero-shot uses the basic thinking template, few-shot additionally provides rule-induction chains as demonstrations, and ERI prompt explicitly instructs the model to induce and apply the underlying rule. We report average accuracy (%) on VL-ICL and TrueMICL, averaged over all other configurations.
Figure 4: Qualitative case study. The blue region indicates the support set, and the green region indicates the query sample. We report the outputs of Qwen2.5-VL-7B-Instruct, VLAA-Thinker-3B, OpenVLThinker-3B, and our method for comparison. Incorrect reasoning steps are highlighted in red, while correct ones are highlighted in green.
Figure 5: Comparison of answer correctness and rule consistency across four models. Rule consistency is evaluated using Qwen2.5-72B-Instruct as the LLM judge. We report average accuracy (%) on VL-ICL and TrueMICL, averaged over all other configurations.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| temperature | 0.0 |
| min p | 0.0 |
| top p | 1.0 |
| top k | -1 |
| repetition penalty | 1.0 |
| frequency penalty | 0.0 |
Table 4: Inference hyperparameters.
| Hyperparameter | Value |
|---|---|
| batch size | 128 |
| learning rate | 1e-5 |
| train epochs | 6 |
| lr scheduler type | cosine |
| warmup ratio | 0.1 |
| training precision | bfloat16 |
Table 5: Supervised fine-tuning hyperparameters.
| Hyperparameter | Value |
|---|---|
| max prompt length | 3072 |
| max response length | 2048 |
| rollout batch size | 64 |
| global batch size | 16 |
| min pixels | 3136 |
| max pixels | 313600 |
Table 6: DAPO hyperparameters.
| Method | Mode | Open MI | CLEVR | ||||||||||
| 0 | 1 | 2 | 4 | 5 | Avg | 2 | 3 | 4 | 5 | 6 | Avg | ||
| OpenFlamingo-9B | answer | 0.00 | 5.17 | 49.67 | 80.17 | 80.50 | 43.10 | 24.00 | 24.67 | 24.17 | 26.67 | 24.83 | 24.87 |
| IDEFICS3-8B-Llama3 | answer | 0.00 | 0.00 | 6.83 | 33.00 | 39.33 | 15.83 | 57.00 | 54.00 | 54.50 | 57.83 | 60.83 | 56.83 |
| Phi-3.5-Vision | answer | 0.00 | 2.50 | 20.33 | 29.33 | 29.83 | 16.40 | 32.17 | 26.83 | 28.33 | 26.00 | 28.33 | 28.33 |
| LLaVA-OneVision-7B | answer | 0.00 | 23.83 | 78.00 | 85.50 | 87.33 | 54.93 | 46.33 | 48.83 | 51.17 | 50.50 | 49.83 | 49.33 |
| OpenVLThinker-3B | reasoning | 0.00 | 4.17 | 27.17 | 30.00 | 30.67 | 18.40 | 56.33 | 57.17 | 53.00 | 55.83 | 53.33 | 55.13 |
Table 7: Complete main experimental results comparing ERIM with baseline methods across all evaluation benchmarks.
| Method | Strategy | Open MI | CLEVR | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SFT | RL | 0 | 1 | 2 | 4 | 5 | Avg | 2 | 3 | 4 | 5 | 6 | Avg | |
| Qwen2.5VL | ✗ | ✗ | 0.00 | 12.17 | 81.50 | 90.33 | 89.50 | 54.70 | 51.00 | 52.67 | 53.33 | 54.67 | 55.17 | 53.37 |
| +Direct-SFT | Direct | ✗ | 0.00 | 98.50 | 99.67 | 100.00 | 99.67 | 79.57 | 60.17 | 61.17 | 61.17 | 62.83 | 59.83 | 61.03 |
| +Think-RL | ✗ | ✓ | 0.00 | 91.83 | 98.83 | 99.33 | 99.33 | 77.87 | 49.00 | 51.17 | 51.33 | 50.67 | 51.00 | 50.63 |
| +ERIM | Mixed | ✓ | 0.00 | 83.67 | 96.67 | 97.67 | 96.17 | 74.83 | 64.00 | 64.33 | 65.33 | 66.83 | 63.83 | 64.87 |
| Method | Strategy | Operator Induction | Interleaved Operator Induction | |||||||||||
Table 8: Complete ablation results analyzing the impact of different training strategies.
| Method | Mode | ShapeWorld | |||||
|---|---|---|---|---|---|---|---|
| 2 | 3 | 4 | 5 | 6 | Avg | ||
| OpenFlamingo-9B | answer | 36.67 | 23.33 | 21.11 | 35.56 | 30.00 | 29.33 |
| Phi-3.5-Vision | answer | 41.11 | 25.56 | 26.67 | 28.89 | 30.00 | 30.45 |
| OpenVLThinker-3B | reasoning | 37.78 | 26.67 | 44.44 | 37.78 | 40.00 | 37.33 |
| VLAA-Thinker-3B | reasoning | 40.00 | 33.33 | 46.67 | 42.22 | 36.67 | 39.78 |
| Qwen2.5-VL-3B | answer | 30.00 | 33.33 | 35.56 | 33.33 | 31.11 | 32.67 |
Table 9: Results on out-of-domain evaluation benchmark. We report accuracy (%) of different methods under varying n-shot settings, along with the average performance over n-shots.
More in this topic
Multimodal ICL
Multimodal in-context learning (ICL) studies how models use examples and other context to solve tasks involving images, text, audio, and video.
See all 51 papers