Multimodal in-context learning (ICL) is an important capability of multimodal large language models (MLLMs). Its performance is closely tied to the demonstrations it uses, making demonstration retrieval a core problem for ICL. Common methods use fixed scoring methods to retrieve all the demonstrations at once, which may introduce redundancy and noise. Context-aware retrieval methods have emerged to address this problem by allowing incremental retrieval from a growing context. However, none of them lets the MLLM learn how to judge what new demonstrations it needs and then carry out progressive retrieval on its own. To this end, we introduce ActICL, which trains an MLLM agent to acquire and use an ICL context via tool-augmented reasoning. An ActICL agent analyzes its own information gaps during iterative reasoning and tool-use actions. Based on these gaps, it performs targeted retrieval to acquire a set of initial candidates. It can then use the existing context to further filter the candidates and refine the context, or it can iteratively perform a new retrieval. When the agent judges that the context is sufficient for the task, or when the resource limit is reached with the user request met, it sends the acquired context to an answering model, either the agent itself or another MLLM. We use a two-stage post-training procedure that jointly optimizes the agent's acquisition and answering. With an adaptive number of demonstrations, ActICL improves average performance by 11.6 percentage points over standard similarity-based retrieval across seven benchmarks and three MLLMs, and by 4.4 points compared to the strongest prior baseline. The contexts acquired by ActICL agents can also be used effectively by frontier models. When GPT-5.6-Sol performs 4-shot ICL, ActICL's contexts improve its performance by 3.5 points over standard retrieval, averaged over three benchmarks.
Figures & tables
Figure 1: Given a multimodal query, we seek to construct a context with demonstrations that helps an MLLM answer the query correctly. Prior work has largely (a) used off-the-shelf similarity-based retrievers, which often return redundant demonstrations, or (b) trained context-aware retrievers, which can fail to generalize at test time. Our approach (c), ActICL, treats ICL context construction as an iterative agentic task, training an agent to formulate complex search queries, process and filter retrieved evidence, and decide when to submit a context to an answering model.
Figure 2: The overall illustration of ActICL. Given a query and acquisition instructions, the agent reasons at every step to choose among four actions: search submits a goal whose enabled channels retrieve previews from the demonstration bank, read reveals full demonstrations, and curate adds or removes read demonstrations in the retained context, until solve commits an admissible context. The agent learns on-policy which contexts make ICL effective and how to acquire them.
Methods
TextVQA
MME-R
CharXiv
GVL
MathVision
MDK12
VL-ICL
Avg.
Qwen3-VL-8B
0.735
0.238
0.473
0.554
0.528
0.462
0.293
0.469
+Rand. Sample
0.728
0.259
0.482
0.527
0.540
0.449
0.562
0.507
+MMICES
0.743
0.278
0.495
0.583
0.543
0.485
0.695
0.546
+ContextNav
0.764
0.295
0.527
0.605
0.558
0.510
0.761
0.574
+LSD
0.779
0.308
0.519
0.629
0.552
0.518
0.784
0.584
+ProMSA
0.819
0.326
0.546
0.684
0.575
0.561
0.873
0.626
Table 1: Comparison of ICL performance with baseline methods. Bold indicates the best accuracy, and underlining indicates the best accuracy excluding ActICL. The first row for each model reports zero-shot performance. All other results use four shots, except ActICL in the adaptive setting.
Method
MME-R
MathVision
MMStar
MuirBench
BLINK
4-shot
8-shot
4-shot
8-shot
4-shot
8-shot
Qwen3-VL-8B
0.278
0.316
0.543
0.557
0.746
0.753
0.653
0.702
+LoRA
0.295
0.324
0.536
0.569
0.762
0.784
0.709
0.735
+DARA
0.306
0.305
0.529
0.568
0.770
0.769
0.684
0.719
+ActICL
0.344
0.375
0.591
0.584
0.783
0.797
0.717
0.744
Qwen3-VL-30B-A3B
0.308
0.332
0.597
0.609
0.751
0.746
0.629
0.683
Table 2: Performance comparison with baseline methods on three multimodal ICL tasks and two multi-image understanding tasks. All few-shot prompts are constructed using MMICES.
Method
PerceptionBench
MindCube
MMMU-Pro
Qwen3.5-397B-A17B
0.458
0.477
0.757
+MMICES
0.472
0.492
0.761
+ContextNav
0.487
0.524
0.772
+ActICL
0.500
0.545
0.780
GPT-5.6-Luna (max)
0.559
0.638
0.785
+MMICES
0.568
0.634
0.802
Table 3: 4-shot performance of different retrievers with larger-scale open-source and frontier closed-source models. ContextNav uses Gemini3.5-Flash as its policy, while ActICL uses Qwen3.5-9B.
Figure 6
Figure 4: Multimodal retrieval ablation on Qwen3.5-9B under the adaptive setting.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Tier
Calls H
Bacq
Bctx
Tight
14
98,304
8,192
Standard
20
196,608
16,384
Wide
28
393,216
32,768
Appendix
Table 5: Resource tiers. H bounds the number of acquisition calls, Bacq bounds the cumulative input and generated tokens of all acquisition calls, and Bctx bounds the retained demonstrations.
Query mixture
Pool
Source
RL (%)
RL
SFT
Bank
ReasonVQA
10
2,000
400
8,000
A-OKVQA
8
1,600
320
5,000
VCR (Q → A)
8
1,600
320
7,500
DocVQA
10
2,000
400
5,500
ChartQA
8
1,600
320
5,000
Appendix
Table 6: Training mixture and demonstration-pool targets. RL counts are instances, and SFT counts are accepted trajectory–answer pairs per backbone. Bank counts are distinct records and are allocated independently of the query mixture.
Core routes (%)
Profile
Source
Similarity
Cross
Prob. (%)
Source-heavy
50
25
25
20
Mixed
25
50
25
50
Similarity-heavy
12.5
62.5
25
30
Appendix
Table 7: Sampling profiles for the 256-record core. Route percentages apply within the core, and the last column gives the probability of selecting each profile.
Fixed count n
Tier
Adaptive
1
2
4
8
Total
Tight
10
2.5
4
2.5
1
20
Standard
25
1.5
7
11
5.5
50
Wide
15
1
4
6.5
3.5
30
Total
50
5
15
20
10
100
Appendix
Table 8: Joint distribution of count requests and resource tiers, as percentages of all training instances. A configuration is admitted only after a feasible acquisition path under its limits has been validated.
Setting
RS-SFT
Joint RL
Training volume and batching
Training volume
4,000 pairs, 2 epochs
20,000 instances, 1 pass
Global batch
16 pairs
32 groups per rollout batch
Optimizer minibatch
16 pairs
8 complete groups
Optimizer updates
500
2,500
Optimizer and numerics
Appendix
Table 9: Training settings per backbone. A joint-RL group contains all K acquisition trajectories of an instance and the answers sampled from its valid contexts.
Parameter
Symbol
Value
Acquisition trajectories per instance
K
4
Answers per valid context
M
2
Clipping width
ϵ
0.2
Answer baseline mixture
μ
0.1
Answer reference baseline
bref
0.5
Fixed loss normalizer
Lref
8,192
Appendix
Table 10: Joint RL hyperparameters. The cost schedule depends on the fraction of instances sampled and is independent of the learning-rate schedule.
Benchmark
Available pool (bank / query)
Stratification
Released splits preserved
TextVQA
train / validation
native boundary
GVL
train / test
native boundary, per graph task
VL-ICL Bench
released image-to-text support / query construction
per subtask
New 6:4 partition
MME-Reasoning
public labeled release
reasoning category and subtype, difficulty where available
Appendix
Table 11: Evaluation pools and stratification. Benchmarks with released splits keep their native bank / query boundaries. The others receive a new partition that targets a 6:4 bank / query ratio within each stratum.
Method
MME-R
CharXiv
MathVision
Avg.
Base
0.343
0.675
0.728
0.582
SFT only
0.431
0.743
0.854
0.676
Joint RL only
0.423
0.748
0.850
0.674
w/o acquisition
0.402
0.740
0.838
0.660
w/o answering
0.415
0.745
0.847
0.669
ActICL
0.427
0.751
0.862
0.680
Appendix
Table 12: Ablation on training stages with Qwen3.5-9B under four-shot requests. “w/o acquisition” and “w/o answering” run joint RL after SFT without the corresponding part.
Variant
MME-R
CharXiv
MathVision
Avg.
w/o read
0.445
0.758
0.852
0.685
w/o curate
0.448
0.771
0.859
0.693
Add-only curate
0.457
0.782
0.862
0.700
ActICL
0.468
0.788
0.870
0.709
Appendix
Table 13: Ablation on read and curate with Qwen3.5-9B under the adaptive setting.
Variant
MME-R
CharXiv
MathVision
Avg.
#Shots
Acq. (k)
w/o resource costs
0.461
0.779
0.868
0.703
2.94
32.7
w/o cost schedule
0.459
0.764
0.857
0.693
2.78
30.2
w/o fixed reference
0.442
0.779
0.857
0.693
2.10
28.9
ActICL
0.468
0.788
0.870
0.709
2.62
26.3
Appendix
Table 14: Reward ablation with Qwen3.5-9B under the adaptive setting. #Shots is the mean committed count and Acq. is the mean number of acquisition tokens, both averaged over the three benchmarks.
Training bank
MME-R
CharXiv
MathVision
Avg.
Own bank
Source-only
0.459
0.767
0.862
0.696
Random
0.448
0.759
0.858
0.688
ActICL
0.468
0.788
0.870
0.709
Mixed bank
Source-only
0.422
0.743
0.844
0.670
Appendix
Table 15: Ablation on training-bank construction with Qwen3.5-9B under the adaptive setting. The top block evaluates each benchmark on its own support bank, and the bottom block on a mixed bank that merges the support banks of the three benchmarks.
In-context learning (ICL) is widely used in multimodal large language models (MLLMs) and achieves strong performance across a wide range of multimodal tasks. However, existing multimodal ICL methods often rely on surface level imitation of in-context demonstrations, making it difficult for MLLMs to align their responses with the reasoning path required by the given multimodal input. This limitation becomes more pronounced in complex multimodal tasks, thereby restricting further improvements in MLLM performance. To address this issue, we propose a new multimodal ICL framework that combines contrastive demonstration modeling with the self-refinement capability of MLLMs. Specifically, our framework reformulates each demonstration by explicitly contrasting a suboptimal response with a better response under the same input, together with a reasoning path that reveals how the response should be refined. This contrastive formulation makes the reasoning path toward the desired response more explicit and guides the MLLM beyond superficial imitation. Furthermore, because effective refinement depends on the current response, we introduce a response-conditioned retrieval mechanism to select demonstrations whose reasoning paths are more relevant to the current response. In addition, we use a lightweight alignment controller to predict response quality and determine whether further refinement is needed. Experiments on three types of multimodal tasks show that the proposed framework consistently improves MLLM performance, with particularly notable gains on visual question answering (VQA).
Mingbo Yang, Wenqiang Wang, Zhaolu Kang +4
Sun Yat-Sen University Shenzhen, Guangdong, China · Peking University Beijing, Beijing, China · Sun Yat-Sen University Shenzhen, Guangdong, China Pengcheng Laboratory +1
Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility, multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates attention distributions to better match demonstration-induced attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at token level across layers and heads based on the current query. Finally, we propose a layer-wise hyperbolic anchor distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration-query relationships induced by ICDs. Extensive experiments across six different multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.
In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities. We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison. Across six open-weight models and 38 tasks, multimodal ICL consistently underperforms text-only ICL, with gaps varying by task family. To test whether this gap can be recovered, we target visual access, task framing, and reasoning through three interventions. Their combination recovers strong multimodal ICL performance on a diagnostic subset, despite limited or inconsistent individual effects. To distinguish difficulties in executing tasks from those in inferring them, we evaluate models with explicit task instructions, revealing a modality gap even when the task is known. We then examine how adding demonstration inputs and outputs reshapes this gap, highlighting demonstrations' dual role as additional context to process and evidence about the task. The dataset is available at https://github.com/lab-flair/TwinICL.
Zihan Xue, Po-Yi Lu, Serhii Honcharenko +5
University of California, Los Angeles · 2National Taiwan University · 3Texas A&M University +2