Organizations: Aselsan Research · Department of Computer Science, University of Copenhagen · Department of Medical Informatics, Amsterdam UMC, University of Amsterdam · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands · Koç University, Department of Computer Engineering · Koç University, KUIS AI Center · Hacettepe University, Department of Computer Engineering
As multimodal large language models (MLLMs) advance in handling interleaved image-text data, assessing their few-shot learning capabilities remains an open challenge. In this paper, we introduce FewMMBench, a comprehensive benchmark designed to evaluate MLLMs under few-shot conditions, with a focus on In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. Covering a diverse suite of multimodal understanding tasks, from attribute recognition to temporal reasoning, FewMMBench enables systematic analysis across task types, model families, and prompting strategies. We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings. Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning. Retrieval-based demonstrations and increased context size also yield limited gains. These results highlight FewMMBench as a rigorous testbed for diagnosing and advancing few-shot capabilities in multimodal LLMs. The data is available at: https://huggingface.co/datasets/mustafaa/FewMMBench
Figures & tables
Figure 1: Performance of selected MLLMs on FewMMBench across different evaluation settings. We compare instruction-tuned and non-instruction-tuned models under zero-shot, few-shot (random and similarity-based), and CoT-augmented few-shot configurations. Results show that few-shot prompting does not consistently improve performance for instruction-tuned models, even when demonstrations are semantically similar to the query or when the number of examples increases. Notably, CoT prompting often leads to a performance drop, suggesting modality-specific limitations in current CoT strategies.
Figure 2: Dataset Curation Pipeline for FewMMBench . (a) Task instances are collected and organized based on linguistically meaningful phenomena. (b) We extract visual and textual features for each instance and construct query sets using a Graph Cut-based submodular selection strategy, ensuring both diversity and representativeness. (c) CoT rationales are generated using the Qwen2.5-VL-7B-Instruct model. If the initial prediction is incorrect, the correct answer is injected and a new rationale is generated. An automated filter retains only high-quality examples.
Figure 3: Task distributions and representative examples in FewMMBench . The pie chart illustrates the distribution of samples across the nine tasks in the benchmark. Each surrounding example depicts a sample question corresponding to a specific task.These examples highlight the diversity of visual-linguistic reasoning skills covered by FewMMBench , spanning low-level perception, numerical reasoning, and high-level cognitive inference.
Figure 4: Accuracy performance of top-performing MLLMs from each model family on FewMMBench , evaluated with 0-4-8 shots across three settings: Random, Similar, and Similar with CoT settings.
Figure 5: Pairwise accuracy performance of the top-performing model from three model families on FewMMBench , evaluated with 0, 4, and 8 shots across two settings: Random and Similar.
Figure 6: Uncertainty quantification on FewMMBench for Qwen model family, showing probability distributions over correct (blue) and incorrect (red) answers under 0, 4, and 8-shot prompting with Random and Similar settings.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Distribution of semantic similarity scores between query and demonstration examples across tasks in the FewMMBench
Figure 8: Distribution of reasoning quality scores assigned by human annotators to Chain-of-Thought demonstrations.
Abbreviation
Full Model Name
QW2.5VL-3B-I
Qwen2.5-VL-3B-Instruct
QW2.5VL-7B-I
Qwen2.5-VL-7B-Instruct
QW2VL-2B
Qwen2-VL-2B
QW2VL-2B-I
Qwen2-VL-2B-Instruct
QW2VL-7B
Qwen2-VL-7B
QW2VL-7B-I
Qwen2-VL-7B-Instruct
Appendix
Table 3: Abbreviations of the evaluated MLLMs.
Zero-Shot Setting
Model
Attribute
Object
Plurality
Counting
Relations
Action
Commonsense
Temporal
Coreference
Avg.
QW2.5VL-3B-I
80.4
85.0
84.8
65.8
67.0
78.8
66.8
62.0
86.8
75.3
QW2.5VL-7B-I
85.8
87.8
86.8
74.0
75.8
90.0
75.2
63.2
92.8
81.3
QW2VL-2B
11.6
31.8
12.0
26.0
18.6
4.0
21.2
22.8
17.6
18.4
QW2VL-2B-I
72.4
74.8
72.0
62.8
57.8
78.4
58.0
54.8
58.8
65.5
QW2VL-7B
50.0
62.6
15.6
59.2
39.4
10.4
31.6
47.6
9.2
36.2
Appendix
Table 4: Accuracy performance of the evaluated MLLMs on FewMMBench , varying in model size and pretraining strategies, evaluated with 0-4-8 shots across three settings: Random (R), Similar (S), and Similar with CoT (S+C) settings. In the R setting, few-shot examples are randomly selected. In the S setting, few-shot examples are selected based on visual and textual similarity. In the S+C setting, examples are also selected based on visual and textual similarity but additionally include a CoT description. Models with the suffix ’I’ indicate instruction-tuned versions.
Zero-Shot Setting
Model
Attribute
Object
Plurality
Counting
Relations
Action
Commonsense
Temporal
Coreference
Avg.
QW2.5VL-3B-I
83.0
88.2
80.4
67.2
68.4
89.6
68.0
61.2
83.6
76.6
QW2.5VL-7B-I
83.8
89.0
86.4
74.2
77.8
92.8
77.6
58.8
90.8
81.2
QW2VL-2B
41.4
72.4
68.8
51.8
48.4
78.4
44.0
32.4
52.0
54.4
QW2VL-2B-I
60.0
77.0
76.4
60.2
56.0
83.2
54.4
42.8
63.6
63.7
QW2VL-7B
73.8
85.0
80.0
68.2
63.6
92.0
69.2
52.0
84.0
74.2
Appendix
Table 5: Pairwise accuracy results on FewMMBench . For each question, models are presented with multiple answer options. We compute the perplexity score for each option and select the one with the lowest perplexity as the model’s prediction. Accuracy is then calculated based on the correctness of these selections.
Figure 9: Uncertainty quantification for other Qwen models, showing the probability distributions assigned to correct (blue) and incorrect (red) predictions across different prompting strategies.
Figure 10: Distribution of correctly answered samples across models in FewMMBench . Each bar represents the number of benchmark samples that were correctly answered by a specific number of models, grouped into four buckets: [0–6], [7–13], [14–20], and [21+] out of 26 total models. Results are shown across seven different evaluation settings, including zero-shot and various few-shot configurations with random or semantically similar demonstrations, with and without chain-of-thought (CoT) prompting. The distribution illustrates the challenging nature of the benchmark, with a substantial portion of samples answered correctly by only a few models.
Figure 11: Visualization of PCA, t-SNE, and UMAP projections for the Attribute Recognition task. The top row presents representations from the Multiple-Choice QA setting, while the bottom row corresponds to the Caption/Foil Classification setting.
Figure 12: Visualization of PCA, t-SNE, and UMAP projections for the Visual Object Recognition task. The top row presents representations from the Multiple-Choice QA setting, while the bottom row corresponds to the Caption/Foil classification setting.
Figure 13: Visualization of PCA, t-SNE, and UMAP projections for the Plurality Recognition task.
Figure 14: Visualization of PCA, t-SNE, and UMAP projections for the Object Counting task. The top row presents representations from the Multiple-Choice QA setting, while the bottom row corresponds to the Caption/Foil Classification setting.
Figure 15: Visualization of PCA, t-SNE, and UMAP projections for the Spatial Relations Understanding task. The top row presents representations from the Multiple-Choice QA setting, while the bottom row corresponds to the Caption/Foil classification setting.
Figure 16: Visualization of PCA, t-SNE, and UMAP projections for the Action Recognition task.
Figure 17: Visualization of PCA, t-SNE, and UMAP projections for the Commonsense Reasoning task.
Figure 18: Visualization of PCA, t-SNE, and UMAP projections for the Temporal Reasoning task.
Figure 19: Visualization of PCA, t-SNE, and UMAP projections for the Coreference Resolution task.
Figure 20: Failure cases of CoT usage in MLLMs. Shown are common CoT failure types: (i) hallucinated reasoning inconsistent with the image, (ii) missing CoT despite prompting, (iii) irrelevant or malformed outputs, (iv) infinite reasoning loops, and (v) correct reasoning but incorrect final answers, possibly due to interference from prior examples in few-shot settings. These issues reveal the fragility of current CoT methods in complex multimodal settings.
Figure 21: Example model predictions on instances from the Attribute Recognition task, where demonstrations are selected based on visual and textual similarity.
Figure 22: Example model predictions on instances from the Visual Object Recognition task, where demonstrations are selected based on visual and textual similarity.
Figure 23: Example model predictions on instances from the Plurality Recognition task, where demonstrations are selected based on visual and textual similarity.
Figure 24: Example model predictions on instances from the Object Counting task, where demonstrations are selected based on visual and textual similarity.
Figure 25: Example model predictions on instances from the Spatial Relations Understanding task, where demonstrations are selected based on visual and textual similarity.
Figure 26: Example model predictions on instances from the Action Recognition task, where demonstrations are selected based on visual and textual similarity.
Figure 27: Example model predictions on instances from the Commonsense Reasoning task, where demonstrations are selected based on visual and textual similarity.
Figure 28: Example model predictions on instances from the Temporal Reasoning task, where demonstrations are selected based on visual and textual similarity.
Figure 29: Example model predictions on instances from the Coreference Resolution task, where demonstrations are selected based on visual and textual similarity.
Figure 30: Example model predictions on instances from the Attribute Recognition task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 31: Example model predictions on instances from the Visual Object Recognition task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 32: Example model predictions on instances from the Plurality Recognition task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 33: Example model predictions on instances from the Object Counting task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 34: Example model predictions on instances from the Spatial Relations Understanding task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 35: Example model predictions on instances from the Action Recognition task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 36: Example model predictions on instances from the Commonsense Reasoning task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 37: Example model predictions on instances from the Temporal Reasoning task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 38: Example model predictions on instances from the Coreference Resolution task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Multimodal Large Language Models (MLLMs) have demonstrated remarkable abilities when analyzing images, yet translating these capabilities to few-shot image classification remains challenging. To bridge this gap, we present DeCoDe, a simple yet effective technique that enables off-the-shelf MLLMs to act as strong few-shot classifiers without any additional training. Our approach builds on the idea of few-shot classification as a set of pairwise image comparisons, decomposing the task into a set of binary decisions. Given a query image and a support image from a candidate class, the MLLM is prompted to decide whether the two images depict the same class. The logit corresponding to an affirmative response is then used as a similarity score to assign the query image to the most likely class. While this already yields good results, we show that providing additional high-level information, such as the data domain, to the model further improves performance. Our evaluation provides an extensive analysis of various inference variants on a suite of twelve datasets, six established and six newly curated few-shot benchmarks spanning across diverse domains. The results show that the proposed simple decomposition technique can turn off-the-shelf MLLMs into powerful few-shot learners, significantly outperforming current state-of-the-art few-shot methods on both standard and novel domains. Code is available at https://github.com/yunhanwang1105/DeCoDe.
Yunhan Wang, Eshika Khandelwal, Edson Araujo +3
Tuebingen AI Center, University of Tuebingen, Germany
In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available here.
Yu Wang, Sharon Li
Department of Computer Sciences, University of Wisconsin-Madison
Few-shot in-context learning (ICL) with multi-modal large language models (MLLMs) enables task adaptation without parameter updates, but its performance is highly sensitive to the quality and coverage of the selected demonstrations. While unlabeled multi-modal data is abundant, it remains elusive how to exploit them for ICL. We propose MAG (MAnifold-Guided semi-supervised in-context demonstra- tion selection), an efficient framework that leverages unlabeled data to improve multi-modal ICL. MAG formulates demonstration selection as a semi-supervised propagation problem on a multi-modal graph and adopts a two-stage strategy: (i) relevance score propagation identifies a compact set of high-impact unlabeled samples for pseudo-labeling, reducing MLLM inference cost; (ii) multi-modal relevance is used to select the final demonstrations. We show that textual represen- tations are more effective for relevance propagation, while both visual and textual modalities are crucial for high-quality demonstration selection. Experiments on eight multi-modal benchmarks demonstrate that MAG consistently outperforms strong baselines in label-scarce regimes, achieving significant gains with a limited pseudo-labeling budget.
Zirui Cheng, Xun Xu, Tiankai Chen +7
College of Design and Engineering, National University of Singapore · Institute for Infocomm Research, ASTAR, Singapore