cs.CLFeb 25, 2026
SaveFewMMBench: A Benchmark for Multimodal Few-Shot Learning
Organizations: Aselsan Research · Department of Computer Science, University of Copenhagen · Department of Medical Informatics, Amsterdam UMC, University of Amsterdam · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands ·
Abstract
As multimodal large language models (MLLMs) advance in handling interleaved image-text data, assessing their few-shot learning capabilities remains an open challenge. In this paper, we introduce FewMMBench, a comprehensive benchmark designed to evaluate MLLMs under few-shot conditions, with a focus on In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. Covering a diverse suite of multimodal understanding tasks, from attribute recognition to temporal reasoning, FewMMBench enables systematic analysis across task types, model families, and prompting strategies. We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings. Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning. Retrieval-based demonstrations and increased context size also yield limited gains. These results highlight FewMMBench as a rigorous testbed for diagnosing and advancing few-shot capabilities in multimodal LLMs. The data is available at: https://huggingface.co/datasets/mustafaa/FewMMBench
Figures & tables
Figure 1: Performance of selected MLLMs on FewMMBench across different evaluation settings. We compare instruction-tuned and non-instruction-tuned models under zero-shot, few-shot (random and similarity-based), and CoT-augmented few-shot configurations. Results show that few-shot prompting does not consistently improve performance for instruction-tuned models, even when demonstrations are semantically similar to the query or when the number of examples increases. Notably, CoT prompting often leads to a performance drop, suggesting modality-specific limitations in current CoT strategies.
Figure 2: Dataset Curation Pipeline for FewMMBench . (a) Task instances are collected and organized based on linguistically meaningful phenomena. (b) We extract visual and textual features for each instance and construct query sets using a Graph Cut-based submodular selection strategy, ensuring both diversity and representativeness. (c) CoT rationales are generated using the Qwen2.5-VL-7B-Instruct model. If the initial prediction is incorrect, the correct answer is injected and a new rationale is generated. An automated filter retains only high-quality examples.
Figure 3: Task distributions and representative examples in FewMMBench . The pie chart illustrates the distribution of samples across the nine tasks in the benchmark. Each surrounding example depicts a sample question corresponding to a specific task.These examples highlight the diversity of visual-linguistic reasoning skills covered by FewMMBench , spanning low-level perception, numerical reasoning, and high-level cognitive inference.
Figure 4: Accuracy performance of top-performing MLLMs from each model family on FewMMBench , evaluated with 0-4-8 shots across three settings: Random, Similar, and Similar with CoT settings.
Figure 5: Pairwise accuracy performance of the top-performing model from three model families on FewMMBench , evaluated with 0, 4, and 8 shots across two settings: Random and Similar.
Figure 6: Uncertainty quantification on FewMMBench for Qwen model family, showing probability distributions over correct (blue) and incorrect (red) answers under 0, 4, and 8-shot prompting with Random and Similar settings.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Distribution of semantic similarity scores between query and demonstration examples across tasks in the FewMMBench
Figure 8: Distribution of reasoning quality scores assigned by human annotators to Chain-of-Thought demonstrations.
| Abbreviation | Full Model Name |
|---|---|
| QW2.5VL-3B-I | Qwen2.5-VL-3B-Instruct |
| QW2.5VL-7B-I | Qwen2.5-VL-7B-Instruct |
| QW2VL-2B | Qwen2-VL-2B |
| QW2VL-2B-I | Qwen2-VL-2B-Instruct |
| QW2VL-7B | Qwen2-VL-7B |
| QW2VL-7B-I | Qwen2-VL-7B-Instruct |
Table 3: Abbreviations of the evaluated MLLMs.
| Zero-Shot Setting | ||||||||||||||||||||||||||||||
| Model | Attribute | Object | Plurality | Counting | Relations | Action | Commonsense | Temporal | Coreference | Avg. | ||||||||||||||||||||
| QW2.5VL-3B-I | 80.4 | 85.0 | 84.8 | 65.8 | 67.0 | 78.8 | 66.8 | 62.0 | 86.8 | 75.3 | ||||||||||||||||||||
| QW2.5VL-7B-I | 85.8 | 87.8 | 86.8 | 74.0 | 75.8 | 90.0 | 75.2 | 63.2 | 92.8 | 81.3 | ||||||||||||||||||||
| QW2VL-2B | 11.6 | 31.8 | 12.0 | 26.0 | 18.6 | 4.0 | 21.2 | 22.8 | 17.6 | 18.4 | ||||||||||||||||||||
| QW2VL-2B-I | 72.4 | 74.8 | 72.0 | 62.8 | 57.8 | 78.4 | 58.0 | 54.8 | 58.8 | 65.5 | ||||||||||||||||||||
| QW2VL-7B | 50.0 | 62.6 | 15.6 | 59.2 | 39.4 | 10.4 | 31.6 | 47.6 | 9.2 | 36.2 | ||||||||||||||||||||
Table 4: Accuracy performance of the evaluated MLLMs on FewMMBench , varying in model size and pretraining strategies, evaluated with 0-4-8 shots across three settings: Random (R), Similar (S), and Similar with CoT (S+C) settings. In the R setting, few-shot examples are randomly selected. In the S setting, few-shot examples are selected based on visual and textual similarity. In the S+C setting, examples are also selected based on visual and textual similarity but additionally include a CoT description. Models with the suffix ’I’ indicate instruction-tuned versions.
| Zero-Shot Setting | ||||||||||||||||||||
| Model | Attribute | Object | Plurality | Counting | Relations | Action | Commonsense | Temporal | Coreference | Avg. | ||||||||||
| QW2.5VL-3B-I | 83.0 | 88.2 | 80.4 | 67.2 | 68.4 | 89.6 | 68.0 | 61.2 | 83.6 | 76.6 | ||||||||||
| QW2.5VL-7B-I | 83.8 | 89.0 | 86.4 | 74.2 | 77.8 | 92.8 | 77.6 | 58.8 | 90.8 | 81.2 | ||||||||||
| QW2VL-2B | 41.4 | 72.4 | 68.8 | 51.8 | 48.4 | 78.4 | 44.0 | 32.4 | 52.0 | 54.4 | ||||||||||
| QW2VL-2B-I | 60.0 | 77.0 | 76.4 | 60.2 | 56.0 | 83.2 | 54.4 | 42.8 | 63.6 | 63.7 | ||||||||||
| QW2VL-7B | 73.8 | 85.0 | 80.0 | 68.2 | 63.6 | 92.0 | 69.2 | 52.0 | 84.0 | 74.2 | ||||||||||
Table 5: Pairwise accuracy results on FewMMBench . For each question, models are presented with multiple answer options. We compute the perplexity score for each option and select the one with the lowest perplexity as the model’s prediction. Accuracy is then calculated based on the correctness of these selections.
Figure 9: Uncertainty quantification for other Qwen models, showing the probability distributions assigned to correct (blue) and incorrect (red) predictions across different prompting strategies.
Figure 10: Distribution of correctly answered samples across models in FewMMBench . Each bar represents the number of benchmark samples that were correctly answered by a specific number of models, grouped into four buckets: [0–6], [7–13], [14–20], and [21+] out of 26 total models. Results are shown across seven different evaluation settings, including zero-shot and various few-shot configurations with random or semantically similar demonstrations, with and without chain-of-thought (CoT) prompting. The distribution illustrates the challenging nature of the benchmark, with a substantial portion of samples answered correctly by only a few models.
Figure 11: Visualization of PCA, t-SNE, and UMAP projections for the Attribute Recognition task. The top row presents representations from the Multiple-Choice QA setting, while the bottom row corresponds to the Caption/Foil Classification setting.
Figure 12: Visualization of PCA, t-SNE, and UMAP projections for the Visual Object Recognition task. The top row presents representations from the Multiple-Choice QA setting, while the bottom row corresponds to the Caption/Foil classification setting.
Figure 13: Visualization of PCA, t-SNE, and UMAP projections for the Plurality Recognition task.
Figure 14: Visualization of PCA, t-SNE, and UMAP projections for the Object Counting task. The top row presents representations from the Multiple-Choice QA setting, while the bottom row corresponds to the Caption/Foil Classification setting.
Figure 15: Visualization of PCA, t-SNE, and UMAP projections for the Spatial Relations Understanding task. The top row presents representations from the Multiple-Choice QA setting, while the bottom row corresponds to the Caption/Foil classification setting.
Figure 16: Visualization of PCA, t-SNE, and UMAP projections for the Action Recognition task.
Figure 17: Visualization of PCA, t-SNE, and UMAP projections for the Commonsense Reasoning task.
Figure 18: Visualization of PCA, t-SNE, and UMAP projections for the Temporal Reasoning task.
Figure 19: Visualization of PCA, t-SNE, and UMAP projections for the Coreference Resolution task.
Figure 20: Failure cases of CoT usage in MLLMs. Shown are common CoT failure types: (i) hallucinated reasoning inconsistent with the image, (ii) missing CoT despite prompting, (iii) irrelevant or malformed outputs, (iv) infinite reasoning loops, and (v) correct reasoning but incorrect final answers, possibly due to interference from prior examples in few-shot settings. These issues reveal the fragility of current CoT methods in complex multimodal settings.
Figure 21: Example model predictions on instances from the Attribute Recognition task, where demonstrations are selected based on visual and textual similarity.
Figure 22: Example model predictions on instances from the Visual Object Recognition task, where demonstrations are selected based on visual and textual similarity.
Figure 23: Example model predictions on instances from the Plurality Recognition task, where demonstrations are selected based on visual and textual similarity.
Figure 24: Example model predictions on instances from the Object Counting task, where demonstrations are selected based on visual and textual similarity.
Figure 25: Example model predictions on instances from the Spatial Relations Understanding task, where demonstrations are selected based on visual and textual similarity.
Figure 26: Example model predictions on instances from the Action Recognition task, where demonstrations are selected based on visual and textual similarity.
Figure 27: Example model predictions on instances from the Commonsense Reasoning task, where demonstrations are selected based on visual and textual similarity.
Figure 28: Example model predictions on instances from the Temporal Reasoning task, where demonstrations are selected based on visual and textual similarity.
Figure 29: Example model predictions on instances from the Coreference Resolution task, where demonstrations are selected based on visual and textual similarity.
Figure 30: Example model predictions on instances from the Attribute Recognition task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 31: Example model predictions on instances from the Visual Object Recognition task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 32: Example model predictions on instances from the Plurality Recognition task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 33: Example model predictions on instances from the Object Counting task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 34: Example model predictions on instances from the Spatial Relations Understanding task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 35: Example model predictions on instances from the Action Recognition task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 36: Example model predictions on instances from the Commonsense Reasoning task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 37: Example model predictions on instances from the Temporal Reasoning task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
Figure 38: Example model predictions on instances from the Coreference Resolution task, where demonstrations are selected based on visual and textual similarity, and Chain-of-Thought reasoning is employed.
More in this topic
Multimodal Model Evaluation
Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.
See all 770 papers