Multimodal ICL

ICL: In-Context Learning

Multimodal in-context learning asks whether models can infer task rules or acquire useful information from examples at inference time, without updating their weights. Common approaches retrieve or learn to select demonstrations, structure prompts for reasoning, and compress examples into task vectors, adapters, or reusable memory; applications include visual question answering, classification, audio understanding, and video interaction. Benchmarks increasingly test not only answers but whether models ground context, apply new rules, and adapt to changing criteria. A central tension is that strong zero-shot performance does not guarantee effective use of demonstrations: multimodal grounding, noisy or irrelevant context, and transfer from examples to new queries remain persistent challenges.

Latest papers

49 papers9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models
Jun 10, 2026Garvita Allabadi, Matteo Sodano, Roberto Estevão +4Multimodal ICL Multimodal Large Language Models
University of Illinois Urbana Champaign · University of Bonn · Microsoft
Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning
Jun 10, 2026Zhirui Chen, Ziwei Chen, Ling ShaoMemory-Augmented Language Models Multimodal ICL
UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, China · ByteDance Inc.
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
Jun 3, 2026Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh AfghahEfficient Multimodal Inference Multimodal ICL
Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, United States.
MMCL-Bench: Multimodal Context Learning from Visual Rules, Procedures, and Evidence
May 12, 2026Yifan Chen, Fei Yin, Qingyan Bai +2Multimodal ICL Multimodal Reasoning
University of Cambridge · HKUST · Tsinghua University
PromptDx: Differentiable Prompt Tuning for Multimodal In-Context Alzheimer's Diagnosis
May 9, 2026Lujia Zhong, Yihao Xia, Shuo Huang +2Multimodal ICL Medical Diagnosis
Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California · Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California · Alfred E. Mann Department of Biomedical Engineering, Viterbi School of Engineering, University of Southern California
Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
May 4, 2026Haoyu Wang, Haonan Wang, Yuyan Chen +5Multimodal ICL In-Context Learning
Tencent QQ · Fudan University · Cornell University
Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
Apr 15, 2026Yu Wang, Sharon LiMultimodal ICL In-Context Learning
Department of Computer Sciences, University of Wisconsin-Madison
Multimodal Large Language Models for Multi-Subject In-Context Image Generation
Apr 8, 2026Yucheng Zhou, Dubing Chen, Huan Zheng +1Multi-Subject Image Generation Image Generation
SKL-IOTSC, CIS, University of Macau
Few-shot Writer Adaptation via Multimodal In-Context Learning
Mar 31, 2026Tom Simon, Stephane Nicolas, Pierrick Tranouez +2Optical Character Recognition Few-Shot Learning
LITIS EA4108, University of Rouen Normandy, France · LITIS EA4108, INSA of Rouen Normandy, France
Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis
Mar 29, 2026Wenkai Zhao, Zipei Wang, Mengjie Fang +3Multimodal ICL Medical VLMs
CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences · School of Software, North University of China
UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy
Mar 25, 2026Yicheng Xu, Jiangning Zhang, Zhucun Xue +5Multimodal ICL Unified Multimodal Models
1Zhejiang University, China. · 2Shanghai Jiaotong University, China. · 3National University of Singapore, Singapore. +1
Learning to Select Visual In-Context Demonstrations
Mar 24, 2026Eugene Lee, Yu-Chi Lin, Jiajie DiaoIn-Context Example Selection Multimodal ICL
University of Cincinnati · University of California, Los Angeles
When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal
Mar 20, 2026Yen-Ting Piao, Jay Chiehen Liao, Wei-Tang Chien +5Audio-Language Model Evaluation LLM Evaluation
National Taiwan University, Taiwan
Retrieving Counterfactuals Improves Visual In-Context Learning
Mar 17, 2026Guangzhi Xiong, Sanchit Sinha, Zhenghao He +1Multimodal ICL Retrieval-Augmented ICL
University of Virginia
Parallel In-context Learning for Large Vision Language Models
Mar 17, 2026Shin'ya Yamaguchi, Daiki Chijiwa, Tamao Sakao +1Product of Experts In-Context Learning
NTT
HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks
Mar 13, 2026Xiaoyu Li, Yuhang Liu, Xuanshuo Kang +4Multimodal ICL Multimodal Large Language Models
University of Electronic Science and Technology of China · Institute of Electronics and Information Industry Technology of Kashgar
Differentially Private Multimodal In-Context Learning
Mar 5, 2026Ivoline C. Ngong, Zarreen Reza, Joseph P. NearPrivacy Protection in VLMs Multimodal ICL
University of Vermont, Burlington, VT, USA · Independent Researcher
Large Multimodal Models as General In-Context Classifiers
Feb 26, 2026Marco Garosi, Matteo Farina, Alessandro Conti +2Multimodal Classification Classification
University of Trento · Fondazione Bruno Kessler
FewMMBench: A Benchmark for Multimodal Few-Shot Learning
Feb 25, 2026Mustafa Dogan, Ilker Kesen, Iacer Calixto +2Multimodal Model Evaluation Multimodal Large Language Models
Aselsan Research · Department of Computer Science, University of Copenhagen · Department of Medical Informatics, Amsterdam UMC, University of Amsterdam +4

Jun 10, 2026J/K move · Enter open · S save

GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models

Garvita Allabadi, Matteo Sodano, Roberto Estevão, Yuxiong Wang+3

University of Illinois Urbana Champaign · University of Bonn · Microsoft

Multimodal ICL Multimodal Large Language Models In-Context Learning Multimodal Foundation Models Retrieval-Augmented ICL

Open paper AskPDF arXiv

In-Context Learning (ICL) has become a powerful mechanism for adapting Large Language Models (LLMs) to new tasks without fine-tuning. Extending this concept to Large Multimodal Models (LMMs), Multimodal In-Context Learning (M-ICL) relies on retrieving relevant examples, such as images, captions, or question-answer pairs, to guide predictions across tasks like classification, captioning, and visual question answering (VQA). Most existing approaches select in-context examples based on feature-space similarity, assuming that semantically similar samples provide the most useful context. However, our systematic analysis reveals that this assumption does not always hold: visually similar examples are not necessarily those that most effectively enhance in-context learning performance. To address this, we propose the Guided Retrieval of In-context Prompts (GRIP), a learnable vision-only retrieval framework that leverages feedback from LMMs to identify examples that truly improve model predictions. GRIP learns to distinguish beneficial from detrimental in-context examples through contrastive training, refining retrieval beyond pure similarity. Across three multimodal tasks, namely classification, captioning, and VQA, GRIP improves consistently over similarity-based retrieval on Qwen2.5-VL-7B, with its strongest gains in classification on Idefics2-8B. Moreover, we demonstrate that retrievers trained with feedback from one open LMM can be transferred to other models without retraining, including closed-source GPT-4o and Gemini, enabling scalable and cost-efficient deployment of M-ICL. Code will be published upon acceptance.