HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

Xiaoyu Li; Yuhang Liu; Xuanshuo Kang; Zheng Luo; Fangqi Lou; Xiaohua Wu; Zihan Xiong

cs.CVMar 13, 2026

HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

Authors: Xiaoyu Li, Yuhang Liu, Xuanshuo Kang, Zheng Luo, Fangqi Lou, Xiaohua Wu, Zihan Xiong

Organizations: University of Electronic Science and Technology of China · Institute of Electronics and Information Industry Technology of Kashgar

Abstract

In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.

Figures & tables

Figure 1: Overview of the HiFICL framework. (a) The overall training process involves a frozen LMM backbone and layer-wise trainable virtual key-value pairs, optimized via a final task loss. (b) Inside each attention head, these virtual pairs are used to compute the high-fidelity attention output according to our derived formula ( Eq. 1 ). (c) Conceptually, unlike previous linear shift methods that only learn an additive offset ( $y=x+b$ ), HiFICL introduces a dynamic scaling factor ( $\alpha$ ) to modulate the original attention ( $y=ax+b$ ). This non-linear partitioning provides a more faithful fit to the underlying data manifold.

Figure 2: Architectural comparison of HiFICL and MimIC. (a) HiFICL implements the full, non-linear attention dynamic. (b) MimIC simplifies the ICL effect into a uni-directional linear shift.

Figure 3: Illustration of the Low-Rank Adaptation (LoRA) mechanism. LoRA injects static, trainable low-rank updates ( $\Delta W$ ) into the frozen weight matrices of an attention module.

Model	Method	# Params (M)	VQAv2	OK-VQA	COCO
LLaVA	Zero-shot	-	13.02	5.10	1.1516
	8-shot ICL	-	68.19	43.84	1.2085
	LoRA	19.7 ( $\times$ 8.95)	70.12	48.19	1.0665
	LIVE	0.13 ( $\times$ 0.05)	74.17	51.77	1.2743
	MimIC	17.0 ( $\times$ 7.7)	74.40	52.29	1.3169
	HiFICL	2.2 ( $\times$ 1.0)	74.66	54.19	1.3315

Table 1: Performance comparison on VQAv2, OK-VQA, and COCO. Best results are bolded , second best are underlined .

Figure 4: Data efficiency on VQAv2 and COCO across different models and training set sizes.

Training Paradigm:
Method / Variant	VQAv2	OK-VQA	COCO
HiFICL (Ours)	72.08	59.56	1.2951
+ Teacher	$70.09$	$59.13$	$1.2844$
Dual LoRA Components:
- LoRA on K	$70.58$	$55.72$	$1.2652$
- LoRA on V	$69.31$	$56.86$	$1.2618$

Table 2: Ablation study of HiFICL’s core components on the Idefics2 model.

Figure 5: Training efficiency comparison on the Idefics2 model. Bars show the relative cost of MimIC normalized against our method HiFICL (Baseline=1.0x). MimIC’s paradigm incurs substantial overhead.

Figure 6: Relative inference runtime comparison on VQAv2 using the LLaVA-Interleave-7b model. All runtimes are normalized to the Zero-shot baseline (1x).

Figure 7: Effect of rank (

r

) on HiFICL’s performance across different datasets using the Idefics2 model. The optimal rank varies with task complexity, peaking at

r=8

for VQAv2 and

r=16

for OK-VQA.

	0-shot	8-shot ICL	LoRA	LIVE	MimIC	HiFICL
CHAIRs $\downarrow$	2.8	5.6	3.3	4.4	4.0	3.2
CHAIRi $\downarrow$	3.3	3.9	2.4	3.1	2.9	2.2
Recall $\uparrow$	27.2	44.2	44.3	44.8	45.4	45.7

Table 3: Hallucination analysis on COCO Captioning (Idefics2). Lower CHAIR is better, higher Recall is better.

Task	Prefix prompt	ICL prompt	Stop words
VQAv2 / OK-VQA	Instruction: provide an answer to the question. Use the image to answer.	Image: {image} Question: {question} Answer: {answer}\n	“Question”, “Answer”, “Image”
COCO	—	Image: {image} Caption: {caption}\n	“Caption”, “Image”
COCO ICL	Instruction: provide a short caption of the input image.\n

Table 4: Prompt templates used in our experiments. Curly brackets {} indicate fields filled with instance-specific content.

Hyperparameter	Value
Optimizer	AdamW
Base Learning Rate	5e-3
LR Schedule	Cosine w/ 10% Warmup
Weight Decay	0.05
Precision	16-mixed
Batch Size (per GPU)	2

Table 5: Main hyperparameters for all trainable methods. Specific learning rates for LoRA and LIVE’s magnitude parameter are noted in the text.

Dataset	LLaVA-Interleave	Idefics2
VQAv2	8	8
OK-VQA	4	16
COCO	8	4

Table 6: The optimal rank

r

used in HiFICL for each dataset and backbone model.

Backbone	Method	VQAv2	OK-VQA	COCO
Idefics1-9B	Zero-shot	29.25	30.54	63.06
	32-shot ICL	56.18	48.48	105.89
	MimIC	59.64	52.05	114.89
	HiFICL	59.71	51.93	115.21
LLaVA-v1.6 (Mistral-7B)	Zero-shot	70.00	63.00	0.7157
	8-shot ICL	68.00	56.00	0.6678

Table 7: Performance comparison across different backbone models. Best results in each group are bolded , second best are underlined .

Figure 8: Qualitative examples on VQAv2 (left) and COCO Captioning (right) using the LLaVA-Interleave-7b model.

Figure 9: Schematic illustrations of different LoRA variants. (a) Standard LoRA applies a dense low-rank update using matrices $A$ and $B$ . (b) MoE-based LoRA decomposes the update into multiple smaller ”expert” pairs $\{A_{i},B_{i}\}$ and uses a router to select a sparse combination. (c) FlyLoRA utilizes a frozen random matrix for $A$ and only trains a sparse matrix $B$ , activating a small subset of its columns for each input.

Figure 10: Training efficiency comparison between LoRA and HiFICL on the Idefics2 model. Costs (GPU Memory, Training Time, FLOPs) are relative to HiFICL (Baseline=1.0x).

Model	Method	# Params (M)	VQAv2	OK-VQA	COCO
LLaVA	LoRA	9.8 ( $\times$ 4.5)	75.75	54.28	1.3307
	HiFICL	2.2 ( $\times$ 1.0)	74.66	53.19	1.3315
Idefics2	LoRA	8.8 ( $\times$ 4.0)	69.58	60.18	1.3448
	HiFICL	2.2 ( $\times$ 1.0)	72.08	58.96	1.2851

Table 8: Performance comparison between LoRA and HiFICL with an optimal rank of

r=8

. Best results for each model are bolded .

Multimodal ICL

Multimodal in-context learning (ICL) studies how models use examples and other context to solve tasks involving images, text, audio, and video.

See all 46 papers

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
Apr 15, 2026Yu Wang, Sharon LiMultimodal ICL In-Context Learning
Department of Computer Sciences, University of Wisconsin-Madison
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
Jun 3, 2026Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh AfghahEfficient Multimodal Inference Multimodal ICL
Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, United States.
UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy
Mar 25, 2026Yicheng Xu, Jiangning Zhang, Zhucun Xue +5Multimodal ICL Unified Multimodal Models
1Zhejiang University, China. · 2Shanghai Jiaotong University, China. · 3National University of Singapore, Singapore. +1

Apr 15, 2026J/K move · Enter open · S save

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

Yu Wang, Sharon Li

Department of Computer Sciences, University of Wisconsin-Madison

Multimodal ICL In-Context Learning Multimodal Large Language Models Modality Gap in VLMs

Open paper AskPDF arXiv

In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available here.

Abstract

Multimodal ICL

Explore similar work

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy