Parallel In-context Learning for Large Vision Language Models

Shin'ya Yamaguchi; Daiki Chijiwa; Tamao Sakao; Taku Hasegawa

cs.CVMar 17, 2026

Parallel In-context Learning for Large Vision Language Models

Authors: Shin'ya Yamaguchi, Daiki Chijiwa, Tamao Sakao, Taku Hasegawa

Organizations: NTT

Abstract

Large vision-language models (LVLMs) employ multi-modal in-context learning (MM-ICL) to adapt to new tasks by leveraging demonstration examples. While increasing the number of demonstrations boosts performance, they incur significant inference latency due to the quadratic computational cost of Transformer attention with respect to the context length. To address this trade-off, we propose Parallel In-Context Learning (Parallel-ICL), a plug-and-play inference algorithm. Parallel-ICL partitions the long demonstration context into multiple shorter, manageable chunks. It processes these chunks in parallel and integrates their predictions at the logit level, using a weighted Product-of-Experts (PoE) ensemble to approximate the full-context output. Guided by ensemble learning theory, we introduce principled strategies for Parallel-ICL: (i) clustering-based context chunking to maximize inter-chunk diversity and (ii) similarity-based context compilation to weight predictions by query relevance. Extensive experiments on VQA, image captioning, and classification benchmarks demonstrate that Parallel-ICL achieves performance comparable to full-context MM-ICL, while significantly improving inference speed. Our work offers an effective solution to the accuracy-efficiency trade-off in MM-ICL, enabling dynamic task adaptation with substantially reduced inference overhead.

Figures & tables

Figure 1: Parallel-ICL. Instead of using a full demonstration context, we propose partitioning it into smaller chunk contexts (context chunking) and then integrating the logits from the chunked contexts to output (context compilation) for efficient inference in multi-modal in-context learning (MM-ICL) by large vision-language models (LVLMs). Parallel-ICL enhances inference speed while maintaining competitive performance to the original MM-ICL.

Figure 2: Pipeline of Parallel-ICL . We cluster demonstrations in a multi-modal feature space and utilize each cluster as chunks (context chunking). Then, we process chunk-wise contexts with LVLMs and weight their outputs (logits) based on query-chunk similarity, composing an ensemble for the final prediction as PoE (context compilation). This can be computed by the weighted sum of outputs at the logit level.

Method	Token Length	Accuracy	Latency (sec)
Method	Token Length	Accuracy	Total	Prefill	Decoding
Zero-shot	2,557	0.00	0.099	0.087	0.012
MM-ICL (8-shot)	23,318	56.90	1.004	0.977	0.027
MM-ICL (16-shot)	44,027	58.20	2.376	2.343	0.033
MM-ICL (32-shot)	84,959	58.90	3.479	3.444	0.035

Table 1: Analysis of LVLM Inference . We perform the inference on demo-based task learning of MI-Bench-ICL (demo-based learning) [ 35 ] with LLaVA-OV-7B [ 29 ] . Accuracy increases proportionally with the number of demonstrations, while inference time increases significantly.

Panel of Figure 3 — Figure 3: Performance trend of Parallel-ICL . Parallel-ICL ( $K$ > 1) consistently reduces latency across models. Notably, at $N=32$ , $K=2$ and $K=4$ can outperform full-context MM-ICL ( $K=1$ ) in accuracy, potentially alleviating the “lost in the middle” problem [ 36 ] .

	Latency	Speedup	Approx. Ratio	MI-Bench-ICL				GQA	TextVQA	COCO Caption
	Latency	Speedup	Approx. Ratio	C-VQA	O-VQA	Hall.	Demo	GQA	TextVQA	COCO Caption
LLaVA-OV-7B
MM-ICL (32-shot)	9.662	1.000 $\times$	100.00%	65.20	61.60	79.20	58.90	68.79	73.86	99.63
DivPrune (32 $\rightarrow$ 16-shot)	5.065	1.908 $\times$	99.63%	63.80	61.00	77.50	61.70	67.99	73.60	99.74
DivPrune (32 $\rightarrow$ 8-shot)	2.834	3.410 $\times$	94.39%	50.90	61.70	79.70	60.60	64.89	70.86	90.08
Parallel-ICL (32-shot, 2-chunk)	6.673	1.448 $\times$	100.72 %	63.30	61.30	78.60	63.40	68.87	74.36	101.01

Table 2: Performance comparison on multi-modal benchmarks (32-shot) . Best and second-best scores for each LVLM are bolded and underlined . Parallel-ICL not only achieves significant speedup, but also yields competitive or superior accuracy (Approx. Ratio > 100%) compared to full-context MM-ICL. Combining Parallel-ICL with DivPrune yields an excellent accuracy-efficiency trade-off.

Figure 4: Diversity-Relevance Tradeoff (chunk number

K

in marker) . Increasing the number of chunks improves inter-chunk diversity but decreases the task relevance. This trade-off can be balanced by selecting an appropriate

K

Method	Accuracy	Latency	Diversity ( $\times 10^{-2}$ ) $\uparrow$	Relevance $\uparrow$
MM-ICL	58.90	3.479	N/A	1.000
Parallel-ICL	63.40	2.999	0.378	0.854
w/ random chunking	59.30	2.853	0.247	0.801
w/ uniform compilation	60.00	2.846	0.378	0.777
w/ textual task features	58.90	2.853	0.273	0.786
w/ visual task features	58.20	2.936	0.310	0.778

Table 3: Ablation study on MI-Bench-ICL (Demo) with LLaVA-OV-7B . Our principled components, k-means-based chunking (vs. random) and similarity-based compilation (vs. uniform), are both crucial for accuracy by improving diversity and relevance.

Model	Method	Accuracy	Latency	Speedup
InternVL3.5-8B	MM-ICL (32-shot)	76.60	4.402	1.000 $\times$
	Parallel-ICL (32-shot, 2-chunk)	81.90	3.785	1.163 $\times$
	Parallel-ICL (32-shot, 4-chunk)	80.20	3.338	1.319 $\times$
InternVL3.5-14B	MM-ICL (32-shot)	83.00	6.425	1.000 $\times$
	Parallel-ICL (32-shot, 2-chunk)	84.30	5.435	1.182 $\times$
	Parallel-ICL (32-shot, 4-chunk)	84.30	4.722	1.361 $\times$

Table 4: Model scalability of Parallel-ICL on MI-Bench-ICL (Demo) . Parallel-ICL consistently improves speed and maintains or enhances the accuracy of MM-ICL, demonstrating its effectiveness across different model scales.

Product of Experts

See all 9 papers

Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding
Mar 25, 2026Fatih Ilhan, Gaowen Liu, Ramana Rao Kompella +5Efficient VLM Inference Large Vision-Language Models
Georgia Institute of Technology · Cisco Research
In-Context Collapse in Vision-Language Models and How to Mitigate it?
Aug 3, 2026Mohammad RostamiVLM Robustness Vision-Language Models
Amazon Generative AI Innovation Center∗, USA.
Personalize Your Large Vision-language Models With In-context Prompt Tuning
May 29, 2026Yanshu Li, Jiaqian Li, Kuai Yu +4Vision-Language Models Large Vision-Language Models
Brown University · Columbia University · University of Alabama at Birmingham +2

Mar 25, 2026J/K move · Enter open · S save

Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding

Fatih Ilhan, Gaowen Liu, Ramana Rao Kompella, Selim Furkan Tekin+4

Georgia Institute of Technology · Cisco Research

Efficient VLM Inference Large Vision-Language Models KV-Cache Compression Long-Context VLMs

Open paper AskPDF arXiv

Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory overhead during decoding, especially when the query and answer of VLMs consist of long sequences of visual and text tokens. This paper presents AttentionPack, an adaptive and attention-aware optimization framework tailored for large vision-language models with improving memory-efficiency during decoding, focusing on addressing the challenges due to the increased high number of visual inputs and interactions, particularly in long-context tasks with multiple high-resolution images or videos. AttentionPack is novel in two aspects: (i) We introduce a multi-head attention compaction method for economically storing key and value matrices by exploiting the implicit low-rank structure, and (ii) we develop a token-specific attention-aware decompression mechanism to reduce latency overhead. Experimental results on multiple benchmarks demonstrate that AttentionPack improves memory efficiency by up to 8x, enabling higher batch sizes and faster batch inference while preserving the model output quality or longer context lengths for superior retrieval performance. We also report the effectiveness of AttentionPack combined with eviction, quantization and kernel fusion, showing further efficiency gains for resource-limited environments.

Abstract

Product of Experts

Explore similar work

Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding

In-Context Collapse in Vision-Language Models and How to Mitigate it?

Personalize Your Large Vision-language Models With In-context Prompt Tuning