Generalizing Multimodal Large Language Models (MLLMs) to novel video domains is essential for real-world deployment but remains challenging due to the scarcity of labeled data. While In-Context Learning (ICL) offers a training-free adaptation path, standard methods rely on large annotated pools, which are often impractical in specialized environments like industrial or surgical settings since they require the experts' annotations. To bridge this gap, we introduce VIOLA (Video In-cOntext Learning with minimal Annotation), a label-efficient framework that synergizes minimal expert supervision with abundant unlabeled data. First, to maximize the efficiency of a strict annotation budget, we propose density-uncertainty-weighted sampling. Unlike standard diversity or uncertainty strategies that risk selecting visual outliers, our method leverages density estimation to identify samples that are simultaneously diverse, representative, and informative. Second, to utilize the remaining unlabeled data without noise propagation, we construct a hybrid pool and introduce confidence-aware retrieval and confidence-aware prompting. These mechanisms explicitly model label reliability, retrieving demonstrations based on a composite score of similarity and confidence while enabling the MLLM to adaptively distinguish between verified ground truths and noisy pseudo-labels. Extensive experiments across nine diverse benchmarks using three MLLMs demonstrate that our framework significantly outperforms various baselines in low-resource settings, achieving robust adaptation with minimal annotation costs. Project page: https://fujiry0.github.io/VIOLA-project-page/
Figures & tables
Figure 1: Problem setup and performance comparison. (a) Unlike methods requiring large labeled datasets, our approach, VIOLA, strategically selects a minimal subset of informative samples for expert annotation while leveraging abundant unlabeled videos via pseudo-labeling. This constructs a hybrid pool from which the model retrieves relevant demonstrations for inference. (b) Performance comparison with Naive ICL (which randomly selects samples for annotation), averaged across seven classification datasets. VIOLA achieves robust adaptation with significantly reduced annotation costs.
Figure 2: Overview of our proposed framework. The pipeline consists of three stages. First, we acquire expert labels for a small, informative subset ( DL ) using density-uncertainty-weighted sampling (Selective annotation stage). Then, we generate high-confidence pseudo-labels via in-context pseudo-annotation to construct a hybrid pool DH (Pseudo-annotation stage). Lastly, we predict final answers via confidence-aware retrieval and prompting during inference.
Methods
Classification
Captioning
Drive & Act
EgoPet
EgoSurgery
ENIGMA
UCF-Crime
Xsports
MammAlps
Bora
CapERA
Qwen2-VL
Zero-shot
7.2
14.7
25.5
4.9
31.5
18.5
43.8
0.261
0.274
Random
20.9
19.0
26.8
58.8
37.9
21.5
64.2
0.338
0.378
+Pseudo Annotation
15.4
32.4
21.8
58.2
39.3
8.0
46.4
0.257
0.312
+VideoICL [ 31 ]
11.2
16.2
28.3
57.5
34.7
23.5
63.2
0.330
0.335
VoteK [ 55 ]
18.1
21.2
13.2
54.8
36.3
26.5
64.4
0.359
0.382
Table 1: Main results on diverse video understanding benchmarks. We evaluate our framework against various baselines across classification and captioning tasks, using a fixed pool of 20 labeled samples for each dataset. Δ denotes the performance gain of Ours relative to the Zero-shot baseline. Results for VideoICL are reproduced using the official repository 2 2 2 https://github.com/KangsanKim07/VideoICL .
Figure 3: Performance trends under varying oracle annotation budgets. We compare our framework against baselines using Qwen2-VL-7B, varying the labeled pool size from 20 to 100 samples.
λ
0.0
0.1
0.3
0.5
0.7
0.9
1.0
Drive&Act
24.3
27.4
26.7
26.4
26.3
24.5
23.0
ENIGMA
57.0
58.2
58.2
58.5
56.0
56.9
56.9
XSports
22.2
23.9
23.9
25.9
23.9
21.7
16.5
CapEra
0.368
0.393
0.393
0.393
0.382
0.373
0.368
Table 2: Ablation study of density-uncertainty-weighted Sampling.
Sel. Anno.
CA Retrieval
CA Prompt
Drive&Act
XSports
Bora
CapEra
20.9
18.5
0.338
0.378
✓
✓
21.9
15.1
0.335
0.372
✓
✓
17.6
15.0
0.364
0.373
✓
✓
24.3
22.2
0.362
0.368
✓
✓
✓
26.4
25.9
0.365
0.393
Table 3: Ablation study of Selective Annotation via Density-Uncertainty-weighted Selection and Confidence-Aware (CA) Inference mechanisms.
CA Retrieval
CA Prompt
Drive&Act
CapEra
0
50
75
90
95
99
0
50
75
90
95
99
18.7
22.7
23.6
14.5
16.1
17.4
0.356
0.358
0.362
0.362
0.368
0.371
✓
18.9
18.1
18.1
18.6
17.6
17.4
0.372
0.373
0.373
0.373
0.373
0.372
✓
25.1
25.9
26.1
23.2
24.3
24.5
0.370
0.372
0.374
0.363
0.368
0.371
✓
✓
26.3
26.3
26.3
26.3
26.4
25.4
0.393
0.393
0.393
0.390
0.393
0.393
Table 4: Sensitivity analysis of confidence thresholds for Confidence-Aware (CA) mechanisms.
Pseudo-Annotation Strategy
Xsports
Bora
Zero-shot
24.2
0.337
In-Context (Ours)
25.9
0.365
Table 5: Comparison of pseudo-annotation strategies. We assess the benefit of using in-context examples (In-Context) over raw model predictions (Zero-shot) for generating pseudo-labels.
School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China +1