Conformal Cross-Modal Active Learning

Huy Hoang Nguyen; Cédric Jung; Shirin Salehi; Tobias Glück; Anke Schmeink; Andreas Kugi

cs.CVMar 24, 2026

Conformal Cross-Modal Active Learning

Authors: Huy Hoang Nguyen, Cédric Jung, Shirin Salehi, Tobias Glück, Anke Schmeink, Andreas Kugi

Organizations: AIT Austrian Institute of Technology · Automation & Control Institute, Technical University of Vienna · Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University

Abstract

Foundation models for vision have transformed visual recognition with powerful pretrained representations and strong zero-shot capabilities, yet their potential for data-efficient learning remains largely untapped. Active Learning (AL) aims to minimize annotation costs by strategically selecting the most informative samples for labeling, but existing methods largely overlook the rich multimodal knowledge embedded in modern vision-language models (VLMs). We introduce Conformal Cross-Modal Acquisition (CCMA), a novel AL framework that bridges vision and language modalities through a teacher-student architecture. CCMA employs a pretrained VLM as a teacher to provide semantically grounded uncertainty estimates, conformally calibrated to guide sample selection for a vision-only student model. By integrating multimodal conformal scoring with diversity-aware selection strategies, CCMA achieves superior data efficiency across multiple benchmarks. Our approach consistently outperforms state-of-the-art AL baselines, demonstrating clear advantages over methods relying solely on uncertainty or diversity metrics.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of our proposed AL framework CCMA for image classification by exploring conformal prediction with multimodal uncertainty and diversity for efficient data acquisition . Given labeled data $\mathcal{L}$ and an unlabeled pool $\mathcal{U}$ , a frozen vision encoder (student) and a frozen VLM (teacher) serve as feature extractors, while a simple linear classifier is trained on the student features. (1) A selective subpool is formed via CLIP-space clustering. (2–3) Student and teacher posteriors $p_{S}$ , $p_{T}$ are calibrated into conformal sets $\Gamma_{S},\Gamma_{T}$ . (4) Multimodal disagreement $\Omega(x)$ combines entropy and Jensen-Shannon (JS) divergence for uncertainty scoring $\delta(x)$ . (5) Top-ranked samples are oversampled and clustered with uncertainty-weighted coverage for the final diverse selection.

Panel of Figure 2 — Figure 2: Labels required to reach target accuracies of 80% ( blue ), 85% ( orange ), and 90% ( green ) on Food101 and DomainNet-Real. Lower values indicate higher label efficiency. CCMA consistently reaches each accuracy threshold with fewer labeled samples than uncertainty- and coverage-based baselines, demonstrating improved sample efficiency across both datasets, especially in low-budget regimes.

Figure 3: CCMA diagnostics in the overlap and the fraction Top-1 disagreement between teacher and student.

Panel of Figure 4 — Figure 4: Test mean accuracy over 5 seeds for CCMA with other AL methods on Caltech256 and Caltech101 datasets.

Panel of Figure 5 — Figure 5: Effect of oversampling factor $\kappa$ .

Figure 6: Motivation for investigating the impact of the teacher model in CCMA method. We report the CCMA’s accuracy in three different regimes when the teacher is confident (

\tau=0.01

), weak (

\tau=0.30

), and disabled on the Food101 benchmark.

Figure 7: AULC over round vs mean query time per round on the CIFAR100 dataset.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: CCMA diagnostic curves across AL iterations. (a) Overlap between teacher and student conformal sets, measuring the agreement in predicted label supports. Higher overlap indicates stronger cross-modal consistency. (b) Symmetric difference between the two sets, capturing the size of their disagreement region. Larger values reflect a persistent teacher–student mismatch. (c) Top-1 disagreement rate, the probability that the teacher and student assign different most-likely classes. These metrics reveal how the informativeness of teacher–student disagreement evolves: datasets like CIFAR100 and Food101 exhibit sustained mismatch (useful for CCMA), whereas Caltech101/256 align quickly, reducing the value of cross-modal uncertainty and shifting the problem toward coverage-based selection.

Panel of Figure 9 — Figure 9: Teacher–student confidence dynamics across AL iterations. (a) JS divergence between teacher and student posteriors restricted to their shared support, quantifying cross-modal disagreement. (b) Student top-1 confidence, showing how quickly the student becomes certain as labeled data accumulates. (c) The teacher’s top-1 confidence, which remains relatively stable across iterations since the teacher is frozen. These curves highlight the regimes where CCMA is most effective: datasets like CIFAR100 and Food101 maintain moderate teacher–student disagreement (useful for cross-modal query scoring), whereas Caltech101/256 exhibit rapid confidence saturation and early agreement, reducing the benefit of disagreement-based acquisition.

Figure 10: Effect of oversampling factor

\kappa

on DomainNetReal

Variant	AULC	Final@R	Time/rd (s)	AULC/s
V1	0.859	0.915	3.015	0.285
V2	0.828	0.899	0.780	1.061
V3	0.835	0.906	0.512	1.629
V4	0.809	0.896	2.929	0.276
V5	0.793	0.885	0.583	1.360

Appendix

Table 2: Mean over 5 seeds. Final accuracy at the last round (Final@R), AULC computed over rounds, and AULC per second on the CIFAR100 dataset.

Panel of Figure 11 — Figure 11: Test mean accuracy over 5 seeds for CCMA with DINOv2 ViT-g14 as a student model and CLIP ViT-L14 as a teacher model with other AL methods on CIFAR100, Food101, and DomainNet-Real datasets.

Panel of Figure 12 — Figure 12: Test mean accuracy over 5 seeds for CCMA with DINOv2 ViT-g14 as a student model and CLIP ViT-g14 as a teacher model with other AL methods on CIFAR100, Food101, and DomainNet-Real datasets.

Panel of Figure 13 — Figure 13: Test mean accuracy over 5 seeds for CCMA with DINOv2 ViT-L14 as a student model and CLIP ViT-L14 as a teacher model with other AL methods on CIFAR100, Food101, and DomainNet-Real datasets.

Panel of Figure 14 — Figure 14: Test mean accuracy over 5 seeds for CCMA with DINOv2 ViT-L14 as a student model and CLIP ViT-g14 as a teacher model with other AL methods on CIFAR100, Food101, and DomainNet-Real datasets.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 274 papers

Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning
Mar 26, 2026Yuqiao Zeng, Xu Wang, Tengfei Liang +3Active Learning Multimodal Classification
Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education; State Key Laboratory of Advanced Rail Autonomous Operation; School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · School of Psychology & Neuroscience, University of Glasgow, Glasgow, UK
Leveraging Vision-Language Models as Weak Annotators in Active Learning
May 1, 2026Phuong Ngoc Nguyen, Kaito Shiku, Ryoma Bise +2Vision-Language Models Active Learning
Kyushu University, Japan.
Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention
May 12, 2026Hamza Ahmed Durrani, Rafay Suleman DurraniVision-Language Models Cross-Modal Alignment
Sejong University, Computer Science Engineering · Technische Universität Ilmenau, Computer Engineering

Mar 26, 2026J/K move · Enter open · S save

Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning

Yuqiao Zeng, Xu Wang, Tengfei Liang, Yiqing Hao+2

Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education; State Key Laboratory of Advanced Rail Autonomous Operation; School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · School of Psychology & Neuroscience, University of Glasgow, Glasgow, UK

Active Learning Multimodal Classification Uncertainty-Aware Multimodal Fusion Reinforcement Learning

Open paper AskPDF arXiv

Multimodal learning integrates complementary information from different modalities such as image, text, and audio to improve model performance, but its success relies on large-scale labeled data, which is costly to obtain. Active learning (AL) mitigates this challenge by selectively annotating informative samples. In multimodal settings, many approaches implicitly assume that modality importance is stable across rounds and keep selection rules fixed at the fusion stage, which leaves them insensitive to the dynamic nature of multimodal learning, where the relative value of modalities and the difficulty of instances shift as training proceeds. To address this issue, we propose RL-MBA, a reinforcement-learning framework for modality-balanced, difficulty-aware multimodal active learning. RL-MBA models sample selection as a Markov Decision Process, where the policy adapts to modality contributions, uncertainty, and diversity, and the reward encourages accuracy gains and balance. Two key components drive this adaptability: (1) Adaptive Modality Contribution Balancing (AMCB), which dynamically adjusts modality weights via reinforcement feedback, and (2) Evidential Fusion for DifficultyAware Policy Adjustment (EFDA), which estimates sample difficulty via uncertainty-based evidential fusion to prioritize informative samples. Experiments on Food101, KineticsSound, and VGGSound demonstrate that RL-MBA consistently outperforms strong baselines, improving both classification accuracy and modality fairness under limited labeling budgets.

Abstract

Active Learning

Explore similar work

Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning

Leveraging Vision-Language Models as Weak Annotators in Active Learning

Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention