Organizations: Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education; State Key Laboratory of Advanced Rail Autonomous Operation; School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · School of Psychology & Neuroscience, University of Glasgow, Glasgow, UK
Multimodal learning integrates complementary information from different modalities such as image, text, and audio to improve model performance, but its success relies on large-scale labeled data, which is costly to obtain. Active learning (AL) mitigates this challenge by selectively annotating informative samples. In multimodal settings, many approaches implicitly assume that modality importance is stable across rounds and keep selection rules fixed at the fusion stage, which leaves them insensitive to the dynamic nature of multimodal learning, where the relative value of modalities and the difficulty of instances shift as training proceeds. To address this issue, we propose RL-MBA, a reinforcement-learning framework for modality-balanced, difficulty-aware multimodal active learning. RL-MBA models sample selection as a Markov Decision Process, where the policy adapts to modality contributions, uncertainty, and diversity, and the reward encourages accuracy gains and balance. Two key components drive this adaptability: (1) Adaptive Modality Contribution Balancing (AMCB), which dynamically adjusts modality weights via reinforcement feedback, and (2) Evidential Fusion for DifficultyAware Policy Adjustment (EFDA), which estimates sample difficulty via uncertainty-based evidential fusion to prioritize informative samples. Experiments on Food101, KineticsSound, and VGGSound demonstrate that RL-MBA consistently outperforms strong baselines, improving both classification accuracy and modality fairness under limited labeling budgets.
Figures & tables
Figure 1: Fixed rules vs. adaptive strategy in multimodal active learning. The unlabeled pool contains items with different difficulty levels and different dominant modality. Left (1–2): fixed strategies keep selecting hard, dominant-modality samples and do not adjust when the value of each modality or the difficulty of samples changes across rounds. Right (3–4): an adaptive strategy reweights modality contributions and difficulty, yielding more balanced, budget-efficient batches.
Figure 2: Overview of RL-MBA. Each round consists of (1) multimodal fusion and clustering, (2) evidential uncertainty and difficulty estimation, (3) scoring to form a candidate set, (4) policy-based set selection, (5) retraining, and (6) reward and policy update.
Figure 3: Core components. (a) AMCB updates simplex weights w from round-level modality gains. (b) EFDA fuses Dirichlet evidence to obtain calibrated uncertainty/difficulty; the RL reward updates the selection policy (and optionally uncertainty heads if enabled in implementation).
Figure 4: The example from Food101, KineticsSound and VGGSound
Method
Food101
KineticsSound
VGGSound
Random
0.8470
0.4650
0.2173
Entropy
0.8480
0.4650
0.2043
GCNAL
0.8510
0.4600
0.2033
CoreSet
0.8422
0.4600
0.2013
DeepFool
0.8500
0.4680
0.1973
BALD
0.8450
0.4550
0.1993
Table 1: Top-1 Accuracy on multimodal datasets with 3,000 labelled samples . This corresponds to 6.6% of Food101, 20.4% of KineticsSound, and 2.9% of VGGSound training sets.
Figure 5: Performance comparison between the proposed method and other conventional AL strategies on Food101, KineticsSound, and VGGSound. The metric selected is top-1 accuracy (Top-1) on multimodal and unimodal classification.
Figure 6: Modality contribution ϕ on Food101 (1k–7k labels). ϕ is Shapley-derived; circles = Text, crosses = Image. RL - MBA progressively increases text weight, while baselines remain static.
Figure 7: Combined ablation and reward analysis. (a) Ablation on KineticsSound comparing BMMAL, RL-MBA w/ AMCB, RL-MBA w/ EFDA, and RL-MBA (Full). (b) Reward settings on Food101 comparing Relative, Absolute, and Incremental rewards.
Variant
Food101
KineticsSound
VGGSound
BMMAL (Baseline)
0.8609
0.4745
0.2053
RL-MBA w/ AMCB
0.8621
0.4771
0.2059
RL-MBA w/ EFDA
0.8637
0.4802
0.2177
RL-MBA (Full)
0.8650
0.4841
0.2223
Table 2: Ablation study results (TOP-1 Accuracy) on Food101, KineticsSound, and VGGSound with 3,000 labels.
Strategy
Pred
Sel
Train
Val
Policy Update
Round Total
GCNAL
450.15
700.54
639.16
56.11
0.00
1845.96
DeepFool
263.12
1273.25
639.16
56.11
0.00
2231.64
BALD
519.21
0.44
639.16
56.11
0.00
1214.92
BADGE
162.10
312.84
639.16
56.11
0.00
1170.21
BMMAL
146.00
310.12
639.16
56.11
0.00
1151.39
RL - MBA
155.41
33.48
639.16
56.11
0.23
884.39
Table 3: Food101 per-round wall-clock breakdown (s) with fixed Train/Val across methods. Round Total is computed as Pred + Sel + Train + Val + Policy.
Multimodal learning enables neural networks to integrate information from heterogeneous sources, but active learning in this setting faces distinct challenges. These include missing modalities, differences in modality difficulty, and varying interaction structures. These are issues absent in the unimodal case. While the behavior of active learning strategies in unimodal settings is well characterized, their behavior under such multimodal conditions remains poorly understood. We introduce a new framework for benchmarking multimodal active learning that isolates these pitfalls using synthetic datasets, allowing systematic evaluation without confounding noise. Using this framework, we compare unimodal and multimodal query strategies and validate our findings on two real-world datasets. Our results show that models consistently develop imbalanced representations, relying primarily on one modality while neglecting others. Existing query methods do not mitigate this effect, and multimodal strategies do not consistently outperform unimodal ones. These findings highlight limitations of current active learning methods and underline the need for modality-aware query strategies that explicitly address these pitfalls. Code and benchmark resources will be made publicly available.
Dustin Eisenhardt, Yunhee Jeong, Florian Buettner
German Cancer Research Center (DKFZ), Heidelberg, Germany · German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany · Institute of Informatics, Goethe University Frankfurt, Frankfurt am Main, Germany +3
Foundation models for vision have transformed visual recognition with powerful pretrained representations and strong zero-shot capabilities, yet their potential for data-efficient learning remains largely untapped. Active Learning (AL) aims to minimize annotation costs by strategically selecting the most informative samples for labeling, but existing methods largely overlook the rich multimodal knowledge embedded in modern vision-language models (VLMs). We introduce Conformal Cross-Modal Acquisition (CCMA), a novel AL framework that bridges vision and language modalities through a teacher-student architecture. CCMA employs a pretrained VLM as a teacher to provide semantically grounded uncertainty estimates, conformally calibrated to guide sample selection for a vision-only student model. By integrating multimodal conformal scoring with diversity-aware selection strategies, CCMA achieves superior data efficiency across multiple benchmarks. Our approach consistently outperforms state-of-the-art AL baselines, demonstrating clear advantages over methods relying solely on uncertainty or diversity metrics.
Huy Hoang Nguyen, Cédric Jung, Shirin Salehi +3
AIT Austrian Institute of Technology · Automation & Control Institute, Technical University of Vienna · Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University
Multimodal learning often suffers from modality imbalance, where modalities that converge faster dominate optimization while others remain undertrained. Existing approaches typically mitigate this issue by strengthening the weak modality or adjusting optimization gradients. However, such strategies mainly compensate for optimization rate discrepancies, often at the expense of the strong modality's optimization capacity, without analyzing how these discrepancies arise at the modality level. Based on theoretical insights and empirical observations, we argue that the discrepancy of learning pace arises from differences in the mapping difficulty between modality-specific feature space and the shared label space. To address this issue, we propose Balanced Multimodal Label Reshaping (BMLR), the first method that promotes multimodal balance from the label-side design. BMLR reshapes the cross-modal label space to equalize mapping difficulty across modalities, thereby facilitating modality interaction and injecting richer inter-class information into each modality. Extensive experiments across multiple architectures demonstrate that BMLR consistently improves multimodal performance and exhibits strong compatibility with diverse model designs. The source code will be released soon.