Mind the Gap: A Framework for Assessing Pitfalls in Multimodal Active Learning

Dustin Eisenhardt; Yunhee Jeong; Florian Buettner

cs.LGMar 31, 2026

Mind the Gap: A Framework for Assessing Pitfalls in Multimodal Active Learning

Authors: Dustin Eisenhardt, Yunhee Jeong, Florian Buettner

Organizations: German Cancer Research Center (DKFZ), Heidelberg, Germany · German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany · Institute of Informatics, Goethe University Frankfurt, Frankfurt am Main, Germany · Crop Science Division, Bayer AG, Frankfurt am Main, Germany · Department of Medicine, Goethe University Frankfurt, Frankfurt am Main, Germany · Frankfurt Cancer Institute (FCI), Frankfurt am Main, Germany

Abstract

Multimodal learning enables neural networks to integrate information from heterogeneous sources, but active learning in this setting faces distinct challenges. These include missing modalities, differences in modality difficulty, and varying interaction structures. These are issues absent in the unimodal case. While the behavior of active learning strategies in unimodal settings is well characterized, their behavior under such multimodal conditions remains poorly understood. We introduce a new framework for benchmarking multimodal active learning that isolates these pitfalls using synthetic datasets, allowing systematic evaluation without confounding noise. Using this framework, we compare unimodal and multimodal query strategies and validate our findings on two real-world datasets. Our results show that models consistently develop imbalanced representations, relying primarily on one modality while neglecting others. Existing query methods do not mitigate this effect, and multimodal strategies do not consistently outperform unimodal ones. These findings highlight limitations of current active learning methods and underline the need for modality-aware query strategies that explicitly address these pitfalls. Code and benchmark resources will be made publicly available.

Figures & tables

Panel of Figure 1 — Figure 1: Examples of the QuintFeatures dataset.

Dataset	ModDrop	AULC
		Modality A	Both	Modality B
Missing		Rare	Both	Frequent
	Yes	1.01 ± 0.08	7.94 ± 0.10	7.95 ± 0.11
	No	0.97 ± 0.05	7.94 ± 0.13	7.96 ± 0.13
Share		CIFAR-10	Both	MNIST
	Yes	5.10 ± 0.11	8.91 ± 0.01	8.81 ± 0.02

Table 1: Random Query Method on Synthetic Data.

\mathrm{AULC}

for the ‘high-label’ regime using the Random query method, evaluated on each modality combination separately. + : Scores of partially predictive modalities are scaled by a factor of ten for comparability.

Dataset Regime		Metric	AULC per Query Method
Dataset Regime			BADGE	BALD	BMMAL	Entropy	GRACE	KCG	Random
Missing	Low	Rare	1.05	1.07	1.13	1.19	1.10	1.19	0.97
		Both	1.13	1.02	1.06	1.12	1.02	1.22	0.96
		Frequent	1.08	0.91	0.96	0.94	0.91	1.10	0.92
	Mid	Rare	1.50	2.33	2.46	3.49	2.91	4.99	1.01
		Both	5.77	5.25	5.63	5.01	5.22	6.09	5.51

Table 2: Query Method Comparison on Synthetic Data.

\mathrm{AULC}

per query method on synthetic datasets across all label regimes. Results are reported separately for each modality combination. Best entries are highlighted in bold , the weak modality in italics .

Label Regime	Query Method
	BADGE	BALD	BMMAL	Entropy	GRACE	KCG	Random
Low	16	22	14	17	15	11	17
Mid	22	21	14	17	17	6	15
High	14	11	15	21	15	20	16

Table 3: Summary of Query Method Rankings on Synthetic Data. In order to compare the query methods, we sum the ranks obtained by the query methods for all datasets. The sum of ranks indicates the overall performance. Lower is better. The best entries are highlighted in bold .

Dataset Regime		Metric	AULC per Query Method
Dataset Regime			BADGE	BALD	BMMAL	Entropy	GRACE	KCG	Random
Food101	Low	Image	0.98	0.90	0.94	0.83	0.83	0.88	0.97
		Both	2.42	3.45	3.38	3.42	3.44	3.05	2.54
		Text	1.80	2.96	2.86	3.01	3.01	2.50	1.90
	Mid	Image	0.96	0.92	0.93	0.78	0.94	0.93	1.04
		Both	3.42	4.47	3.22	3.70	3.14	3.43	3.23

Table 4: Query Method Comparison on Real-World Data.

\mathrm{AULC}

per query method on real-world datasets across all label regimes. Results are reported separately for each modality combination. Best entries are highlighted in bold , the weak modality in italics .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Label Regime	Epochs	Batch Size	Learning Rate	Weight Decay	Augment.
Missing		60	128	0.1	5e-4	RandAug
Share		200	1024	0.1	5e-3	RandAug
Unique		60	128	0.1	5e-4	Basic
Synergy		60	128	0.1	5e-4	Basic
Food101		60	128	0.1	5e-4	RandAug
MIMIC	Low	60	256	0.01	5e-5	None

Appendix

Table 5: Training Settings

Dataset	Label Regime	Initial Budget	Validation Size	Acquisition Size	Iterations
Missing Share	Low	50	250	50	10
Missing Share	Mid	250	1250	250	10
	High	1000	5000	1000	10
Unique Synergy MIMIC	Low	500	2500	500	10
Unique Synergy MIMIC	Mid	1000	5000	1000	10
	High	5000	5000	5000	10

Appendix

Table 6: Active Learning Settings

Panel of Figure 2 — Figure 2: Results for Missing with ModDrop

Panel of Figure 3 — Figure 3: Results for Missing without ModDrop

Panel of Figure 4 — Figure 4: Results for Share with ModDrop

Panel of Figure 5 — Figure 5: Results for Share without ModDrop

Panel of Figure 6 — Figure 6: Results for Unique with ModDrop

Panel of Figure 7 — Figure 7: Results for Unique without ModDrop

Panel of Figure 8 — Figure 8: Results for Synergy with ModDrop

Panel of Figure 9 — Figure 9: Results for Synergy without ModDrop

Panel of Figure 10 — Figure 10: Results for Food101 with ModDrop

Panel of Figure 11 — Figure 11: Results for MIMIC with ModDrop

Explore similar work

Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning
Mar 26, 2026Yuqiao Zeng, Xu Wang, Tengfei Liang +3Active Learning Multimodal Classification
Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education; State Key Laboratory of Advanced Rail Autonomous Operation; School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · School of Psychology & Neuroscience, University of Glasgow, Glasgow, UK
Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning
May 26, 2025Sanghyuk Chun, Olga RussakovskyCross-Modal Alignment Multimodal Model Evaluation
1Princeton University
Unsupervised Learning for Missing Modalities in Multimodal Learning
Jun 14, 2026Hassan Ismkhan, Hamid BouchahciaMissing-Modality Learning Unsupervised Learning
Bournemouth University Bournemouth, UK

Mar 26, 2026J/K move · Enter open · S save

Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning

Yuqiao Zeng, Xu Wang, Tengfei Liang, Yiqing Hao+2

Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education; State Key Laboratory of Advanced Rail Autonomous Operation; School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China · School of Psychology & Neuroscience, University of Glasgow, Glasgow, UK

Active Learning Multimodal Classification Uncertainty-Aware Multimodal Fusion Reinforcement Learning

Open paper AskPDF arXiv

Multimodal learning integrates complementary information from different modalities such as image, text, and audio to improve model performance, but its success relies on large-scale labeled data, which is costly to obtain. Active learning (AL) mitigates this challenge by selectively annotating informative samples. In multimodal settings, many approaches implicitly assume that modality importance is stable across rounds and keep selection rules fixed at the fusion stage, which leaves them insensitive to the dynamic nature of multimodal learning, where the relative value of modalities and the difficulty of instances shift as training proceeds. To address this issue, we propose RL-MBA, a reinforcement-learning framework for modality-balanced, difficulty-aware multimodal active learning. RL-MBA models sample selection as a Markov Decision Process, where the policy adapts to modality contributions, uncertainty, and diversity, and the reward encourages accuracy gains and balance. Two key components drive this adaptability: (1) Adaptive Modality Contribution Balancing (AMCB), which dynamically adjusts modality weights via reinforcement feedback, and (2) Evidential Fusion for DifficultyAware Policy Adjustment (EFDA), which estimates sample difficulty via uncertainty-based evidential fusion to prioritize informative samples. Experiments on Food101, KineticsSound, and VGGSound demonstrate that RL-MBA consistently outperforms strong baselines, improving both classification accuracy and modality fairness under limited labeling budgets.

Abstract

Explore similar work

Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning

Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

Unsupervised Learning for Missing Modalities in Multimodal Learning