Dynamic Alignment and Calibration for Multimodal Learning

Jinghao Xu; Zhenhua Guo; Xiaofeng Zhu; Xiaoshuang Shi

cs.CVOct 6, 2026

Dynamic Alignment and Calibration for Multimodal Learning

Authors: Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu, Xiaoshuang Shi

Organizations: School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu,China · Tianyijiaotong Technology Ltd.,China · School of Computer Science and Technology, Hainan University, Haikou, China

Abstract

Dynamic multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existing methods still suffer from two limitations: (i) static cross-modal alignment strategies usually impose uniform constraints on all samples while overlooking sample-wise variations, potentially leading to unreasonable over-alignment; and (ii) confidence- or uncertainty-aware fusion methods often fail to adequately account for feature magnitude and confidence differences across modalities. For modality pairs with significant feature magnitude differences or small confidence gaps, it might be unreliable to strictly align fusion weights according to confidence. To address these issues, we propose an Alignment- and Calibration-driven Multimodal Learning framework (ACML). Specifically, ACML incorporates a dynamic cross-modal triplet alignment module, which enforces strong semantic consistency for high-confidence positive pairs while encouraging diverse representation learning between high- and low-confidence positive pairs according to their confidence gaps. Additionally, ACML introduces a difference-aware attention calibration strategy that adaptively adjusts attention regularization based on feature magnitude and confidence differences across modalities, thereby mitigating biases caused by unreasonable fusion constraints. Extensive experiments on multiple multimodal benchmark datasets demonstrate that ACML consistently achieves superior performance and robustness over recent state-of-the-art methods.

Figures & tables

Figure 1: Overview of the proposed framework ACML. In $\mathcal{L}_{\text{DCTA}}$ , $\triangle$ and $\square$ denote different categories, colors represent modalities, and color intensity reflects confidence.

\toprule \multirow 2*Dataset	\multirow 2*Method	$\epsilon=\mathbf{0.0}$		$\epsilon=\mathbf{5.0}$		$\epsilon=\mathbf{10.0}$
		Avg±Std	Worst	Avg±Std	Worst	Avg±Std	Worst
\midrule \multirow 10* \makecell NYU Depth V2	Depth	61.97 ±0.99	60.40	49.65 ±3.22	43.27	42.42 ±2.42	36.85
	RGB	62.57 ±0.82	61.32	52.56 ±2.28	47.55	46.16 ±3.10	38.84
	LateFusion	69.22 ±0.98	67.43	56.35 ±2.84	51.22	48.65 ±3.40	42.36
	Concat	68.70 ±0.67	67.89	55.76 ±2.23	49.54	48.17 ±3.36	40.21
	TMC	71.42 ±0.84	70.03	61.44 ±2.19	56.58	53.36 ±3.36	45.87

Table 1: Classification comparison when 50% of the modalities are corrupted with Gaussian noise, i.e., zero mean with variance of

\epsilon

. We highlight the optimal results using bold text and indicate the suboptimal results with underlining for clarity.

\toprule \multirow 2*Configuration	$\epsilon=\mathbf{0.0}$		$\epsilon=\mathbf{5.0}$		$\epsilon=\mathbf{10.0}$
\cmidrule (lr)2-3 \cmidrule (lr)4-5 \cmidrule (lr)6-7	Avg±Std	Worst	Avg±Std	Worst	Avg±Std	Worst
\midrule Baseline	70.87 ±0.79	69.27	62.61 ±1.53	58.87	55.02 ±3.38	46.79
Baseline + $\mathcal{L}_{\text{DCTA}}$	72.37 ±0.80	70.80	63.82 ±1.97	58.56	55.70 ±2.91	48.62
Baseline + $\mathcal{L}_{\text{DAC}}$	71.67 ±0.82	70.32	63.66 ±2.58	58.42	53.93 ±4.33	47.59
\rowcolor gray!20 ACML	72.72 ±0.71	71.56	65.43 ±1.63	60.86	58.11 ±2.69	49.69
\midrule ACML ( $\mathcal{L}_{\text{DCTA}}\rightarrow\mathcal{L}_{\text{SC}}$ )	72.48 ±0.69	71.10	65.06 ±1.74	60.86	57.09 ±2.75	51.07

Table 2: Ablation studies on NYU Depth V2. We report both the average performance and the worst-case performance under different noise levels.

Figure 2: 2D

t

-SNE visualization of multimodal features.

\triangle

indicates the image modality, and

\square

denotes the text modality. Each color corresponds to a distinct class, with darker shades indicating higher confidence.

Figure 3: Performance sensitivity to hyperparameters

\lambda_{1}

and

\lambda_{2}

across two datasets.

\toprule \multirow 2* Missing rate $\eta$	Train/Test		Validation Set				Testing Set
\cmidrule (lr)2-3 \cmidrule (lr)4-7 \cmidrule (lr)8-11	Image	Text	DCP	(+ACML)	SyP	(+ACML)	DCP	(+ACML)	SyP	(+ACML)
\midrule \multirow 3*50%	100%	50%	82.33	84.72	82.64	83.68	82.11	84.42	81.93	83.85
	50%	100%	89.23	89.44	89.07	89.16	89.12	89.44	89.18	89.39
	75%	75%	85.25	85.86	85.20	86.28	85.24	86.03	86.17	86.50
\multirow 3*70%	100%	30%	79.18	80.61	79.39	80.21	78.87	80.39	79.28	80.27
	30%	100%	87.53	88.11	87.62	87.87	87.32	87.77	87.27	87.75

Table 3: Accuracy comparison on FOOD101 validation and test sets under different missing rates (baselines vs. +ACML)

\toprule \multirow 2*Method	BRCA			ROSMAP
\cmidrule (lr)2-4 \cmidrule (lr)5-7	Accuracy	Weighted F 1	Macro F 1	Accuracy	F 1	AUC
\midrule MOGONet	82.9 ±1.8	82.5 ±1.7	77.4 ±1.7	81.5 ±2.3	82.1 ±1.2	87.4 ±1.2
MMDyn	87.7 ±0.3	88.0 ±0.5	84.5 ±0.5	84.2 ±1.3	84.6 ±0.7	91.2 ±0.7
MLCLNet	86.4 ±1.6	87.8 ±1.7	82.6 ±1.8	84.4 ±1.5	85.2 ±1.5	89.3 ±1.1
DMIB	86.0 ±1.6	86.0 ±0.8	81.6 ±0.9	84.9 ±1.8	85.3 ±1.7	91.6 ±0.7
GCFANet	88.6 ±1.5	88.9 ±1.6	85.3 ±1.6	86.3 ±1.4	88.3 ±1.6	91.5 ±1.2

Table 4: Classification results (%) on the BRCA and ROSMAP datasets.

Explore similar work

Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification
Sep 23, 2026Longfei Huang, Xiangyu Wu, Yang YangMultimodal Learning Multimodal Classification
Nanjing University of Science and Technology · Alibaba Group
When to Align, When to Predict: A Phase Diagram for Multimodal Learning
Jun 9, 2026Ilay Kamai, Hugues Van Assel, Aviv Regev +2Multimodal Learning Multimodal Representations
Technion · Genentech · Brown University
Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals
Jun 1, 2026Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang +2Multimodal Fusion Multimodal Representations
Adelaide University Adelaide, Australia · Shandong University Jinan, China

Sep 23, 2026, cs.LGJ/K move · Enter open · S save

Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification

Longfei Huang, Xiangyu Wu, Yang Yang

Nanjing University of Science and Technology · Alibaba Group

Multimodal Learning Multimodal Classification Cross-Modal Information Asymmetry

Open paper AskPDF arXiv

Multimodal learning (MML) falls into the optimization dilemma due to the modality imbalance phenomenon, leading to suboptimal overall performance in practice. While many attempts primarily focus on balancing the optimization dynamics across modalities to address this issue, we identify a subtle yet critical flaw: optimization yields asymmetric gains in predictive certainty, with the strong modality more confident than the weak one, driving imbalanced modality contributions. In this paper, our analysis reveals that this flaw stems from unimodal characteristics rather than multimodal learning, and this confidence discrepancy can be corrected by positive cross-modal intervention. Based on this insight, we propose multimodal Max Confidence Regularization (MaxCR) to dynamically intervene in modality semantic confidence. Specifically, the semantic confidence of each modality is tracked using a nonlinear sparsity measure. We then design max suppression and max excitation based on this measure to regularize strong and weak modalities, respectively. They penalize and encourage the top-1 confidence, thereby constraining multimodal prediction. To this end, strong and weak modalities are expected to make calibrated confidence, thereby improving the overall performance. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines.

Abstract

Explore similar work

Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification

When to Align, When to Predict: A Phase Diagram for Multimodal Learning

Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals