Adaptive Confidence Regularization for Multimodal Failure Detection

Moru Liu; Hao Dong; Olga Fink; Mario Trapp

cs.CVMar 2, 2026

Adaptive Confidence Regularization for Multimodal Failure Detection

Authors: Moru Liu, Hao Dong, Olga Fink, Mario Trapp

Organizations: Technical University of Munich · ETH Zürich · EPFL · Fraunhofer IKS

Abstract

The deployment of multimodal models in high-stakes domains, such as self-driving vehicles and medical diagnostics, demands not only strong predictive performance but also reliable mechanisms for detecting failures. In this work, we address the largely unexplored problem of failure detection in multimodal contexts. We propose Adaptive Confidence Regularization (ACR), a novel framework specifically designed to detect multimodal failures. Our approach is driven by a key observation: in most failure cases, the confidence of the multimodal prediction is significantly lower than that of at least one unimodal branch, a phenomenon we term confidence degradation. To mitigate this, we introduce an Adaptive Confidence Loss that penalizes such degradations during training. In addition, we propose Multimodal Feature Swapping, a novel outlier synthesis technique that generates challenging, failure-aware training examples. By training with these synthetic failures, ACR learns to more effectively recognize and reject uncertain predictions, thereby improving overall reliability. Extensive experiments across four datasets, three modalities, and multiple evaluation settings demonstrate that ACR achieves consistent and robust gains. The source code will be available at https://github.com/mona4399/ACR.

Figures & tables

Figure 1: (Left) Multimodal models substantially enhance FD performance compared to unimodal models, without the need for complex designs. (Right) Advanced OOD detection methods underperform on FD tasks, while the simple MSP baseline surprisingly remains the most effective.

Figure 2: Misclassified samples exhibit a significantly higher proportion of confidence degradation compared to correctly classified ones.

Figure 3: Our ACR framework integrates two principal components. The Adaptive Confidence Loss is designed to penalize the phenomenon of confidence degradation. The Multimodal Feature Swapping serves to generate challenging, failure-aware training instances. This process enables the model to learn to more effectively identify and reject uncertain samples.

Figure 4: Visualization on outliers generated by Multimodal Feature Swapping with different $n_{\text{swap}}$ (96, 128, 256, 512). Small swaps produce hard negatives that lie near the in-distribution manifold, while larger swaps create more distinct outliers further away.

	HMDB51				EPIC-Kitchens				HAC				Kinetics-600
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
MaxLogit	34.76	85.65	57.02	86.20	114.22	76.92	80.00	74.25	53.61	85.12	58.97	82.11	63.90	81.25	69.85	81.24
Energy	35.78	85.03	58.68	86.20	114.91	76.67	78.95	74.25	54.33	84.80	58.97	82.11	66.48	80.12	76.58	81.24
Entropy	30.24	87.87	57.85	86.20	114.09	76.83	78.42	74.25	42.63	89.46	61.54	82.11	47.30	86.85	65.22	81.24
MSP	29.56	88.28	52.07	86.20	115.03	76.52	76.84	74.25	42.90	89.27	66.67	82.11	46.29	87.33	61.29	81.24
DOCTOR	29.65	88.42	52.46	86.20	114.92	76.57	79.47	74.25	42.60	89.46	64.10	82.11	46.37	87.28	62.27	81.24

Table 1: FD performance on action recognition datasets with video and optical flow modalities.

Figure 5: FD under distribution shift on HAC dataset. The performance of five types of corruption on videos under the severity level of

5

is reported.

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
MSP	29.56	88.28	52.07	86.20
ACL	24.48	90.32	43.97	86.77
MFS	25.11	90.55	46.22	86.43
ACL + MFS	19.97	92.02	41.96	87.23

Table 2: Effect of each component.

	video+audio				optical flow+audio				video+optical flow+audio				Average
	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
MaxLogit	28.35	84.41	68.00	88.07	117.73	83.08	72.46	68.35	26.34	87.82	75.00	87.16	57.47	85.10	71.82	81.19
Energy	29.32	83.90	68.00	88.07	120.43	82.37	75.36	68.35	27.63	86.99	75.00	87.16	59.13	84.42	72.79	81.19
Entropy	25.10	86.03	64.00	88.07	116.95	82.75	66.67	68.35	21.40	91.00	57.14	87.16	54.48	86.59	62.60	81.19
MSP	26.79	86.30	57.69	88.07	118.04	82.46	71.01	68.35	20.99	91.35	42.86	87.16	55.27	86.70	57.19	81.19
DOCTOR	27.10	86.02	53.85	88.07	118.03	82.51	72.46	68.35	21.03	91.33	46.43	87.16	55.39	86.62	57.58	81.19

Table 3: FD performance on HAC dataset with different modality combinations.

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	Accuracy $\uparrow$
MSP	31.77	88.17	58.59	85.40
DOCTOR	31.76	88.18	59.38	85.40
Mixup	41.12	87.29	56.76	83.12
RegMixup	41.67	89.41	54.27	81.30
OpenMix	33.16	88.30	55.12	84.26
A2D	34.47	88.38	55.80	84.72

Table 4: Ablation on different architectures.

	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
MSP	94.14	39.35	86.20
Mixup	95.35	24.92	84.72
A2D	94.33	29.90	86.66
CRL	94.14	32.29	85.75
ACR (ours)	96.82	20.47	87.23

Table 5: Performance under both OOD and misclassified samples.

Methods	OOD Datasets										ID ACC $\uparrow$
	Kinetics-600		UCF101		EPIC-Kitchens		HAC		Average
	FPR95 $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	AUROC $\uparrow$
MSP	39.11	88.78	46.64	86.40	17.33	95.99	39.91	89.10	35.75	90.07	87.23
+AN	29.42	90.73	40.02	88.08	13.34	96.43	28.16	91.63	27.74	91.72	86.89
+Ours	27.71	92.81	34.89	89.10	12.20	97.76	21.78	94.25	24.15	93.48	87.23
Energy	32.95	92.48	44.93	87.95	8.10	97.70	32.95	92.28	29.73	92.60	87.23

Table 6: Multimodal OOD detection using video and optical flow, with HMDB51 as ID.

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
MSP	29.56	88.28	52.07	86.20
Random Noise	24.86	90.82	42.86	86.43
Random Drop	22.80	91.24	46.09	86.89
Feature Mixing	21.79	91.33	42.11	87.00
MFS	19.97	92.02	41.96	87.23

Table 7: Compare with other feature space augmentation methods.

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	mIoU $\uparrow$
MSP	33.90	79.97	55.49	59.25
ACR (ours)	21.90	84.51	52.51	63.56

Table 8: Results on SemanticKITTI for 3D semantic segmentation.

Figure 6: Score distribution for correct and wrong predictions. Our method leads to more clearly separated distributions, thereby enhancing the efficacy of failure detection.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: An illustration of the four main datasets we used for multimodal failure detection.

\widetilde{\mathbf{E}}^{1}[s_{1}{:}s_{1}{+}n_{\text{swap}}]\leftarrow\mathbf{E}^{2}[s_{2}{:}s_{2}{+}n_{\text{swap}}]

Appendix

Algorithm 1 Multimodal Feature Swapping

\widetilde{\mathbf{E}}^{1}[s_{1}{:}s_{1}{+}n_{\text{swap}}]\leftarrow\mathbf{E}^{3}[s_{3}{:}s_{3}{+}n_{\text{swap}}]

Appendix

Algorithm 2 Multimodal Feature Swapping with Three Modalities

	AURC $\downarrow$	AUROC $\uparrow$	FPR95 $\downarrow$	ACC $\uparrow$
MSP	30.24	86.63	62.75	86.03
ACR	15.38	87.79	58.06	91.51

Appendix

Table 11: Results on Office-Home dataset by fusing images from distinct domains.

Method	Correct Predictions ( $\uparrow$ )				Incorrect Predictions ( $\downarrow$ )
Method	HMDB51	EPIC	HAC	Kinetics	HMDB51	EPIC	HAC	Kinetics
w/o ACL	0.95	0.91	0.94	0.92	0.80	0.73	0.74	0.64
w/ ACL	0.96	0.93	0.96	0.93	0.63	0.69	0.62	0.57

Appendix

Table 12: Average fused confidence on correct and incorrect predictions.

Figure 10: An illustration of how MFS enables the detection and rejection of uncertain predictions, thereby improving model reliability. The failure-aware outliers generated by MFS are treated as uncertain because they introduce cross-modal inconsistency. Accordingly, their soft label $\mathbf{y}_{\text{swapped}}$ reduces the ground-truth class probability from $1$ to $1-\lambda$ , compared to $\mathbf{y}_{\text{true}}$ . Training with cross-entropy loss on $\mathbf{y}_{\text{swapped}}$ ensures that the prediction confidence on failure-aware outliers remains lower than on normal features, which is exactly the desired outcome.

Explore similar work

Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning
Mar 26, 2026Diyar Altinses, Andreas SchwungMultimodal Learning Multimodal Anomaly Detection
Department of Automation Technology and Learning Systems, South Westphalia University of Applied Sciences, Soest, Germany
Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification
Sep 23, 2026Longfei Huang, Xiangyu Wu, Yang YangModality Imbalance Multimodal Optimization
Nanjing University of Science and Technology · Alibaba Group
Adaptive Confidence-weighted Expansion for Trustworthy Multi-Omics Multimodal Fusion
Jul 22, 2026Mohammad Raahemi, Ali Sekhavati, Alireza Maleki +1Probability Calibration Uncertainty Quantification
Faculty of Engineering, University of Ottawa, Ottawa ON K1N 1A2, Canada · RIV Lab, Department of Computer Engineering, Bu-Ali Sina University, Hamedan, Iran · School of Computing and Communications, Lancaster University, Lancaster, UK

Mar 26, 2026J/K move · Enter open · S save

Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning

Diyar Altinses, Andreas Schwung

Department of Automation Technology and Learning Systems, South Westphalia University of Applied Sciences, Soest, Germany

Multimodal Learning Multimodal Anomaly Detection Reconstruction-Based Anomaly Detection Lipschitz Neural Networks

Open paper AskPDF arXiv

Modern multimodal systems deployed in industrial and safety-critical environments must remain reliable under partial sensor failures, signal degradation, or cross-modal inconsistencies. This work introduces a mathematically grounded framework for fault-tolerant multimodal representation learning that unifies self-supervised anomaly detection and error correction within a single architecture. Building upon a theoretical analysis of perturbation propagation, we derive Lipschitz- and Jacobian-based criteria that determine whether a neural operator amplifies or attenuates localized faults. Guided by this theory, we propose a two-stage self-supervised training scheme: pre-training a multimodal convolutional autoencoder on clean data to preserve localized anomaly signals in the latent space, and expanding it with a learnable compute block composed of dense layers for correction and contrastive objectives for anomaly identification. Furthermore, we introduce layer-specific Lipschitz modulation and gradient clipping as principled mechanisms to control sensitivity across detection and correction modules. Experimental results on multimodal fault datasets demonstrate that the proposed approach improves both anomaly detection accuracy and reconstruction under sensor corruption. Overall, this framework bridges the gap between analytical robustness guarantees and practical fault-tolerant multimodal learning.

Abstract

Explore similar work

Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning

Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification

Adaptive Confidence-weighted Expansion for Trustworthy Multi-Omics Multimodal Fusion