DBMF: A Dual-Branch Multimodal Framework for Out-of-Distribution Detection

Jiangbei Yue; Darren Treanor; Venkataraman Subramanian; Sharib Ali

cs.CVApr 9, 2026

DBMF: A Dual-Branch Multimodal Framework for Out-of-Distribution Detection

Authors: Jiangbei Yue, Darren Treanor, Venkataraman Subramanian, Sharib Ali

Organizations: University of Leeds, LS2 9JT, UK

Abstract

The complex and dynamic real-world clinical environment demands reliable deep learning (DL) systems. Out-of-distribution (OOD) detection plays a critical role in enhancing the reliability and generalizability of DL models when encountering data that deviate from the training distribution, such as unseen disease cases. However, existing OOD detection methods typically rely either on a single visual modality or solely on image-text matching, failing to fully leverage multimodal information. To overcome the challenge, we propose a novel dual-branch multimodal framework by introducing a text-image branch and a vision branch. Our framework fully exploits multimodal representations to identify OOD samples through these two complementary branches. After training, we compute scores from the text-image branch ( $S_t$ ) and vision branch ( $S_v$ ), and integrate them to obtain the final OOD score $S$ that is compared with a threshold for OOD detection. Comprehensive experiments on publicly available endoscopic image datasets demonstrate that our proposed framework is robust across diverse backbones and improves state-of-the-art performance in OOD detection by up to 24.84%

Figures & tables

Figure 1: The pipeline of the OOD detection. $\gamma$ is a threshold.

Figure 2: The overview of the DBMF. Our framework consists of the training phase, inference phase, and OOD detection. After training two branches, we compute $S_{t}$ and $S_{t}$ , resulting in the OOD score $S$ . The final detection is based on $S$ and a threshold $\gamma$ .

Backbone	ResNet18				DeiT
Dataset	Kvasir-v2		GastroVision		Kvasir-v2		GastroVision
Metric	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95
MSP	90.30	41.72	66.93	90.56	87.05	40.18	70.00	90.74
ODIN	91.77	35.44	69.79	79.27	88.41	36.40	73.37	83.68
Mahalanobis	84.05	54.06	65.93	89.69	94.50	21.86	75.68	81.43
Energy	88.85	52.36	70.31	79.79	85.77	44.02	75.35	83.68

Table 1: The comparison of experimental results on two datasets across different backbones and metrics. We show the best results in bold and underline the second-best results. Higher AUROC or lower FPR95 indicates better OOD detection performance.

Figure 3: Qualitative comparison of OOD score distributions on Kvasir-v2 and GastroVision between NERO and our framework. OOD scores are plotted along the horizontal axis, while the vertical axis shows the corresponding probability density.

Backbone	ResNet18				DeiT
Dataset	Kvasir-v2		GastroVision		Kvasir-v2		GastroVision
Metric	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95
Text-image	88.11	42.60	78.97	77.72	92.28	34.52	82.12	63.96
Vision	91.50	30.40	74.31	76.27	92.09	22.86	82.51	60.51
DBMF	92.11	25.70	81.51	67.60	94.55	18.40	85.84	51.90

Table 2: Ablation study on two datasets. The best results are in bold.

OOD Detection

Out-of-distribution (OOD) detection identifies inputs unlike a model’s training data so systems can flag uncertainty, reject inputs, or adapt safely.

See all 222 papers

Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition
Jun 23, 2026Lars Doorenbos, Duc Manh Vu, Serdar Ozsoy +1OOD Detection Multimodal Learning
University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Training-Free Out-of-Distribution Detection for Pathology Whole-Slide Images
Aug 2, 2026Sabri Mustafa Kahya, Richard R. Chen, Muhammet Sami Yavuz +4Computational Pathology OOD Detection
Prototype Fusion: A Training-Free Multi-Layer Approach to OOD Detection
Mar 24, 2026Shreen Gul, Mohamed Elmahallawy, Ardhendu Tripathy +1OOD Detection
Missouri University of Science and Technology, Rolla, MO 65401, USA · Washington State University, Richland, WA 99354, USA

Jun 23, 2026J/K move · Enter open · S save

Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition

Lars Doorenbos, Duc Manh Vu, Serdar Ozsoy, Juergen Gall

University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany

OOD Detection Multimodal Learning

Open paper AskPDF arXiv

The incorporation of additional modalities into action recognition models increases their performance across a wide range of settings. However, how this additional information can contribute to making the models more robust remains underexplored, particularly for the case of multi-modal out-of-distribution (OOD) detection. While methods exist that regularize the multi-modal training process with OOD detection in mind, they still apply off-the-shelf OOD detectors designed for the uni-modal case during inference, discarding important information. Based on an interesting relationship we find between the multi-modal and uni-modal predictions, we propose to use this signal to build a post-hoc detector explicitly designed for the multi-modal scenario. We combine this new source of information with a feature-space score, which detects off-manifold samples in the multi-modal space, and normalize them by the multi-modal logits. In doing so, the proposed hybrid detector is compatible with existing training-time approaches and consistently improves performance. Experiments on a wide range of established datasets from the MultiOOD benchmark show that, on average, our approach outperforms the state of the art. Our results show the importance of explicitly considering the different modalities at inference time for multi-modal OOD detection.

Abstract

OOD Detection

Explore similar work

Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition

Training-Free Out-of-Distribution Detection for Pathology Whole-Slide Images

Prototype Fusion: A Training-Free Multi-Layer Approach to OOD Detection