Uncertainty-Aware Vision-Language Segmentation for Medical Imaging

Aryan Das; Tanishq Rachamalla; Koushik Biswas; Swalpa Kumar Roy; Vinay Kumar Verma

cs.CVFeb 16, 2026

Uncertainty-Aware Vision-Language Segmentation for Medical Imaging

Authors: Aryan Das, Tanishq Rachamalla, Koushik Biswas, Swalpa Kumar Roy, Vinay Kumar Verma

Organizations: VIT Bhopal · SAHE, Andhra Pradesh · IIIT Delhi · Tezpur University Assam · IIT Kanpur

Abstract

We introduce a novel uncertainty-aware multimodal segmentation framework that leverages both radiological images and associated clinical text for precise medical diagnosis. We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion and long-range dependency modelling. To guide learning under ambiguity, we propose the Spectral-Entropic Uncertainty (SEU) Loss, which jointly captures spatial overlap, spectral consistency, and predictive uncertainty in a unified objective. In complex clinical circumstances with poor image quality, this formulation improves model reliability. Extensive experiments on various publicly available medical datasets, QATA-COVID19, MosMed++, and Kvasir-SEG, demonstrate that our method achieves superior segmentation performance while being significantly more computationally efficient than existing State-of-the-Art (SoTA) approaches. Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks. Code: https://github.com/arya-domain/UA-VLS

Figures & tables

Figure 1: Overview of the proposed architecture. The model integrates visual and frozen text encoders and the Modality Decoding Attention Block (MoDAB), which incorporates Self-Attention and Cross-Attention along with a State Space Mixer (SSMix) for efficient multimodal fusion. The decoder reconstructs segmentation masks from the fused features through a multi-stage upsampling pathway.

Figure 2: Comparison of Grad-CAM-Based Attention Visualizations Between the Proposed Model and Baseline methods

Figure 3: Qualitative Comparison of Predicted Segmentation Maps with Baseline Models

Modality	Method	Trainable Params (M)	Flops (G)	QATA-COV19		MosMedData++		Kvasir-Seg
Modality	Method	Trainable Params (M)	Flops (G)	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)
MonoModels	U-Net [ 1 ]	14.8	50.3	78.91	69.32	64.30	50.50	82.33	74.26
	UNet++ [ 3 ]	74.5	94.6	79.47	70.05	71.63	58.14	82.79	73.94
	AttUNet [ 2 ]	34.9	101.9	79.11	69.83	66.07	52.68	82.94	74.17
	nnUNet [ 9 ]	19.1	412.7	80.30	70.62	72.32	60.14	84.22	75.41
	TransUNet [ 10 ]	105	56.7	78.44	68.84	71.13	58.28	90.53	85.94

Table 1: Comparison of Monomodal and Multimodal State-of-The-Art (SoTA) methods on medical image segmentation across three datasets: QaTa-COV19, MosMed++, and Kvasir-SEG. Metrics include Dice score (%), mean Intersection over Union (mIoU, %), number of Trainable Parameters (Millions), and Floating-Point Operations (FLOPs) per second (Billions). Black : best, Green : second best, Blue : third best values.

Method	Dice (%)	mIoU (%)
Loss Function
Dice loss	93.44	85.76
BCE loss	92.03	85.26
Textual Guidance
Inference w/o Text Prompts	87.28	81.32
Training w/o MoDAB	85.15	73.86

Table 2: Ablation results on the Kvasir-SEG dataset. Each condition evaluates the model after removing or replacing a key component. Metrics reported are Dice score (%) and mIoU (%).

Explore similar work

DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation
Aug 6, 2026Jiaxuan Li, Qing Xu, Xiangjian He +4Cross-Modality Medical Image Segmentation Medical VLMs
School of Computer Science, University of Nottingham Ningbo China, Ningbo, Zhejiang, China · School of Computer Science, University of Nottingham, UK
MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Feb 23, 2026Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari +3Medical Image Segmentation Text-Guided Image Segmentation
Concordia University, Montreal, Canada
BiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation
Feb 25, 2026Saivan Talaei, Fatemeh Daneshfar, Abdulhady Abas Abdullah +1Multimodal Robustness Medical VLMs
Department of Computer Engineering, University of Kurdistan, Sanandaj, Kurdistan, Iran · Artificial Intelligence and Innovation Center, University of Kurdistan, Erbil, Kurdistan, Iraq · Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland

Aug 6, 2026J/K move · Enter open · S save

DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation

Jiaxuan Li, Qing Xu, Xiangjian He, Yue Li+3

School of Computer Science, University of Nottingham Ningbo China, Ningbo, Zhejiang, China · School of Computer Science, University of Nottingham, UK

Cross-Modality Medical Image Segmentation Medical VLMs Medical Image Segmentation Vision-Language Alignment

Open paper AskPDF arXiv

Cross-modal alignment of visual and textual representations is fundamental to multimodal medical image understanding, yet remains hindered by uncertainty in both modalities under real-world clinical conditions. Existing vision-language segmentation methods rely on deterministic cross-modal matching, which overlooks aleatoric uncertainty from ambiguous boundaries and epistemic uncertainty from limited training data, leading to fragile performance under domain shift. To address this issue, we propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) upon frozen encoders to explicitly model representational uncertainty. Specifically, the PCM-Adapter comprises two sequential modules for progressive probabilistic alignment. We first devise a Mahalanobis Alignment Module (MAM) that models textual tokens as Gaussian distributions and computes patch-text compatibility via Mahalanobis distance, yielding variance-conditioned matching that downweights unreliable feature dimensions. Moreover, we devise a Distribution Flow Module (DFM) that estimates modality-wise confidence parameters and performs vision-guided refinement of textual distributions, accommodating distributional variation across imaging modalities. Extensive experiments across eight medical segmentation benchmarks demonstrate that DistMedVL outperforms state-of-the-art methods with only 6.3M trainable parameters, exhibiting superior data efficiency, perturbation robustness and cross-dataset generalization.

Abstract

Explore similar work

DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

BiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation