Reliability-Guided Vision-Language Decoding with Text-Aware Skip Refinement for Medical Image Segmentation
Authors: Sivan Talaei, Fatemeh Daneshfar, Abdulhady Abas Abdullah, Mourad Oussalah
Organizations: Department of Computer Engineering, University of Kurdistan, Sanandaj, Kurdistan, Iran · Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Kurdistan, Iraq · Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland
Medical image segmentation remains challenging when targets exhibit limited visual distinction, irregular structure, or substantial variation across imaging modalities. Medical text can reduce this ambiguity, but indiscriminate fusion may introduce unreliable guidance. We present RELiF, a vision--language framework that integrates text through reliability-guided decoding. Its Reliability-guided Fusion Gate (RFG) adaptively controls cross-modal updates, while the Text-guided Skip Refinement Adapter (TSRA) refines high-resolution features using text guidance. RELiF is evaluated on radiographic, CT, and endoscopic segmentation tasks, where it shows competitive performance across diverse imaging characteristics and target appearances. Additional analyses examine efficiency and stability. The complementary uncertainty representation derived from the reliability response also shows an association with prediction difficulty, with higher uncertainty values occurring more frequently in pixels associated with segmentation errors. Code: https://github.com/SeivanTalaie/RELiF.
Figures & tables
Figure 1: RFG behavior under controlled image and text degradation on QaTa-COV19. Lines show the mean gate response across the test set and three RFG stages, while bars indicate Dice gains (pp) over ungated fusion.
Figure 2: Overview of RELiF: (a) vision–language segmentation architecture. (b) Text-guided Skip Refinement Adapter (TSRA). (c) Reliability-guided Decoding Attention Block (RDAB) with its Reliability-guided Fusion Gate (RFG).
Method
QaTa-COV19
MosMedData+
Kvasir-SEG
Dice(%)
mIoU(%)
Dice(%)
mIoU(%)
Dice(%)
mIoU(%)
Monomodal models
U-Net (’15) [ 12 ]
78.91
69.32
64.30
50.50
82.33
74.26
nnUNet (’21) [ 5 ]
80.30
70.62
72.32
60.14
84.22
75.41
UCTransNet (’22) [ 14 ]
79.00
69.34
65.71
52.55
91.04
87.31
Multimodal models
Table 1: Comparison with state-of-the-art methods. best and second-best results are bold and underlined , respectively.
Figure 3: Qualitative segmentation results across three datasets.
Method
Regime
QaTa-COV19
MosMedData+
Kvasir-SEG
Dice(%)
mIoU(%)
Dice(%)
mIoU(%)
Dice(%)
mIoU(%)
BiCLIP [ 13 ]
50%
89.83
81.53
79.72
66.27
86.55
76.29
10%
87.16
77.25
69.23
52.95
71.72
55.91
LViT [ 7 ]
50%
78.81
69.55
58.98
47.28
80.66
73.30
10%
74.72
64.99
51.07
39.59
55.68
44.38
RELiF
50%
91.10
83.66
78.06
64.01
91.95
85.10
Table 2: Segmentation performance with 50% and 10% of the training data across three datasets.
Dataset
Error prevalence(%)
AUROC ↑
AUPRC ↑
Random AUPRC
AUPRC lift ↑
QaTa-COV19
2.03
0.7724
0.0824
0.0203
4.06 ×
MosMedData+
0.65
0.8389
0.0199
0.0065
3.04 ×
Kvasir-SEG
1.88
0.8473
0.1035
0.0188
5.51 ×
Table 3: Uncertainty-based pixel error detection using RFG-derived uncertainty response. Error prevalence defines the random AUPRC baseline, while AUPRC lift measures gains.
Method
MosMedData+
Kvasir-SEG
Dice(%)
mIoU(%)
Dice(%)
mIoU(%)
Component Ablation
Base
78.48
64.58
92.37
85.82
Base + RFG
78.92
65.18
92.89
86.72
Base + TSRA
79.06
65.37
92.91
86.76
Full RELiF (RFG + TSRA)
79.27
65.66
93.28
87.41
Table 4: Ablation study of RELiF evaluates individual components and textual guidance strategies using Dice and mIoU.
School of Software Technology & DUT-RU International School of Information Science and Engineering, Dalian University of Technology, Dalian, China · School of Future Technology, Dalian University of Technology, Dalian, China · Central Hospital of Dalian University of Technology, Dalian, China +2