Adaptive Bidirectional Task Interaction for Joint Segmentation and Classification of Breast Ultrasound

Abdullah Al Shafi; Md Kawsar Mahmud Khan Zunayed; Safin Ahmmed; Sk Imran Hossain; Engelbert Mephu Nguifo

cs.CVMar 1, 2026

Adaptive Bidirectional Task Interaction for Joint Segmentation and Classification of Breast Ultrasound

Authors: Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo

Organizations: Khulna University of Engineering and Technology, Bangladesh · University Clermont Auvergne, France

Abstract

Joint lesion segmentation and tissue classification in breast ultrasound are usually trained with a shared encoder, so the two branches stop exchanging information once their decoders separate. That is exactly where boundary detail and semantic evidence are most complementary. The proposed method restores this exchange during decoding and, because its value differs between images, lets the network decide per image how much to keep. A Task Interaction Module (TIM) at each of four decoder levels passes pooled boundary context into the classification representation and modulates decoder channels with class-conditioned priors. An Adaptive Interaction Weighting (AIW) unit then blends interacted and original features with a coefficient computed for each image and level. On BUSI the model reaches 74.19% IoU and 90.60% accuracy, and on BUSI-WHU 86.40% IoU and 95.00% accuracy, ahead of encoder-sharing multi-task, transformer segmentation and decoder-interaction baselines evaluated under the same protocol. The ablation shows that multi-scale context and cross-task exchange are not independent: applied separately they contribute 4.00 points of IoU in total, applied together 6.76. Adding the adaptive blend to task interaction alone raises AUC from 94.41% to 97.31%, indicating that the blend acts primarily on the classification branch. Code: https://github.com/C-loud-Nine/Adaptive-Task-Interaction-BUS.

Figures & tables

Figure 1: Overview of the proposed architecture. Multi-scale context fusion (MSCF) augments features inside every encoder stage and decoder block. TIM applies bidirectional task interaction at all four decoder levels ( $D_{1}$ – $D_{4}$ ); AIW blends the interacted and original features at each level. Dual heads produce the segmentation mask and the classification prediction. Arrow colour distinguishes the signal paths and the legend identifies each layer type.

CNN-Based
Method	BUSI (780 images)				BUSI-WHU (927 images)
	Seg		Clf		Seg		Clf
	IoU	Dice	Acc	F1	IoU	Dice	Acc	F1
U-Net [ 23 ] †	66.80	80.05	86.50	84.70	77.50	87.30	90.80	89.10
Attention U-Net [ 21 ] †	68.20	81.05	87.80	86.00	79.20	88.40	92.10	90.50
UNet++ [ 27 ] †	69.50	81.95	88.21	86.90	80.70	89.30	92.80	91.41

Table 1: Comparison on BUSI and BUSI-WHU. All models were trained and evaluated under the protocol of Section 3.1. IoU is foreground intersection over union. Transformer baselines are segmentation-only architectures and were not extended with classification heads, so their classification columns are empty. Best results in bold .

Figure 2: Qualitative segmentation results on BUSI [ 2 ] for three representative test cases. Columns show the input image, ground truth, and predictions from U-Net [ 23 ] , Att-U-Net [ 21 ] , UNet++ [ 27 ] , Swin-UNet [ 4 ] , MTI-Net [ 25 ] and the proposed method.

Components			Segmentation				Classification
MSCF	TIM	AIW	IoU	Dice	Sens	Prec ${}_{\text{s}}$	Acc	F1	AUC	Prec ${}_{\text{c}}$
–	–	–	67.43	80.55	75.06	79.73	84.62	82.33	94.90	82.16
✓	–	–	69.12	81.74	79.12	80.19	88.89	88.34	96.30	86.70
–	✓	–	68.94	81.80	79.83	79.76	86.32	84.00	94.41	86.04
–	✓	✓	69.74	82.17	83.07	79.32	88.89	88.34	97.31	87.24
✓	✓	✓	74.19	85.25	80.87	85.47	90.60	89.84	97.66	89.95

Table 2: Component ablation on BUSI. Baseline : EfficientNet-B4 encoder with a U-Net decoder, attention-gated skips and a pooled classification head, without MSCF, TIM or AIW. Each row adds the ticked components to that baseline, all other settings unchanged. Rows 3 and 4 differ only in AIW and therefore isolate its contribution. A row with AIW but without TIM is not included because AIW interpolates toward TIM’s output and reduces to the identity in its absence. Prec

{}_{\text{s}}

and Prec

{}_{\text{c}}

are segmentation and classification precision. Best results in bold .

Explore similar work

BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis
Jun 27, 2026Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed +2Semi-Supervised Medical Image Segmentation Medical Image Classification
Khulna University of Engineering & Technology, Bangladesh · University Clermont Auvergne, France
Specificity- and Calibration-Aware Breast Ultrasound Segmentation via Entropy-Guided Boundary Supervision
Jun 21, 2026Manar Alsaid, Mandip Shrestha, Mohammad AbbasLesion Segmentation False-Negative Rate
Department of Computer Science East Texas A&M University Commerce, USA · Texas Woman’s University Denton, USA
HADS-Net:A Hybrid Attention-Augmented Dual-Stream Network with Physics-Informed Augmentation for Breast Ultrasound Image Classification
May 19, 2026Chinedu Emmanuel Mbonu, Blessing Nwamaka Iduh, Joseph Ikechukwu Odo +1Clinical Ultrasound Dataset Medical Image Classification
Department of Computer Science, Nazarbayev University, Astana · Department of Computer Science, Nnamdi Azikiwe University Awka, Nigeria · Department of Computer Science, Peter University Achina-Onneh, Nigeria

Jun 27, 2026, cs.CVJ/K move · Enter open · S save

BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis

Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain+1

Khulna University of Engineering & Technology, Bangladesh · University Clermont Auvergne, France

Semi-Supervised Medical Image Segmentation Medical Image Classification U-Net Cross-Task Interference Functional Magnetic Resonance Imaging

Open paper AskPDF arXiv

Jointly learning to segment and classify medical images demands cross-task synergy, yet encoder-sharing architectures limit decoder reconstruction to task-private representations, permanently discarding the boundary cues and semantic priors each branch could supply to the other. This work introduces BTI-Net, which establishes bidirectional communication at every decoder level through two parallel pathways via Task Interaction Modules (TIM). Spatial boundary context is gated into the classification branch, while global semantic priors multiplicatively modulate the decoder, with refined features propagating progressively from coarse semantics to fine boundary detail across all four decoder resolutions. Since cross-task interaction is not equally reliable for every input, Uncertainty Proxy Attention (UPA) gates each TIM output per instance and per level using three signals that capture cross-task alignment, scene complexity, and prediction confidence, without external annotations or additional inference passes. Experiments on three medical benchmarks spanning ultrasound, dermoscopy, and brain MRI demonstrate consistent improvements in segmentation IoU and classification accuracy over both encoder-sharing and decoder-interaction baselines. Ablation confirms adaptive gating contributes +2.36 IoU over fixed bidirectional interaction, and classification accuracy improves by up to +2.26 points over the strongest multi-task baseline. UPA's uncertainty proxies serve as reliable single-pass task-failure signals without the overhead of stochastic sampling. Code: https://github.com/C-loud-Nine/BTI-Net_MTL

Abstract

Explore similar work

BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis

Specificity- and Calibration-Aware Breast Ultrasound Segmentation via Entropy-Guided Boundary Supervision

HADS-Net:A Hybrid Attention-Augmented Dual-Stream Network with Physics-Informed Augmentation for Breast Ultrasound Image Classification