USCNet: Transformer-Based Multimodal Fusion with Segmentation Guidance for Urolithiasis Classification

Changmiao Wang; Songqi Zhang; Yongquan Zhang; Yifei Wang; Liya Liu; Nannan Li; Xingzhi Li; Jiexin Pan

cs.CVApr 8, 2026

USCNet: Transformer-Based Multimodal Fusion with Segmentation Guidance for Urolithiasis Classification

Authors: Changmiao Wang, Songqi Zhang, Yongquan Zhang, Yifei Wang, Liya Liu, Nannan Li, Xingzhi Li, Jiexin Pan, +4 more

Organizations: C. Wang and X. Wan are with Shenzhen Research Institute of Big Data, Shenzhen 518172, China · Y. Zhang, S. Zhang, and Y. Wang are with Zhejiang University of Finance and Economics, Hangzhou 310018, China · L. Liu is with Anhui University of Finance and Economics, Anhui 233000, China · N. Li is with the School of Computer Science and Engineering, Macau University of Science and Technology, Macao 999078, China · X. Li, J. Pan, and Y. Jiang are with The Second Affiliated Hospital of Chinese University of Hong Kong (Longgang District People’s Hospital of Shenzhen), Shenzhen 518172, China · H. Wang is with the School of Engineering and Energy and Harry Butler Institute, Murdoch University, Perth, WA6150, Australia · A. Elazab is with Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China

Abstract

Kidney stone disease ranks among the most prevalent conditions in urology, and understanding the composition of these stones is essential for creating personalized treatment plans and preventing recurrence. Current methods for analyzing kidney stones depend on postoperative specimens, which prevents rapid classification before surgery. To overcome this limitation, we introduce a new approach called the Urinary Stone Segmentation and Classification Network (USCNet). This innovative method allows for precise preoperative classification of kidney stones by integrating Computed Tomography (CT) images with clinical data from Electronic Health Records (EHR). USCNet employs a Transformer-based multimodal fusion framework with CT-EHR attention and segmentation-guided attention modules for accurate classification. Moreover, a dynamic loss function is introduced to effectively balance the dual objectives of segmentation and classification. Experiments on an in-house kidney stone dataset show that USCNet demonstrates outstanding performance across all evaluation metrics, with its classification efficacy significantly surpassing existing mainstream methods. This study presents a promising solution for the precise preoperative classification of kidney stones, offering substantial clinical benefits. The source code has been made publicly available: https://github.com/ZhangSongqi0506/KidneyStone.

Figures & tables

Fig. 1: The architecture of the USCNet: (a) the visual and textual transformation module, (b) the ViT-UNetSeg module, (c) the MSAF feature fusion module, (d) the classification module, (e) the Transformer encoder block, and (f) the cross-attention principle.

Fig. 2: ROC curve comparison of diagnostic accuracy between USCNet and established methods for infectious urolithiasis classification.

Panel of Figure 3 — Fig. 3: Confusion matrices comparison of USCNet versus multimodal baseline methods, quantitatively evaluating classification performance for infectious urolithiasis diagnosis.

Panel of Figure 4 — Fig. 4: Comparison of 3D multi-planar segmentation results (original images, ground truth annotations, nnUnet and USCNet outputs).

Loss Function Components

Classification

Segmentation

Clinical

CEA

SMA

Acc

\uparrow

(%)

\uparrow

(%)

Rec

\uparrow

(%)

Pre

\uparrow

(%)

AUC

\uparrow

(%)

Dice

\uparrow

(%)

IoU

\uparrow

(%)

HD95

\downarrow

(mm)

\checkmark

63.97

{\pm 3.38}

59.17

{\pm 3.26}

65.14

{\pm 4.37}

54.20

{\pm 5.25}

64.16

{\pm 3.82}

\checkmark

82.27

{\pm 1.02}

69.18

{\pm 1.25}

3.88

{\pm 0.51}

\checkmark

\checkmark

70.56

{\pm 4.88}

66.90

{\pm 5.62}

74.33

{\pm 5.27}

60.90

{\pm 6.15}

70.16

{\pm 4.84}

78.33

{\pm 1.73}

64.30

{\pm 1.73}

7.42

{\pm 0.87}

\checkmark

\checkmark

\checkmark

89.71

{\pm 1.86}

87.16

{\pm 2.27}

87.16

{\pm 1.69}

87.16

{\pm 3.39}

89.28

{\pm 1.79}

82.19

{\pm 0.93}

69.54

{\pm 0.93}

4.31

{\pm 0.30}

\checkmark

\checkmark

\checkmark

\checkmark

89.34

{\pm 1.38}

86.64

{\pm 1.52}

86.24

{\pm 2.06}

87.04

{\pm 1.78}

88.82

{\pm 1.40}

81.44

{\pm 0.95}

68.34

{\pm 0.95}

4.03

{\pm 0.43}

TABLE II: Ablation studies of the proposed dynamic loss function with different components.

Feature Comb.				Classification					Segmentation
$\mathbf{Z}_{3}$	$\mathbf{Z}_{6}$	$\mathbf{Z}_{9}$	$\mathbf{Z}_{12}$	Acc $\uparrow$ (%)	F1 $\uparrow$ (%)	Rec $\uparrow$ (%)	Pre $\uparrow$ (%)	AUC $\uparrow$ (%)	Dice $\uparrow$ (%)	IoU $\uparrow$ (%)	HD95 $\downarrow$ (mm)
-	-	-	$\checkmark$	91.91 $\pm$ 1.42	90.00 $\pm$ 1.74	90.83 $\pm$ 0.94	89.19 $\pm$ 3.13	91.73 $\pm$ 1.30	83.98 $\pm$ 1.09	72.12 $\pm$ 1.42	3.73 $\pm$ 0.82
-	-	$\checkmark$	-	90.44 $\pm$ 1.54	88.29 $\pm$ 1.91	89.91 $\pm$ 1.51	86.73 $\pm$ 3.22	90.35 $\pm$ 1.44	81.93 $\pm$ 0.99	69.49 $\pm$ 0.99	4.66 $\pm$ 0.56
-	$\checkmark$	-	-	90.07 $\pm$ 2.00	87.78 $\pm$ 2.34	88.99 $\pm$ 2.02	86.61 $\pm$ 3.50	89.89 $\pm$ 1.93	82.29 $\pm$ 0.96	69.71 $\pm$ 0.96	4.18 $\pm$ 0.49
$\checkmark$	-	-	-	87.87 $\pm$ 2.10	84.93 $\pm$ 2.43	85.32 $\pm$ 2.14	84.55 $\pm$ 3.02	87.45 $\pm$ 2.05	79.98 $\pm$ 1.07	66.67 $\pm$ 1.07	5.33 $\pm$ 0.25
$\checkmark$	$\checkmark$	-	-	89.71 $\pm$ 1.82	87.39 $\pm$ 2.13	88.99 $\pm$ 1.14	85.84 $\pm$ 3.75	89.59 $\pm$ 1.64	82.05 $\pm$ 1.02	69.47 $\pm$ 1.02	4.59 $\pm$ 0.43

TABLE III: Ablation studies of segmentation-assisted classification with different feature combinations.

Loss Function			Classification					Segmentation
$\mathcal{L}_{\text{BCE}}$	$\mathcal{L}_{\text{Focal}}$	$\mathcal{L}_{\text{Dice}}$	Acc $\uparrow$ (%)	F1 $\uparrow$ (%)	Rec $\uparrow$ (%)	Pre $\uparrow$ (%)	AUC $\uparrow$ (%)	Dice $\uparrow$ (%)	IoU $\uparrow$ (%)	HD95 $\downarrow$ (mm)
$\checkmark$	$-$	$\checkmark$	91.54 $\pm$ 1.55	89.50 $\pm$ 1.85	89.91 $\pm$ 1.72	89.09 $\pm$ 3.01	91.27 $\pm$ 1.49	82.43 $\pm$ 1.25	70.11 $\pm$ 1.58	4.19 $\pm$ 0.49
$-$	$\checkmark$	$\checkmark$	91.18 $\pm$ 1.20	88.68 $\pm$ 1.35	86.24 $\pm$ 2.38	91.26 $\pm$ 1.75	90.36 $\pm$ 1.30	83.02 $\pm$ 2.17	70.96 $\pm$ 2.32	4.02 $\pm$ 1.59
$\checkmark$	$\checkmark$	$\checkmark$	91.91 $\pm$ 1.42	90.00 $\pm$ 1.74	90.83 $\pm$ 0.94	89.19 $\pm$ 3.13	91.73 $\pm$ 1.30	83.98 $\pm$ 1.09	72.12 $\pm$ 1.42	3.73 $\pm$ 0.82

TABLE IV: Ablation studies of different loss functions for segmentation-assisted classification.

Class. / Seg.	Classification					Segmentation
Class. / Seg.	Acc $\uparrow$ (%)	F1 $\uparrow$ (%)	Rec $\uparrow$ (%)	Pre $\uparrow$ (%)	AUC $\uparrow$ (%)	Dice $\uparrow$ (%)	IoU $\uparrow$ (%)	HD95 $\downarrow$ (mm)
0.1 / 0.9	82.35 $\pm$ 2.23	77.14 $\pm$ 2.59	74.31 $\pm$ 2.66	80.20 $\pm$ 3.84	81.02 $\pm$ 2.11	78.86 $\pm$ 1.18	66.67 $\pm$ 1.18	5.66 $\pm$ 0.35
0.3 / 0.7	86.40 $\pm$ 1.39	83.41 $\pm$ 1.36	85.32 $\pm$ 1.65	81.58 $\pm$ 3.18	86.22 $\pm$ 1.14	81.96 $\pm$ 1.11	69.00 $\pm$ 1.11	4.38 $\pm$ 0.64
0.5 / 0.5	89.34 $\pm$ 1.38	86.64 $\pm$ 1.52	86.24 $\pm$ 2.06	87.04 $\pm$ 1.78	88.82 $\pm$ 1.40	81.44 $\pm$ 0.95	68.34 $\pm$ 0.95	4.03 $\pm$ 0.43
0.7 / 0.3	89.71 $\pm$ 1.90	87.27 $\pm$ 2.29	88.07 $\pm$ 1.46	86.49 $\pm$ 3.65	89.44 $\pm$ 1.78	82.04 $\pm$ 1.20	69.25 $\pm$ 1.20	4.18 $\pm$ 0.58
0.9 / 0.1	90.44 $\pm$ 1.67	88.39 $\pm$ 1.93	90.83 $\pm$ 1.21	86.09 $\pm$ 3.67	90.50 $\pm$ 1.47	82.11 $\pm$ 1.02	69.47 $\pm$ 1.02	4.78 $\pm$ 0.31

TABLE V: Effect of the adaptive loss functions on the classification and segmentation performances of the USCNet model.

Subgroup	Performance					N
Subgroup	Acc $\uparrow$ (%)	F1 $\uparrow$ (%)	Rec $\uparrow$ (%)	Pre $\uparrow$ (%)	AUC $\uparrow$ (%)	N
Age Groups
$<$ 40 years	91.80	87.80	94.74	81.82	96.99	61
40-60 years	90.11	88.00	86.84	89.19	94.09	91
$>$ 60 years	91.18	88.00	91.67	84.62	94.32	34
Gender

TABLE VI: Performance analysis of USCNet across different patient subgroups.

Fig. 5: Empirical validation of the dynamic loss threshold through systematic sensitivity analysis.

Explore similar work

XEns-CKD: An Explainable Ensemble-Based Approach for Chronic Kidney Disease Stage Detection
Aug 2, 2026Rehan Ahmad, Gousia Habib, Muhammad Shaban +1Medical Image Classification Clinical Prediction
SVKM’s, NMIMS, Mukesh Patel School of Technology Management and Engineering, Shirpur, Maharashtra, India. · *2FCAI Fellow University of Helsinki and ELLIS Institute Helsinki, Finland · COER University, Roorkee, India +1
Multi-Granularity 3D Kidney Lesion Characterization from CT Volumes
Jun 3, 2026Renjie Liang, Zhengkang Fan, Jinqian Pan +4Medical Image Analysis Detection Transformer
Department of Health Outcomes and Biomedical Informatics, University of Florida, 2004 Mowry Road, Gainesville, 32611, FL, USA · Department of Biostatistics and Health Data Science, Indiana University School of Medicine, 340 West 10th Street, Indianapolis, 46202, IN, USA · Center of Biomedical Informatics, 1101 W 10th St, Indianapolis, Indianapolis, 46202, IN, USA +1
A deep learning framework for glomeruli segmentation with boundary attention
Apr 15, 2026Behnaz Elhaminia, Catherine King, Jiaqi Lv +5Histopathology Image Segmentation Medical Image Segmentation
Tissue Image Analytics (TIA) Centre, Dept. of Computer Science, University of Warwick, UK · Dept. of Immunology and Immunotherapy, University of Birmingham, UK · Renal Unit, Queen Elizabeth Hospital Birmingham, UHB NHS Foundation Trust, UK +2

Aug 2, 2026J/K move · Enter open · S save

XEns-CKD: An Explainable Ensemble-Based Approach for Chronic Kidney Disease Stage Detection

Rehan Ahmad, Gousia Habib, Muhammad Shaban, Ishfaq Ahmad Malik

SVKM’s, NMIMS, Mukesh Patel School of Technology Management and Engineering, Shirpur, Maharashtra, India. · *2FCAI Fellow University of Helsinki and ELLIS Institute Helsinki, Finland · COER University, Roorkee, India+1

Medical Image Classification Clinical Prediction Explainable Medical Image Analysis

Open paper AskPDF arXiv

Chronic kidney disease (CKD) is a silent disease. Its progression may not significantly hamper a person's daily routine. Human kidney function can be classified as normal or as one of the five stages of CKD. Early detection of the CKD stage can help patients understand the functional status of their kidneys and follow medical advice to slow CKD progression. In this paper, we propose XEns-CKD, a novel ensemble vision transformer-based scheme for CKD stage classification using ultrasound images. Three ViTs were trained on a private ultrasound image dataset using different training parameters. The performance of each ViT was evaluated using macro sensitivity, macro specificity, macro precision, macro F1-score, macro Youden index, the Matthews correlation coefficient (MCC), and macro balanced accuracy. The ensemble model achieved an overall classification accuracy of 86.36%. This work also emphasizes identifying and interpreting kidney regions affected by CKD progression. Explainable artificial intelligence techniques, including LIME, LRP, Attention-Min, and Attention-Max, were used to improve model transparency and clinical trust. An attention map combining the Attention-Min and Attention-Max results effectively identified and interpreted kidney regions affected during CKD progression from one stage to another. The attention map also highlighted the effects of CKD progression in these regions. Compared with existing methods, the proposed method classified the five CKD stages and normal kidney status with a 4% improvement in accuracy.

Abstract

Explore similar work

XEns-CKD: An Explainable Ensemble-Based Approach for Chronic Kidney Disease Stage Detection

Multi-Granularity 3D Kidney Lesion Characterization from CT Volumes

A deep learning framework for glomeruli segmentation with boundary attention