cs.CVOct 12, 2026
SaveA Concept-informed Interpretable Artificial Intelligence Framework for Breast Ultrasound Diagnosis
Organizations: School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong, 510515, China. · Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, Guangdong, 510515, China. · Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China. · Department of Ultrasound, Nanfang Hospital, Southern Medical University, Guangzhou, Guangdong, 510515, China. · Department of Radiology, Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou 510080, China. · Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou 510080, China. · Department of Ultrasonic Diagnosis, Zhujiang Hospital, Southern Medical University, Guangzhou, Guangdong, 510282, China. · Department of Radiology, Zhujiang Hospital, Southern Medical University, Guangzhou, Guangdong, China
Abstract
Most existing artificial intelligence systems for breast ultrasound rely primarily on image-driven prediction, with limited incorporation of the knowledge from the Breast Imaging Reporting and Data System (BI-RADS) and the intermediate assessment processes underlying radiologists' workflow, thereby constraining their interpretability and alignment with clinical practice. We curated, to our knowledge, the largest multicenter collection of breast ultrasound images paired with structured BI-RADS descriptors, comprising 9,941 patients across 20 independent datasets. Across 14 testing datasets, IBUS demonstrated consistent generalization performance across different clinical centers and imaging conditions. Incorporating BI-RADS descriptors enabled fine-grained risk stratification, while cross-checking predicted BI-RADS grades against malignancy probabilities identified internally discordant predictions that may warrant additional clinical review. In a reader study involving eight radiologists, IBUS assistance increased diagnostic sensitivity among both junior and senior readers, with the potential of guiding radiologists to make targeted revision in risk stratification. Overall, IBUS exhibits its potential as a second-stage decision-support system that complements radiologists by providing additional risk estimates and interpretable evidence for diagnostically challenging cases. The demo video of IBUS platform is publicly available at https://github.com/zsz1997/IBUS.
Figures & tables
Figure 1: Overview of the dataset and study design. a , Workflow of clinical diagnosis under BI-RADS guideline. The workflow mainly consists of image acquisition, lesion localization, BI-RADS assessment, and clinical decision. b , Architecture of the proposed IBUS framework. IBUS mainly consists of four key components, including a dual-stream feature extractor, a segmentation head, a concept-informed representation coupling module and a classification head. c , Data statistics for all, training and testing data. The study cohort comprised 20 independent datasets (i.e., D1D20) from multiple centers, in which 6 datasets (i.e., D1D6) and 14 datasets (i.e., D7~D20) were used to develop and evaluate IBUS. d , Annotation of structured BI-RADS descriptors for ultrasound images. e , Performance evaluation on multicenter datasets. f , Perturbation analysis on different BI-RADS descriptors. g , Semantic-visual alignment and visualization analysis on model’s attention. h , Risk stratification based on sample similarity retrieval. i , Reader study for assessing IBUS’s capability in assisting clinicians evaluation.
Figure 2: Model generalization across multicenter datasets. a , Radar plot for diagnosis metrics across six competing methods on all (D-All) and each (D7-D20) test sets, with six metrics of AUC, ACC, SEN, SPE, PPV, and NPV compared. b , ROC curves on diagnostically challenging cases (BI-RADS 3 and 4 lesions) across Top-2 models (i.e., MultiTask and USFM) on five datasets (i.e., D10, D17~D20). c , Ablation study on three variants: (1) IBUS w/o Seg: removing the segmentation head; (2) IBUS w/o BI-RADS: removing the stream tailored for BI-RADS descriptors; (3) IBUS w/o GMM: removing risk stratification modeling.
Figure 3: Perturbation analysis of BI-RADS descriptors. a - c , Comparison of prediction performance across IBUS and its variants after removing individual BI-RADS descriptor groups, with three metrics of AUC, SEN, and SPE. d - i , Distribution of descriptors among misclassified samples from IBUS w/o OR ( d ), IBUS w/o SH ( e ), IBUS w/o CA ( f ), IBUS w/o PF ( g ), IBUS w/o EC ( h ) and IBUS w/o MA ( i ), respectively. “MAL” and “BEN” denote malignant and benign, respectively. j , Four representative misclassified cases from IBUS w/o MA.
Figure 4: Semantic–visual alignment and visualization analysis. a - c , Semantic–visual alignment and cross-attention visualization on one representative malignant case. a , Visualization of attention weights across different BI-RADS descriptors. b , cross-attention maps across important BI-RADS descriptors. c , The value change of malignancy probability after removing different BI-RADS descriptors. d - f , Semantic–visual alignment and cross-attention visualization on one representative benign case. d , Visualization of attention weights across different BI-RADS descriptors. e , Cross-attention maps across important BI-RADS descriptors. f , The value change of malignancy probability after removing different BI-RADS descriptors. g , Class activation maps (CAM) visualization from IBUS and IBUS w/o BI-RADS, respectively.
Figure 5: BI-RADS risk stratification performance. a , Process of risk stratification evaluation. b , T-SNE maps visualization of features from IBUS and IBUS w/o BI-RADS. c , Top- confusion matrices for BI-RADS risk stratification between reference BI-RADS and IBUS-predicted BI-RADS. The retrieval accuracy (i.e., ACC) and deviation (i.e., DEV) score for each BI-RADS grade are displayed in the bar chart on the right side of the confusion matrices. d , Proportion of benign and malignant cases in six categories derived from reference BI-RADS grades and Top- ~Top- IBUS-predicted BI-RADS grades. e , Comparison of IBUS’s diagnostic performance on BI-RADS 2 and 5 based on reference BI-RADS grades, Top- IBUS-predicted BI-RADS grades, and selective Top- IBUS-predicted BI-RADS grades, respectively. “n/a” means no corresponding class can be used to calculate the metric.
Figure 6: AI-assisted diagnosis and risk stratification performance in breast lesion assessment. a , The flowchart of reader study. b , ROC curve from IBUS, all readers (i.e., R1R8), junior readers (i.e., R1R4), senior readers (i.e., R5~R8) and individual readers without IBUS assistance. c , ROC curve from IBUS, all readers, junior readers, senior readers and individual readers without and with IBUS assistance. d - e , Comparison of all readers, junior readers and senior readers without and with IBUS assistance on sensitivity ( d ) and specificity ( e ). f - g , Comparison of all readers, junior readers and senior readers without and with IBUS assistance on FNR ( f ) and FPR ( g ). h - i , Cohen’s kappa values on malignancy assessment between readers without ( h ) and with IBUS assistance ( i ). j , BI-RADS grades of benign and malignant cases from eight readers with and without IBUS assitance. k , Cohen’s kappa values on risk stratification between all readers without IBUS assistance. l , Statistics of benign and malignant cases with risk stratification modifications. ’Grade Up’ and ’Grade Down’ refer to raising or lowering the BI-RADS grade, respectively.
| Dataset | Images | Patients | Benign Patients | Malignant Patients |
| D1 | 1,559 | 1,559 | 443 | 1,116 |
| D2 | 846 | 846 | 443 | 403 |
| D3 | 2,023 | 364 | 171 | 193 |
| D4 | 200 | 100 | 48 | 52 |
| D5 | 168 | 168 | 68 | 100 |
| D6 | 163 | 163 | 109 | 54 |
Table S1 : Summary of study data used in this work. The table reports the number of images, all patients, benign patients and malignant patients in each dataset.
| Category | Descriptor ( Abbreviation ) | |
| Orientation | Parallel (OR1) | Unparallel (OR2) |
| Shape | Round (SH1) | Oval (SH2) |
| Irregular (SH3) | ||
| Calcifications | No calcifications (CA1) | Calcifications in mass (CA2) |
| Calcifications outside mass (CA3) | Intraductal calcifications (CA4) | |
| Posterior features | No posterior features (PF1) | Enhanced (PF2) |
Table S2 : BI-RADS descriptors and corresponding abbreviations.
| Method | AUC | ACC | SEN | SPE | PPV | NPV |
| IBUS | 0.849 (0.802-0.892) | 0.780 (0.733-0.823) | 0.846 (0.790-0.902) | 0.708 (0.632-0.777) | 0.759 (0.694-0.818) | 0.810 (0.740-0.876) |
| Readers | ||||||
| ALL | 0.755* (0.724-0.785) | 0.752* (0.718-0.784) | 0.679* (0.628-0.727) | 0.830* (0.791-0.867) | 0.817* (0.769-0.862) | 0.709* (0.649-0.764) |
| Junior | 0.722* (0.687-0.757) | 0.718* (0.680-0.756) | 0.623* (0.565-0.678) | 0.821* (0.776-0.865) | 0.795* (0.740-0.850) | 0.670* (0.608-0.729) |
| Senior | 0.787* (0.753-0.820) | 0.785 (0.750-0.819) | 0.736* (0.680-0.787) | 0.839* (0.797-0.878) | 0.838* (0.792-0.882) | 0.748* (0.687-0.805) |
| Readers with IBUS assistance |
Table S3 : Diagnostic performance of IBUS, junior readers, senior readers without and with IBUS assistance.
| Readers | AUC | ACC | SEN | SPE | PPV | NPV |
| R1 | 0.737 (0.691-0.782) | 0.730 (0.680-0.780) | 0.564 (0.490-0.641) | 0.910 (0.860-0.953) | 0.871 (0.804-0.934) | 0.658 (0.591-0.722) |
| R1 w/ IBUS | 0.793 (0.748-0.835) | 0.790 (0.743-0.833) | 0.718 (0.641-0.785) | 0.868 (0.807-0.921) | 0.855 (0.792-0.911) | 0.740 (0.671-0.802) |
| R2 | 0.702 (0.653-0.750) | 0.697 (0.643-0.747) | 0.577 (0.500-0.654) | 0.826 (0.761-0.888) | 0.783 (0.710-0.858) | 0.643 (0.572-0.711) |
| R2 w/ IBUS | 0.766 (0.720-0.811) | 0.763 (0.717-0.810) | 0.705 (0.634-0.774) | 0.826 (0.761-0.887) | 0.815 (0.752-0.879) | 0.721 (0.655-0.786) |
| R3 | 0.755 (0.704-0.802) | 0.753 (0.703-0.800) | 0.724 (0.653-0.794) | 0.785 (0.717-0.851) | 0.785 (0.716-0.853) | 0.724 (0.658-0.794) |
| R3 w/ IBUS | 0.764 (0.717-0.810) | 0.763 (0.717-0.810) | 0.737 (0.667-0.805) | 0.792 (0.725-0.858) | 0.793 (0.725-0.861) | 0.735 (0.671-0.805) |
Table S4 : Diagnostic performance of individual readers without and with IBUS assistance.
Figure S1 : The distribution of BI-RADS descriptors from datasets D1-D20.
Figure S2 : Performance and visualization on lesion segmentation . a , Segmentation performance of IBUS, MultiTask, UltraFedFM, UltraSAM, and USFM across the test sets. b , Segmentation results and convolutional feature responses from IBUS and MultiTask of four representative cases. Red, green, and blue contours denote the ground-truth, IBUS, and MultiTask boundaries, respectively. Despite using semi-supervised setting, IBUS achieved overall comparable performance with MultiTask. Meanwhile, IBUS showed better performance in delineating indistinct lesion boundaries, in which edge-enhanced convolution generated more concentrated boundary responses than standard convolution in MultiTask.
Figure S3 : Ablation study on visual extractor. IBUS adopted three different feature extractors to build the models, including: 1) ResNet-34; 2) DenseNet-121; 3) EfficientNet-b4. All three backbones achieved comparable performance, indicating that IBUS is insensitive to feature extractor. Among them, ResNet-34 yielded slightly superior performance and was therefore adopted in all subsequent experiments.
Figure S4 : Visualization on IBUS w/o OR, IBUS w/o SH, IBUS w/o PF and IBUS w/o EC. a , Four representative misclassified cases from IBUS w/o OR. d , Four representative misclassified cases from IBUS w/o SH. c , Four representative misclassified cases from IBUS w/o CA. d , Four representative misclassified cases from IBUS w/o PF. e , Four representative misclassified cases from IBUS w/o EC. For IBUS w/o OR, lesions labeled as parallel can be easily misjudged because their boundaries are unclear due to echo attenuation. In IBUS w/o SH, benign lesions with oval appearance are hard to distinguish because of fuzzy borders or uneven echoes, while small irregular malignant cases are easily overlooked. Removing calcifications descriptors substantially increased confusion between benign lesions with tiny hyperechoic areas and malignant lesions with microcalcifications. Obviously, the removal of posterior features makes it difficult to distinguish ambiguous enhancement/attenuation. As see in the cases from IBUS w/o EC, hypoechoic lesions might show similar echo with heterogenous or complex cystic or solid lesions.
Figure S5 : Risk stratification confusion matrices between reference BI-RADS grades and readers’ assigned grades. Across all readers, BI-RADS assessments were predominantly concentrated between BI-RADS 3 and BI-RADS 4C. Misclassifications mainly occurred between adjacent categories, for example, BI-RADS 4A lesions being assigned to BI-RADS 3 or BI-RADS 4B. Overall, grading concordance improve consistently with increasing reader’s experience.
Figure S6 : Risk stratification confusion matrices for benign ( a ) and malignant cases ( b ) in reader study. Comparisons were performed between readers’ assessments without and with IBUS assistance (denoted as “R* vs . R* w/ IBUS”). Both IBUS and readers mainly categorized benign lesions into BI-RADS 3-4A, while malignant lesions into BI-RADS 4B-5. Meanwhile, the mean kappa between readers’ independent and IBUS-assisted assessments was 0.893 (range, 0.725–0.961) and 0.812 (range, 0.553–0.948), respectively. Importantly, no obvious grading errors was introduced with IBUS assistance, such as upgrading BI-RADS 2 lesions to BI-RADS 5 or downgrading BI-RADS 5 lesions to BI-RADS 2.
Figure S7 : Evaluation results of CAMs and distributions of modified cases in P1P3. a , Evaluation results of CAMs from eight readers. Across all eight readers, more than 70% of cases were assigned to P1 or P2, indicating that the model-generated CAMs showed substantial overlap with clinically relevant lesion regions in the majority of cases. These results suggest that IBUS generally localized its attention to lesion-associated regions, providing visually interpretable evidence for its predictions. b , Distribution of cases with modified BI-RADS grade in P1P3. Notably, cases in which readers modified their BI-RADS assessments after IBUS assistance were concentrated in P1 and P2, whereas few modifications occurred in P3. This pattern suggests that changes in risk stratification were more likely to occur when the model’s attention maps were considered clinically plausible by the readers. Rather than indicating uniform acceptance of model predictions, these findings are consistent with readers selectively incorporating IBUS outputs when the accompanying visual evidence was considered informative, potentially providing a safeguard against inappropriate reliance on model predictions.
Figure S8 : Evaluation of lesion localization maps by eight readers. Across all readers, more than 95% of cases were assigned to P1 or P2, indicating that the predicted localization maps covered more than 50% of the lesion region in the vast majority of cases. These findings suggest that IBUS can effectively localize breast lesions and provide spatially relevant evidence for subsequent diagnostic assessment.
Figure S9 : Examples of IBUS prediction basis in reader study. Heatmap analysis of four controversial cases. The red areas indicate a high level of model attention, while the blue areas indicate a low level of model attention. a , IBUS correctly identified the lesion as malignant. In the second assessment conducted 2 weeks later with IBUS assistance, all readers think the model’s attention focus on wrong region, and four out of four junior readers who initially diagnosed it as benign changed their assessment to malignant. Meanwhile, three out four junior readers raise the BI-RADS grades. b , IBUS correctly identified the lesion as malignant. In the second assessment with IBUS assistance, all readers think the model’s attention focus on wrong region, and 4 out of eight readers (2 out of four junior readers and 2 out of four senior readers) who initially diagnosed it as benign changed their assessment to malignant. All readers maintain the initial BI-RADS grades. c , AI correctly identified the lesion as benign. In the second assessment with IBUS assistance, all readers think the model’s attention focus on wrong region, and three out of four senior readers who initially diagnosed it as malignant changed their assessment to benign. Meanwhile, four out of eight readers (two junior and two senior readers) lower the BI-RADS grades. d , AI correctly identified the lesion as benign. In the second assessment with IBUS assistance, all readers think the model’s attention focus on wrong region, and three senior readers who initially diagnosed it as malignant changed their assessment to benign. Meanwhile, three out of four senior readers lower the BI-RADS grades.
Figure S10 : Some representative wrong examples in reader study. Heatmap analysis of four controversial cases. The red areas indicate a high level of model attention, while the blue areas indicate a low level of model attention. a , IBUS correctly identified the lesion as malignant. In the second assessment conducted 2 weeks later with IBUS assistance, seven out of eight readers think the model’s attention focus on wrong region, and seven out of eight readers (4 out of four junior readers and 3 out of four senior readers) who initially diagnosed it as benign did not follow the model’s prediction to change their assessment to malignant, while one senior reader change his assessment to malignant. All readers maintain the initial BI-RADS grades. b , IBUS wrongly identified the lesion as malignant. In the second assessment with IBUS assistance, six out of eight readers think the model’s attention focus on wrong region, and 6 out of eight readers (4 out of four junior readers and 2 out of four senior readers) who initially diagnosed it as benign did not follow the model’s prediction to change their assessment to malignant, while two senior readers change his assessment to malignant. All readers maintain the initial BI-RADS grades. All readers maintain the initial BI-RADS grades. c , AI wrongly identified the lesion as benign. In the second assessment with IBUS assistance, eight out of eight readers think the model’s attention focus on clinically relative region, and 6 out of eight readers (4 out of four junior readers and 2 out of four senior readers) maintained their assessment as malignant, while one senior readers change his assessment to malignant. Meanwhile, one senior readers raised the BI-RADS grades from 3 to 4A. d , AI correctly identified the lesion as benign. In the second assessment with IBUS assistance, all readers think the model’s attention focus on clinically relative region, and 5 out of eight readers (2 out of four junior readers and 3 out of four senior readers) maintained their assessment as malignant. All readers maintain the initial BI-RADS grades.
Figure S11 : BI-RADS descriptors re-evaluation in reader study . Eight radiologists were invited to review the BI-RADS descriptors used by IBUS and re-annotate those they considered inaccurate or inappropriate. The revised descriptors from each reader were subsequently used as inputs to IBUS (denoted as “IBUS w/ R*”) to reassess diagnostic performance and were compared with the original IBUS results. Annotation variability was observed between reader-assigned and original BI-RADS descriptors, with disagreements for most readers concentrated in the partial descriptor of cases. Thus, despite descriptor revision resulted in both increases and decreases in performance across readers, most differences were not statistically significant, suggesting a degree of tolerance to inter-reader variability in BI-RADS descriptor assessment.