UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis

Yali Zhu; Kang Zhou; Dingbang Wu; Gaofeng Meng

cs.CVMar 11, 2026

UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis

Authors: Yali Zhu, Kang Zhou, Dingbang Wu, Gaofeng Meng

Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong

Abstract

Breast ultrasound diagnosis typically proceeds from global lesion localization to local sign assessment and then evidence integration to assign a BI-RADS category and determine benignity or malignancy. Many existing methods rely on end-to-end prediction or provide only weakly grounded evidence, which can miss fine-grained lesion cues and limit auditability and clinical review. To align with the clinical workflow and improve evidence traceability, we propose a hierarchical multi-agent framework, termed UltrasoundAgents. A main agent localizes the lesion in the full image and triggers a crop-and-zoom operation. A sub-agent analyzes the local view and predicts four clinically relevant attributes, namely echogenicity pattern, calcification, boundary type, and edge (margin) morphology. The main agent then integrates these structured attributes to perform evidence-based reasoning and output the BI-RADS category and the malignancy prediction, while producing reviewable intermediate evidence. Furthermore, hierarchical multi-agent training often suffers from error propagation, difficult credit assignment, and sparse rewards. To alleviate this and improve training stability, we introduce a decoupled progressive training strategy. We first train the attribute agent, then train the main agent with oracle attributes to learn robust attribute-based reasoning, and finally apply corrective trajectory self-distillation with spatial supervision to build high-quality trajectories for supervised fine-tuning, yielding a deployable end-to-end policy. Experiments show consistent gains over strong vision-language baselines in diagnostic accuracy and attribute agreement, together with structured evidence and traceable reasoning.

Figures & tables

Figure 1: Hierarchical multi-agent architecture. The main agent analyzes the full image to localize the lesion, triggers a crop-and-zoom operation, and queries the sub-agent on the zoomed view to obtain structured attribute evidence. The main agent then integrates the global context and the attributes to predict malignancy and BI-RADS.

Figure 2: Three-stage training. Stage 1: RL trains $A_{S}$ for attribute recognition. Stage 2: oracle-guided RL trains $A_{M}$ with GT attributes for stable reasoning. Stage 3: refine trajectories and distill via SFT for robustness.

Method	BUSBRA				BUSI				BUDIAT				BrEaST (OOD)				Overall
Method	AUC	Acc	Bi-Acc	$\kappa$	AUC	Acc	Bi-Acc	$\kappa$	AUC	Acc	Bi-Acc	$\kappa$	AUC	Acc	Bi-Acc	$\kappa$	AUC	Acc	Bi-Acc	$\kappa$
Qwen2.5-3B-Zero-Shot	0.458	0.588	0.091	0.003	0.493	0.573	0.156	0.066	0.500	0.743	0.200	-0.053	0.484	0.608	0.078	-0.065	0.476	0.602	0.117	0.014
Qwen2.5-3B-COT-SFT	0.668	0.722	0.563	0.258	0.835	0.833	0.46	0.245	0.722	0.857	0.4	0.129	0.586	0.627	0.176	0.06	0.71	0.751	0.468	0.204
Think-with-Image	0.500	0.693	0.048	0.003	0.528	0.660	0.080	0.019	0.500	0.743	0.314	-0.001	0.526	0.647	0.196	0.005	0.512	0.683	0.101	0.004
ours	0.723	0.813	0.62	0.3	0.784	0.833	0.542	0.244	0.778	0.886	0.4	0.145	0.685	0.725	0.157	0.037	0.741	0.813	0.515	0.224

Table 1: Main diagnostic performance on four datasets. Overall is computed on the pooled set across all datasets (i.e., sample-wise micro average).

Method	BUSBRA					BUSI					BUDIAT					BrEaST (OOD)					Overall
Method	AUC	Acc	Bi-Acc	$\kappa$	IoU	AUC	Acc	Bi-Acc	$\kappa$	IoU	AUC	Acc	Bi-Acc	$\kappa$	IoU	AUC	Acc	Bi-Acc	$\kappa$	IoU	AUC	Acc	Bi-Acc	$\kappa$	IoU
w/o Oracle Training	0.500	0.693	0.497	0.043	0.266	0.569	0.690	0.430	0.020	0.406	0.556	0.771	0.371	0.055	0.411	0.563	0.667	0.098	-0.013	0.346	0.535	0.696	0.413	0.018	0.328
w/o Self-Distill	0.679	0.738	0.524	0.175	0.254	0.771	0.802	0.521	0.241	0.373	0.889	0.943	0.400	0.159	0.348	0.686	0.686	0.137	0.012	0.294	0.726	0.767	0.458	0.173	0.299
ours	0.723	0.813	0.620	0.300	0.638	0.784	0.833	0.542	0.244	0.610	0.778	0.886	0.400	0.145	0.567	0.685	0.725	0.157	0.037	0.537	0.741	0.813	0.515	0.224	0.610
ours-with-GTbox	0.780	0.845	0.583	0.243	1	0.769	0.823	0.542	0.229	1	0.778	0.886	0.429	0.176	1	0.801	0.804	0.176	0.045	1	0.782	0.837	0.501	0.208	1
ours-with-GTattr	0.778	0.856	0.668	0.391	0.622	0.851	0.884	0.663	0.459	0.560	0.759	0.857	0.429	0.181	0.560	0.796	0.784	0.216	0.086	0.395	0.804	0.853	0.582	0.345	0.568

Table 2: Ablation study and upper-bound analysis.

Attribute (Input)	BUSBRA			BUSI			BUDIAT			BrEaST			Overall
Attribute (Input)	Acc	Macro-F1	W-F1	Acc	Macro-F1	W-F1	Acc	Macro-F1	W-F1	Acc	Macro-F1	W-F1	Acc	Macro-F1	W-F1
Boundary (cotsft)	0.511	0.321	0.399	0.620	0.394	0.535	0.514	0.330	0.375	0.490	0.374	0.428	0.537	0.341	0.436
Boundary (whole)	0.535	0.326	0.405	0.656	0.394	0.548	0.514	0.341	0.377	0.529	0.387	0.429	0.564	0.340	0.440
Boundary (lesion)	0.567	0.447	0.510	0.635	0.419	0.568	0.571	0.472	0.460	0.588	0.524	0.566	0.588	0.445	0.526
Edge (cotsft)	0.595	0.462	0.517	0.670	0.540	0.607	0.514	0.440	0.375	0.608	0.618	0.605	0.609	0.473	0.541
Edge (whole)	0.615	0.452	0.512	0.708	0.541	0.618	0.514	0.455	0.377	0.667	0.662	0.655	0.637	0.475	0.546

Table 3: Attribute classification accuracy, Macro-F1, and Weighted-F1 for whole vs lesion inputs.Calcification omitted from the table due to severe imbalance.

Panel of Figure 3 — Figure 3: Qualitative examples of evidence-chain diagnosis and attribute reasoning traces.

Explore similar work

Apr 30, 2026

Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation

Ultrasound interpretation requires both precise lesion localization and holistic clinical reasoning, yet existing methods typically excel at only one of these capabilities: specialized detectors offer strong localization but limited reasoning, whereas multimodal large language models (MLLMs) provide flexible reasoning but weak grounding in specialized medical domains. We present Echo-α, an agentic multimodal reasoning model for ultrasound interpretation that unifies these strengths within an invoke-and-reason framework. Echo-α is trained to coordinate organ-specific detector outputs, integrate them with global visual context, and convert the resulting evidence into grounded diagnostic decisions beyond detector-only inference. This behavior is established through a nine-task supervised curriculum and then refined by sequential reinforcement learning under different reward trade-offs, yielding Echo-α-Grounding for lesion anchoring and Echo-α-Diagnosis for final diagnosis. On multi-center renal and breast ultrasound benchmarks, Echo-α outperforms competitive baselines on both grounding and diagnosis. In particular, on cross-center test sets, Echo-α-Grounding attains 56.73%/43.78% [email protected] and Echo- α-Diagnosis reaches 74.90%/49.20% overall accuracy on renal/breast ultrasound. These results suggest that agentic multimodal reasoning can turn specialized detectors into verifiable clinical evidence, offering a practical route toward ultrasound AI systems that are more accurate, interpretable, and transferable. The repository is at https://github.com/MiliLab/Echo-Alpha.

Jing Zhang, Wentao Jiang, Tao Huang +8

School of Computer Science, Wuhan University, China · The Central Hospital of Wuhan, China · Taizhou Hospital of Zhejiang Province, China +1

Medical Image Analysis Multimodal Reasoning Ultrasound Imaging Medical VLMs Multimodal Agents Medical Image Localization

Ask PDF

Feb 27, 2026

Experience-Guided Self-Adaptive Cascaded Agents for Breast Cancer Screening and Diagnosis with Reduced Biopsy Referrals

We propose an experience-guided cascaded multi-agent framework for Breast Ultrasound Screening and Diagnosis, called BUSD-Agent, that aims to reduce diagnostic escalation and unnecessary biopsy referrals. Our framework models screening and diagnosis as a two-stage, selective decision-making process. A lightweight

screening clinic' agent, restricted to classification models as tools, selectively filters out benign and normal cases from further diagnostic escalation when malignancy risk and uncertainty are estimated as low. Cases that have higher risks are escalated to the

diagnostic clinic' agent, which integrates richer perception and radiological description tools to make a secondary decision on biopsy referral. To improve agent performance, past records of pathology-confirmed outcomes along with image embeddings, model predictions, and historical agent actions are stored in a memory bank as structured decision trajectories. For each new case, BUSD-Agent retrieves similar past cases based on image, model response and confidence similarity to condition the agent's current decision policy. This enables retrieval-conditioned in-context adaptation that dynamically adjusts model trust and escalation thresholds from prior experiences without parameter updates. Evaluation across 10 breast ultrasound datasets shows that the proposed experience-guided workflow reduces diagnostic escalation in BUSD-Agent from 84.95% to 58.72% and overall biopsy referrals from 59.50% to 37.08%, compared to the same architecture without trajectory conditioning, while improving average screening specificity by 68.48% and diagnostic specificity by 6.33%.

Pramit Saha, Mohammad Alsharid, Joshua Strong +1

Department of Engineering Science, University of Oxford · Department of Computer Science, Khalifa University

Clinical Decision Support Self-Improving Agents Multi-Agent Systems Selective Prediction Ultrasound Imaging

Ask PDF

Oct 12, 2026

A Concept-informed Interpretable Artificial Intelligence Framework for Breast Ultrasound Diagnosis

Most existing artificial intelligence systems for breast ultrasound rely primarily on image-driven prediction, with limited incorporation of the knowledge from the Breast Imaging Reporting and Data System (BI-RADS) and the intermediate assessment processes underlying radiologists' workflow, thereby constraining their interpretability and alignment with clinical practice. We curated, to our knowledge, the largest multicenter collection of breast ultrasound images paired with structured BI-RADS descriptors, comprising 9,941 patients across 20 independent datasets. Across 14 testing datasets, IBUS demonstrated consistent generalization performance across different clinical centers and imaging conditions. Incorporating BI-RADS descriptors enabled fine-grained risk stratification, while cross-checking predicted BI-RADS grades against malignancy probabilities identified internally discordant predictions that may warrant additional clinical review. In a reader study involving eight radiologists, IBUS assistance increased diagnostic sensitivity among both junior and senior readers, with the potential of guiding radiologists to make targeted revision in risk stratification. Overall, IBUS exhibits its potential as a second-stage decision-support system that complements radiologists by providing additional risk estimates and interpretable evidence for diagnostically challenging cases. The demo video of IBUS platform is publicly available at https://github.com/zsz1997/IBUS.

Shengzhou Zhong, Ziying Wang, Liang Li +14

School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong, 510515, China. · Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, Guangdong, 510515, China. · Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China. +5

Clinical Decision Support Explainable Medical Image Analysis Medical Image Classification Ultrasound Imaging Concept-Based Explanations

Ask PDF