UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis

Yali Zhu; Kang Zhou; Dingbang Wu; Gaofeng Meng

cs.CVMar 11, 2026

UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis

Authors: Yali Zhu, Kang Zhou, Dingbang Wu, Gaofeng Meng

Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong

Abstract

Breast ultrasound diagnosis typically proceeds from global lesion localization to local sign assessment and then evidence integration to assign a BI-RADS category and determine benignity or malignancy. Many existing methods rely on end-to-end prediction or provide only weakly grounded evidence, which can miss fine-grained lesion cues and limit auditability and clinical review. To align with the clinical workflow and improve evidence traceability, we propose a hierarchical multi-agent framework, termed UltrasoundAgents. A main agent localizes the lesion in the full image and triggers a crop-and-zoom operation. A sub-agent analyzes the local view and predicts four clinically relevant attributes, namely echogenicity pattern, calcification, boundary type, and edge (margin) morphology. The main agent then integrates these structured attributes to perform evidence-based reasoning and output the BI-RADS category and the malignancy prediction, while producing reviewable intermediate evidence. Furthermore, hierarchical multi-agent training often suffers from error propagation, difficult credit assignment, and sparse rewards. To alleviate this and improve training stability, we introduce a decoupled progressive training strategy. We first train the attribute agent, then train the main agent with oracle attributes to learn robust attribute-based reasoning, and finally apply corrective trajectory self-distillation with spatial supervision to build high-quality trajectories for supervised fine-tuning, yielding a deployable end-to-end policy. Experiments show consistent gains over strong vision-language baselines in diagnostic accuracy and attribute agreement, together with structured evidence and traceable reasoning.

Figures & tables

Figure 1: Hierarchical multi-agent architecture. The main agent analyzes the full image to localize the lesion, triggers a crop-and-zoom operation, and queries the sub-agent on the zoomed view to obtain structured attribute evidence. The main agent then integrates the global context and the attributes to predict malignancy and BI-RADS.

Figure 2: Three-stage training. Stage 1: RL trains $A_{S}$ for attribute recognition. Stage 2: oracle-guided RL trains $A_{M}$ with GT attributes for stable reasoning. Stage 3: refine trajectories and distill via SFT for robustness.

Method	BUSBRA				BUSI				BUDIAT				BrEaST (OOD)				Overall
Method	AUC	Acc	Bi-Acc	$\kappa$	AUC	Acc	Bi-Acc	$\kappa$	AUC	Acc	Bi-Acc	$\kappa$	AUC	Acc	Bi-Acc	$\kappa$	AUC	Acc	Bi-Acc	$\kappa$
Qwen2.5-3B-Zero-Shot	0.458	0.588	0.091	0.003	0.493	0.573	0.156	0.066	0.500	0.743	0.200	-0.053	0.484	0.608	0.078	-0.065	0.476	0.602	0.117	0.014
Qwen2.5-3B-COT-SFT	0.668	0.722	0.563	0.258	0.835	0.833	0.46	0.245	0.722	0.857	0.4	0.129	0.586	0.627	0.176	0.06	0.71	0.751	0.468	0.204
Think-with-Image	0.500	0.693	0.048	0.003	0.528	0.660	0.080	0.019	0.500	0.743	0.314	-0.001	0.526	0.647	0.196	0.005	0.512	0.683	0.101	0.004
ours	0.723	0.813	0.62	0.3	0.784	0.833	0.542	0.244	0.778	0.886	0.4	0.145	0.685	0.725	0.157	0.037	0.741	0.813	0.515	0.224

Table 1: Main diagnostic performance on four datasets. Overall is computed on the pooled set across all datasets (i.e., sample-wise micro average).

Method	BUSBRA					BUSI					BUDIAT					BrEaST (OOD)					Overall
Method	AUC	Acc	Bi-Acc	$\kappa$	IoU	AUC	Acc	Bi-Acc	$\kappa$	IoU	AUC	Acc	Bi-Acc	$\kappa$	IoU	AUC	Acc	Bi-Acc	$\kappa$	IoU	AUC	Acc	Bi-Acc	$\kappa$	IoU
w/o Oracle Training	0.500	0.693	0.497	0.043	0.266	0.569	0.690	0.430	0.020	0.406	0.556	0.771	0.371	0.055	0.411	0.563	0.667	0.098	-0.013	0.346	0.535	0.696	0.413	0.018	0.328
w/o Self-Distill	0.679	0.738	0.524	0.175	0.254	0.771	0.802	0.521	0.241	0.373	0.889	0.943	0.400	0.159	0.348	0.686	0.686	0.137	0.012	0.294	0.726	0.767	0.458	0.173	0.299
ours	0.723	0.813	0.620	0.300	0.638	0.784	0.833	0.542	0.244	0.610	0.778	0.886	0.400	0.145	0.567	0.685	0.725	0.157	0.037	0.537	0.741	0.813	0.515	0.224	0.610
ours-with-GTbox	0.780	0.845	0.583	0.243	1	0.769	0.823	0.542	0.229	1	0.778	0.886	0.429	0.176	1	0.801	0.804	0.176	0.045	1	0.782	0.837	0.501	0.208	1
ours-with-GTattr	0.778	0.856	0.668	0.391	0.622	0.851	0.884	0.663	0.459	0.560	0.759	0.857	0.429	0.181	0.560	0.796	0.784	0.216	0.086	0.395	0.804	0.853	0.582	0.345	0.568

Table 2: Ablation study and upper-bound analysis.

Attribute (Input)	BUSBRA			BUSI			BUDIAT			BrEaST			Overall
Attribute (Input)	Acc	Macro-F1	W-F1	Acc	Macro-F1	W-F1	Acc	Macro-F1	W-F1	Acc	Macro-F1	W-F1	Acc	Macro-F1	W-F1
Boundary (cotsft)	0.511	0.321	0.399	0.620	0.394	0.535	0.514	0.330	0.375	0.490	0.374	0.428	0.537	0.341	0.436
Boundary (whole)	0.535	0.326	0.405	0.656	0.394	0.548	0.514	0.341	0.377	0.529	0.387	0.429	0.564	0.340	0.440
Boundary (lesion)	0.567	0.447	0.510	0.635	0.419	0.568	0.571	0.472	0.460	0.588	0.524	0.566	0.588	0.445	0.526
Edge (cotsft)	0.595	0.462	0.517	0.670	0.540	0.607	0.514	0.440	0.375	0.608	0.618	0.605	0.609	0.473	0.541
Edge (whole)	0.615	0.452	0.512	0.708	0.541	0.618	0.514	0.455	0.377	0.667	0.662	0.655	0.637	0.475	0.546

Table 3: Attribute classification accuracy, Macro-F1, and Weighted-F1 for whole vs lesion inputs.Calcification omitted from the table due to severe imbalance.

Panel of Figure 3 — Figure 3: Qualitative examples of evidence-chain diagnosis and attribute reasoning traces.

Explore similar work

Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation
Apr 30, 2026Jing Zhang, Wentao Jiang, Tao Huang +8Medical Image Analysis Multimodal Reasoning
School of Computer Science, Wuhan University, China · The Central Hospital of Wuhan, China · Taizhou Hospital of Zhejiang Province, China +1
Experience-Guided Self-Adaptive Cascaded Agents for Breast Cancer Screening and Diagnosis with Reduced Biopsy Referrals
Feb 27, 2026Pramit Saha, Mohammad Alsharid, Joshua Strong +1Clinical Decision Support Self-Improving Agents
Department of Engineering Science, University of Oxford · Department of Computer Science, Khalifa University
A Concept-informed Interpretable Artificial Intelligence Framework for Breast Ultrasound Diagnosis
Oct 12, 2026Shengzhou Zhong, Ziying Wang, Liang Li +14Clinical Decision Support Explainable Medical Image Analysis
School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong, 510515, China. · Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, Guangdong, 510515, China. · Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China. +5

Apr 30, 2026J/K move · Enter open · S save

Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation

Jing Zhang, Wentao Jiang, Tao Huang, Zhiwei Wang+7

School of Computer Science, Wuhan University, China · The Central Hospital of Wuhan, China · Taizhou Hospital of Zhejiang Province, China+1

Medical Image Analysis Multimodal Reasoning Ultrasound Imaging Medical VLMs Multimodal Agents Medical Image Localization

Open paper AskPDF arXiv

Ultrasound interpretation requires both precise lesion localization and holistic clinical reasoning, yet existing methods typically excel at only one of these capabilities: specialized detectors offer strong localization but limited reasoning, whereas multimodal large language models (MLLMs) provide flexible reasoning but weak grounding in specialized medical domains. We present Echo-α, an agentic multimodal reasoning model for ultrasound interpretation that unifies these strengths within an invoke-and-reason framework. Echo-α is trained to coordinate organ-specific detector outputs, integrate them with global visual context, and convert the resulting evidence into grounded diagnostic decisions beyond detector-only inference. This behavior is established through a nine-task supervised curriculum and then refined by sequential reinforcement learning under different reward trade-offs, yielding Echo-α-Grounding for lesion anchoring and Echo-α-Diagnosis for final diagnosis. On multi-center renal and breast ultrasound benchmarks, Echo-α outperforms competitive baselines on both grounding and diagnosis. In particular, on cross-center test sets, Echo-α-Grounding attains 56.73%/43.78% [email protected] and Echo- α-Diagnosis reaches 74.90%/49.20% overall accuracy on renal/breast ultrasound. These results suggest that agentic multimodal reasoning can turn specialized detectors into verifiable clinical evidence, offering a practical route toward ultrasound AI systems that are more accurate, interpretable, and transferable. The repository is at https://github.com/MiliLab/Echo-Alpha.

Abstract

Explore similar work

Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation

Experience-Guided Self-Adaptive Cascaded Agents for Breast Cancer Screening and Diagnosis with Reduced Biopsy Referrals

A Concept-informed Interpretable Artificial Intelligence Framework for Breast Ultrasound Diagnosis