SMILE: A Structured Multi-Agent Clinical Reasoning for Liver Ultrasound Understanding
Organizations: Yonsei University · The Catholic University of Korea
Abstract
Automated liver ultrasound report generation improves the diagnosis of liver diseases by producing clinically structured descriptions of anatomical and disease-related findings. However, current generative vision-language models (VLMs) often rely on a single reasoning pathway that maps ultrasound appearance directly to textual descriptions. In addition to the inherent vision-language modality gap, this design is limited for liver ultrasound because disease interpretation is constrained by subtle visual evidence, heterogeneous disease patterns, and limited paired supervision. To address these limitations, we propose SMILE, a Structured Multi-agent clInicaL rEasoning framework that uses multiple complementary specialist agents to interpret ultrasound images and consolidates their outputs through a judge-guided consensus mechanism. Our approach employs three specialized agents: an image interpretation agent for morphological descriptions, a clinical-stage reasoning agent, and a visual question answering (VQA) agent for targeted queries. These agent responses are then aggregated and scrutinized by a judge VLM using a consensus filtering prompt to produce a unified, clinically coherent output. Across a public liver ultrasound dataset and an internal clinical dataset, SMILE consistently outperforms state-of-the-art VLMs, achieving up to 19.0% improvement in ROUGE-1 and 18.8% improvement in clinical F1-score, while reaching VQA F1-scores of 0.82 and 0.65 on public and internal datasets, respectively. These results demonstrate the potential of SMILE for data-scarce and limited-supervision clinical settings. Our models, datasets, and code are available here.
Figures & tables
| Dataset | Model | NLG | CE | Cost | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R-1 | R-L | M | BS | Acc | Prec | Rec | F1 | P | L | FLOPs | ||
| D-1 | Qwen-2B [ 35 ] | 0.10 | 0.09 | 0.02 | 0.81 | 0.88 | 0.88 | 0.05 | 0.01 | 4.36 | 13.2 | 3.9 |
| MedMO-4B [ 9 ] | 0.69 | 0.59 | 0.47 | 0.93 | 0.94 | 0.84 | 0.59 | 0.69 | 80.54 | 23.8 | 7.0 | |
| MedGem-4B [ 30 ] | 0.59 | 0.48 | 0.49 | 0.90 | 0.94 | 0.78 | 0.66 | 0.72 | 77.00 | 25.6 | 7.6 | |
| MedAgents [ 31 ] | 0.60 | 0.45 | 0.50 | 0.90 | 0.94 | 0.77 | 0.70 | 0.73 | 55.63 | 228.9 | 81.9 | |
| SMILE (Ours) | 0.72 | 0.61 | 0.56 | 0.93 | 0.96 | 0.85 | 0.80 | 0.82 | 55.63 | 74.3 | 34.0 | |
| Dataset | Model | Accuracy | F1-score |
|---|---|---|---|
| D-1 | Qwen-2B [ 35 ] | 0.62 | 0.77 |
| MedMO-4B [ 9 ] | 0.67 | 0.76 | |
| MedGem-4B [ 30 ] | 0.53 | 0.62 | |
| MedAgents [ 31 ] | 0.50 | 0.41 | |
| SMILE (Ours) | 0.77 | 0.82 | |
| D-2 | Qwen-2B [ 35 ] | 0.32 | 0.44 |
| Dataset | Model | Mean | Std |
|---|---|---|---|
| D-1 | Qwen-2B [ 35 ] | 0.48 | 0.014 |
| MedMO-4B [ 9 ] | 0.45 | 0.009 | |
| MedGem-4B [ 30 ] | 0.49 | 0.020 | |
| MedAgents [ 31 ] | 0.45 | 0.007 | |
| SMILE (Ours) | 0.51 | 0.014 | |
| D-2 | Qwen-2B [ 35 ] | 0.33 | 0.010 |
| Setting | R-1 | R-L | M | CE-F1 | (M) | Calls | (s) | FLOPs |
| Single Agents | ||||||||
| Agent A (Radiology Specialist) | 0.44 | 0.37 | 0.24 | 0.53 | 38.50 | 1 | 25.6 | 7.6 |
| Agent B (Pathology Specialist) | 0.25 | 0.24 | 0.05 | 0.03 | 14.61 | 1 | 13.2 | 3.9 |
| Agent C (VQA Specialist) | 0.18 | 0.14 | 0.06 | 0.26 | 2.52 | 1 | 49.4 | 14.6 |
| Pairwise Agents | ||||||||
| A + B | 0.70 | 0.62 | 0.50 | 0.70 | 53.11 | 3 | 70.2 | 24.4 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Adapter | LoRA |
| Radiology expert | MedGemma-4B, |
| Pathology expert | Qwen2-VL-2B, |
| VQA expert | Qwen2-VL-2B, restricted |
| Additional MedAgents experts | Zero shot |
| Coordinator / moderator | Frozen |
| Canonical entity | Surface variants |
|---|---|
| portal vein | portal vein, main portal vein, portal venous |
| bile duct | bile duct(s), intrahepatic bile ducts, biliary |
| gallbladder | gallbladder, GB |
| pancreas | pancreas, pancreatic |
| spleen | spleen, splenic |
| parenchyma | parenchyma, parenchymal, parenchymal echo, |
| Dataset | Question |
|---|---|
| D-1 | Is the liver parenchymal echogenicity diffusely increased in the imaged portion? |
| Is the hepatic echotexture coarse or heterogeneous in the imaged portion? | |
| Is posterior acoustic attenuation increased in the imaged portion? | |
| D-2 | Is steatosis or fatty change reported for the liver, and is it visually assessable? |
| Is cirrhosis reported, and are cirrhotic morphological features visually assessable? | |
| Is a focal hepatic lesion reported, and is it visually assessable? |
| Component | Backbone | Rank | Trainable |
|---|---|---|---|
| Agent A (radiology) | MedGemma-4B | M | |
| Agent B (pathology) | Qwen2-VL-2B | M | |
| Agent C (VQA) ∗ | Qwen2-VL-2B | M | |
| Judge | MedGemma-4B | frozen | M |
| SMILE total | M |
Radiology Report Generation
Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.
See all 170 papers