Automated liver ultrasound report generation improves the diagnosis of liver diseases by producing clinically structured descriptions of anatomical and disease-related findings. However, current generative vision-language models (VLMs) often rely on a single reasoning pathway that maps ultrasound appearance directly to textual descriptions. In addition to the inherent vision-language modality gap, this design is limited for liver ultrasound because disease interpretation is constrained by subtle visual evidence, heterogeneous disease patterns, and limited paired supervision. To address these limitations, we propose SMILE, a Structured Multi-agent clInicaL rEasoning framework that uses multiple complementary specialist agents to interpret ultrasound images and consolidates their outputs through a judge-guided consensus mechanism. Our approach employs three specialized agents: an image interpretation agent for morphological descriptions, a clinical-stage reasoning agent, and a visual question answering (VQA) agent for targeted queries. These agent responses are then aggregated and scrutinized by a judge VLM using a consensus filtering prompt to produce a unified, clinically coherent output. Across a public liver ultrasound dataset and an internal clinical dataset, SMILE consistently outperforms state-of-the-art VLMs, achieving up to 19.0% improvement in ROUGE-1 and 18.8% improvement in clinical F1-score, while reaching VQA F1-scores of 0.82 and 0.65 on public and internal datasets, respectively. These results demonstrate the potential of SMILE for data-scarce and limited-supervision clinical settings. Our models, datasets, and code are available here.
Figures & tables
Figure 1: Comparison with existing VLMs. Top (Conventional VLM): Utilizing a single VLM, pre-trained on a single source of clinical information, which generates responses with hallucinated findings. Bottom (Our Method): SMILE uses complementary specialist agents adapted with radiology-aligned and pathology-aligned supervision to extract complementary clinical evidence from liver ultrasound images. Red text indicates hallucinated findings, and green text indicates findings corrected by SMILE.
Figure 2: An overview of our proposed method. (a) Training procedure of a single agent as explained in Section 3.3 . (b) Inference from multi-agent under consensus filtering by a judge to obtain a high-fidelity liver ultrasound report.
Dataset
Model
NLG
CE
Cost
R-1
R-L
M
BS
Acc
Prec
Rec
F1
P
L
FLOPs
D-1
Qwen-2B [ 35 ]
0.10
0.09
0.02
0.81
0.88
0.88
0.05
0.01
4.36
13.2
3.9
MedMO-4B [ 9 ]
0.69
0.59
0.47
0.93
0.94
0.84
0.59
0.69
80.54
23.8
7.0
MedGem-4B [ 30 ]
0.59
0.48
0.49
0.90
0.94
0.78
0.66
0.72
77.00
25.6
7.6
MedAgents [ 31 ]
0.60
0.45
0.50
0.90
0.94
0.77
0.70
0.73
55.63
228.9
81.9
SMILE (Ours)
0.72
0.61
0.56
0.93
0.96
0.85
0.80
0.82
55.63
74.3
34.0
Table 1: Quantitative comparison of the proposed SMILE with finetuned state-of-the-art models. We report the NLG, CE and computational cost on the test subsets of Dataset-1 and Dataset-2. The best results are highlighted in bold. P, L, and FLOPs denote trainable parameters, inference latency, and floating-point operations, reported in millions ( 106 ), seconds, and trillion ( 1012 ) operations, respectively.
Dataset
Model
Accuracy
F1-score
D-1
Qwen-2B [ 35 ]
0.62
0.77
MedMO-4B [ 9 ]
0.67
0.76
MedGem-4B [ 30 ]
0.53
0.62
MedAgents [ 31 ]
0.50
0.41
SMILE (Ours)
0.77
0.82
D-2
Qwen-2B [ 35 ]
0.32
0.44
Table 2: VQA-based clinical verification on Dataset-1 (D-1) and Dataset-2 (D-2)
Dataset
Model
Mean
Std
D-1
Qwen-2B [ 35 ]
0.48
0.014
MedMO-4B [ 9 ]
0.45
0.009
MedGem-4B [ 30 ]
0.49
0.020
MedAgents [ 31 ]
0.45
0.007
SMILE (Ours)
0.51
0.014
D-2
Qwen-2B [ 35 ]
0.33
0.010
Table 3: Image text alignment between liver ultrasound images and generated reports. We report the mean and standard deviation of cosine similarity between visual and textual embeddings.
Setting
R-1
R-L
M
CE-F1
P (M)
Calls
L (s)
FLOPs
Single Agents
Agent A (Radiology Specialist)
0.44 †
0.37 †
0.24 †
0.53
38.50
1
25.6
7.6
Agent B (Pathology Specialist)
0.25 †
0.24 †
0.05 †
0.03
14.61
1
13.2
3.9
Agent C (VQA Specialist)
0.18 †
0.14 †
0.06 †
0.26
2.52
1
49.4
14.6
Pairwise Agents
A + B
0.70
0.62
0.50 †
0.70
53.11
3
70.2
24.4
Table 4: Ablation analysis of SMILE on Dataset-1. We report single agents, pairwise agents, judge-input controls, fusion controls, and judge-backbone ablations. P denotes the number of trainable parameters in millions, Calls denotes the number of model invocations per patient, L denotes average inference latency per patient in seconds, and FLOPs denote floating-point operations, reported in trillions ( 1012 ). CE-F1 represents the clinical efficacy F1-score. For the NLG metrics, † indicates that the complete SMILE model significantly outperforms the corresponding ablation according to a two-sided Wilcoxon signed-rank test with Holm correction ( padj<0.05 ). Unmarked NLG values indicate that the difference is not statistically significant. The statistical testing protocol is detailed in Supp. Note F.
Figure 3: Qualitative analysis of SMILE. (a) Successful correction using complementary agent evidence. (b) Failure case where the judge overlooks a correct agent finding. Red, green, and orange denote incorrect, correct, and complementary findings, respectively.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Adapter
LoRA
Radiology expert
MedGemma-4B, r=16
Pathology expert
Qwen2-VL-2B, r=8
VQA expert
Qwen2-VL-2B, restricted r=8
Additional MedAgents experts
Zero shot
Coordinator / moderator
Frozen
Appendix
Table S1 : Hyperparameters of Task-Adapted MedAgents
Canonical entity
Surface variants
portal vein
portal vein, main portal vein, portal venous
bile duct
bile duct(s), intrahepatic bile ducts, biliary
gallbladder
gallbladder, GB
pancreas
pancreas, pancreatic
spleen
spleen, splenic
parenchyma
parenchyma, parenchymal, parenchymal echo,
Appendix
Table S2 : Clinical entity vocabulary used for CE evaluation. Representative surface variants are shown for the 13 canonical entities derived from the Dataset-1 seed terminology. The complete vocabulary contains 100 accepted surface variants.
Dataset
Question
D-1
Is the liver parenchymal echogenicity diffusely increased in the imaged portion?
Is the hepatic echotexture coarse or heterogeneous in the imaged portion?
Is posterior acoustic attenuation increased in the imaged portion?
D-2
Is steatosis or fatty change reported for the liver, and is it visually assessable?
Is cirrhosis reported, and are cirrhotic morphological features visually assessable?
Is a focal hepatic lesion reported, and is it visually assessable?
Appendix
Table S3 : List of closed-ended Questions for VQA
Table S4 : Qualitative case studies (success): the judge overrides erroneous specialist readings (patients 225145 and 219190, Dataset-1). In both cases the radiology specialist reads the liver as normal and the VQA specialist answers the posterior-attenuation question incorrectly ( red ), yet the consolidated report re-establishes the reference findings marked in green , namely increased parenchymal echogenicity, a coarse or heterogeneous echo-texture, and the plump hepatic contour, because the remaining evidence outputs contradict the erroneous reading. [ … ] marks elided template organ sentences (bile ducts, portal vein, gallbladder, pancreas, spleen).
Table S5 : Qualitative case studies (failure): correct findings absent from the available evidence propagate through consensus (patients 216747 and 235520, Dataset-1). A prior cholecystectomy (top) and a right-lobe cystic structure (bottom) are recorded in the reference ( underlined ) but the correct findings are unavailable in the specialist evidence, so the consolidated report contradicts or omits them ( red ). This delimits the mechanism: consensus filtering is effective when corroborating evidence is available but remains limited when the correct attribute is absent from all specialist outputs. Because the surrounding template text is reproduced correctly, n -gram overlap with the reference is largely unaffected, illustrating the limitation of report-similarity metrics (Sec. Supplementary Note B: Clinical Evaluation Protocol ). SCM is the corpus’ masked-measurement token.
Figure S1 : Qualitative case studies for closed-set VQA. For each image and question we compare the responses generated by our multi-agent council against the reference answer. Red indicates incorrect predictions, green indicates correct predictions, and orange highlights complementary information contributed by individual agents during the reasoning process. Examples (a) and (b) are taken from Dataset-1 and Dataset-2, respectively.
Figure S2 : The complete consensus-filtering judge prompt. System and user messages issued to the SMILE judge for every patient, verbatim from the inference code; only the per-case content is replaced by ⟨ slots ⟩ . The three rule families audited in rules_check (R1: non-imaging content, R2: diagnosis claims, R3: agent/VQA mentions) mirror hard constraints 1–3, and the mandatory incorporation rule couples the caption to Agent C’s three parenchymal attributes.
Figure S3 : Effect of increasing single-model adaptation capacity. Performance of MedGemma-4B with increasing LoRA rank is compared with SMILE on Dataset-1. Increasing the trainable capacity improves the single-model baseline, while SMILE retains stronger ROUGE-1, ROUGE-L, and CE-F1 performance at a lower trainable parameter budget.
Component
Backbone
Rank
Trainable
Agent A (radiology)
MedGemma-4B
r=16
38.50 M
Agent B (pathology)
Qwen2-VL-2B
r=8
14.61 M
Agent C (VQA) ∗
Qwen2-VL-2B
r=8
2.52 M
Judge
MedGemma-4B
frozen
0.00 M
SMILE total
55.63 M
Appendix
Table S6 : Where SMILE’s trainable parameters sit. The aggregation step is frozen, so extending the council with a new specialist requires only the specialist adapter and no judge retraining. The parameters are partitioned across independently trained, individually reusable specialists: the largest single training run ( 38.50 M) is smaller than the MedGemma-4B ( 77.00 M) and MedMO-4B ( 80.54 M) single-model baselines, each of which trains one r=32 adapter in a single run. ∗ Agent C adapts a restricted module set (query and key projections only), which is why its budget is well below Agent B’s at the same rank.
Computer Aided Medical Procedures (CAMP), TU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Zhongshan Hospital, Fudan University, China +1