Understanding Task Aggregation for Generalizable Ultrasound Foundation Models
Authors: Fangyijie Wang, Tanya Akumu, Vien Ngoc Dang, Amelia Jiménez-Sánchez, Jieyun Bai, Guénolé Silvestre, Karim Lekadir, Kathleen M. Curran
Organizations: Research Ireland Centre for Research Training in Machine Learning · School of Medicine, University College Dublin, Dublin, Ireland · Departament de Matemàtiques i Informàtica, Universitat de Barcelona, Barcelona, Spain · Department of Cardiovascular Surgery, The First Affiliated Hospital of Jinan University, Jinan University, Guangzhou, China · Auckland Bioengineering Institute, University of Auckland, Auckland, New Zealand · School of Computer Science, University College Dublin, Dublin, Ireland · Institució Catalana de Recerca i Estudis Avançats (ICREA)
Foundation models promise to unify multiple clinical tasks within a single framework, but recent ultrasound studies report that unified models can underperform task-specific baselines. We hypothesize that this degradation arises not from model capacity limitations, but from task aggregation strategies that ignore interactions between task heterogeneity and available training data scale. In this work, we systematically analyze when heterogeneous ultrasound tasks can be jointly learned without performance loss, establishing practical criteria for task aggregation in unified clinical imaging models. We introduce M2DINO, a multi-organ, multi-task framework built on DINOv3 with task-conditioned Mixture-of-Experts blocks for adaptive capacity allocation. We systematically evaluate 27 ultrasound tasks spanning segmentation, classification, detection, and regression under three paradigms: task-specific, clinically-grouped, and all-task unified training. Our results show that aggregation effectiveness depends strongly on training data scale. While clinically-grouped training can improve performance in data-rich settings, it may induce substantial negative transfer in low-data settings. In contrast, all-task unified training exhibits more consistent performance across clinical groups. We further observe that task sensitivity varies by task type in our experiments: segmentation shows the largest performance drops compared with regression and classification. These findings provide practical guidance for ultrasound foundation models, emphasizing that aggregation strategies should jointly consider training data availability and task characteristics rather than relying on clinical taxonomy alone.
Figures & tables
Training
MoE
#
#
Reg
Cls
Seg
Det
Target
Clinical
Paradigm
Tasks
Images
Anatomy
Group
TS
✗
1
1,144
✓
Breast
Breast
TS
✗
1
1,776
✓
Breast
Breast
TS
✗
1
208
✓
Cervical
OB
TS
✗
1
2,818
✓
PS/Fetal head
OB
TS
✗
1
762
✓
Fetal abdomen
OB
Table 1: Experimental settings for evaluating different training paradigms. Seg: segmentation; Cls: classification; Reg: regression; Det: detection. MO: Multi-organ.
Figure 1: Overview of our M2DINO framework. (a) Ultrasound images are processed by a shared DINOv3 encoder augmented with task-conditioned MoE blocks. The unified representation is optimized for segmentation, detection, regression, and classification via task-specific prediction heads. Frozen and trainable components are indicated. (b) A conceptual comparison of the three training paradigms. Although the architecture remains the same, TS ( TS ), CG ( CG ), and AU ( AU ) differ in how tasks are aggregated during training and in whether the MoE is enabled.
Figure 2: Absolute performance of TS , CG , and AU training paradigms across representative tasks: segmentation ( DSC ↑ ), classification ( AUC ↑ ), and regression ( MRE ↓ ). Abd: Abdomen; MO: Multi-organ.
Ultrasound images vary widely across scanners, operators, and anatomical targets, which often causes models trained in one setting to generalize poorly to new hospitals and clinical conditions. The Foundation Model Challenge for Ultrasound Image Analysis (FMC-UIA) reflects this difficulty by requiring a single model to handle multiple tasks, including segmentation, detection, classification, and landmark regression across diverse organs and datasets. We propose a unified multi-task framework based on a transformer visual encoder from the Qwen3-VL family. Intermediate token features are projected into spatial feature maps and fused using a lightweight multi-scale feature pyramid, enabling both pixel-level predictions and global reasoning within a shared representation. Each task is handled by a small task-specific prediction head, while training uses task-aware sampling and selective loss balancing to manage heterogeneous supervision and reduce task imbalance. Our method is designed to be simple to optimize and adaptable across a wide range of ultrasound analysis tasks. The performance improved from 67% to 85% on the validation set and achieved an average score of 81.84% on the official test set across all tasks. The code is publicly available at: https://github.com/saitejalekkala33/FMCUIA-ISBI.git
Ufaq Khan, L. D. M. S. Sai Teja, Ayuba Shakiru +4
MBZUAI, Abu Dhabi, UAE · NIT Silchar, India · Birmingham City University, UK
Ultrasound image analysis plays a crucial role in cancer screening and prenatal diagnosis, yet comprehensive assessment requires jointly addressing tasks such as lesion segmentation and benign-malignant classification. While recent vision foundation models have shown remarkable universal representations, unlocking their potential for ultrasound is bottlenecked by the considerable domain gap from natural images. Existing methods typically fine-tune heavy vision encoders for isolated tasks, incurring substantial computational overhead while overlooking the underlying commonalities across heterogeneous tasks. In this work, we propose FreqDINO++, a frequency-guided multi-task routing vision foundation model for universal ultrasound analysis. We first introduce a Multi-task Routing Adapter (MR-Adapter) to support parameter-efficient integration of task-common and task-specific knowledge, a Frequency-aware Feature Enhancer (F2-Enhancer) is then designed to capture the rich multi-scale frequency characteristics of ultrasound images, and a Task-aligned Collaborative Decoder (TC-Decoder) is devised to promote collaboration between dense and global prediction tasks through global-local token interaction. Extensive experiments on large-scale multi-task and external single-task ultrasound benchmarks demonstrate that FreqDINO++ consistently outperforms strong baselines and recent foundation models across 27 diverse clinical task scenarios, while also showing promising generalization to unseen data. The code is at https://github.com/MingLang-FD/FreqDINO-Plus.
Qing Xu, Yixuan Zhang, Yue Li +7
School of Computer Science, University of Nottingham Ningbo China, Ningbo, Zhejiang, China · School of Mathematical Sciences, University of Nottingham Ningbo China, Ningbo, Zhejiang, China · School of Computer Science, University of Nottingham, Nottingham NG72RD, UK +3
Benchmarking is an increasingly critical part of research in machine learning and the domains where it is applied, including healthcare. Yet, despite the steady development of new ultrasound foundation models in recent years, the development of well-designed benchmarks to evaluate them has lagged behind. This deficiency has led to fragmented and inconsistent evaluations of competing models, making it difficult to measure progress. To address this issue, we introduce UltraBench 2, a comprehensive benchmark with wide anatomical and task coverage, and a focus on standardization, reproducibility, and ease-of-use. Using this benchmark, we compare existing vision foundation models for ultrasound image analysis. Our analyses demonstrate that ultrasound-specific pretraining still leads on classification, but that state-of-the-art general-purpose models have drawn level on segmentation.
Ashwath Radhachandran, Adam Tupper, Christian Gagné +1
Bioengineering Department, University of California, Los Angeles · Institut Intelligence et Données (IID), Université Laval Mila – Quebec AI Institute · Radiological Sciences Department, UCLA David Geffen School of Medicine