Deep EM with Hierarchical Latent Label Modelling for Multi-Site Prostate Lesion Segmentation

Automated 3D segmentation of prostate lesions from biparametric MRI (bp-MRI) is essential for reliable algorithmic analysis, but achieving high precision remains challenging. Volumetric methods must combine multiple modalities while ensuring anatomical consistency, but current models struggle to integrate cross-modal information reliably. While vision-language models (VLMs) are replacing the currently used architectural designs, they still lack the fine-grained, lesion-level semantics required for effective localized guidance. To address these limitations, we propose a new multi-encoder U-Net architecture incorporating three key innovations: (1) an alignment loss that enhances foreground text-image similarity to inject lesion semantics; (2) a heatmap loss that calibrates the similarity map and suppresses spurious background activations; and (3) a final-stage, confidence-gated multi-head cross-attention refiner that performs localized boundary edits in high-confidence regions. A phase-scheduled training regime stabilizes the optimization of these components. Our method consistently outperforms prior approaches, establishing a new state-of-the-art on the PI-CAI dataset through enhanced multi-modal fusion and localized text guidance. Our code is available at https://github.com/NUBagciLab/Prostate-Lesion-Segmentation.

Purpose: Accurate segmentation of prostate cancer on magnetic resonance (MR) images is crucial for planning image-guided interventions such as targeted biopsies, cryoablation, and radiotherapy. However, subtle and variable tumour appearances, differences in imaging protocols, and limited expert availability make consistent interpretation difficult. While automated methods aim to address this, they rely on large expertly-annotated datasets that are often inconsistent, whereas manual delineation remains labour-intensive. This work aims to bridge the gap between automated and manual segmentation through a framework driven by user-provided point prompts, enabling accurate segmentation with minimal annotation effort. Methods: The framework combines reinforcement learning (RL) with a region-growing segmentation process guided by user prompts. Starting from an initial point prompt, region-growing generates a preliminary segmentation, which is iteratively refined through RL. At each step, the RL agent observes the image and current segmentation to predict a new point, from which region growing updates the mask. A reward, balancing segmentation accuracy and voxel-wise uncertainty, encourages exploration of ambiguous regions, allowing the agent to escape local optima and perform sample-specific optimisation. Despite requiring fully supervised training, the framework bridges manual and fully automated segmentation at inference by substantially reducing user effort while outperforming current fully automated methods. Results: The framework was evaluated on two public prostate MR datasets (PROMIS and PICAI, with 566 and 1090 cases). It outperformed the previous best automated methods by 9.9% and 8.9%, respectively, with performance comparable to manual radiologist segmentation, reducing annotation time tenfold.

Accurate lesion segmentation in mammography is important for computer-aided breast cancer diagnosis, but dense pixel-level annotations remain costly and limited. Existing public segmentation datasets usually cover only a small set of lesion types with manually delineated masks, whereas larger mammography datasets often provide only image-level diagnostic labels. This annotation mismatch makes it difficult to train segmentation models that can generalize to rare or weakly annotated lesion categories. To address this problem, we study open-vocabulary mammographic lesion segmentation under a mixed-supervision setting, where pixel-level masks are available for base lesion categories and image-level labels are available for a broader set of categories. We propose a two-branch framework that jointly learns from segmentation data and classification data. The segmentation branch learns lesion boundaries from annotated masks, while the classification branch introduces additional image-level semantic supervision. To better transfer image-level supervision to lesion-relevant regions, we further introduce a Lesion-wise Attentive Classifier that maintains category-specific feature prototypes and generates lesion-aware attention maps for foreground feature pooling. This design reduces the influence of background tissue and encourages more compact lesion representations. Experiments on multiple mammography datasets show that the proposed method consistently improves open-vocabulary segmentation baselines on both base and novel lesion categories, while reducing the gap to fully supervised models that require dense target-domain masks. Qualitative results further indicate that our method produces more complete lesion localization for small, low-contrast, and visually subtle mammographic findings.