Organizations: University at Buffalo, Buffalo NY 14260, USA · ELLIS Institute Finland, Espoo 02150, Finland · Aalto University, Espoo 02150, Finland · Aarhus University, Aarhus 8000, Denmark
Accurate 3D anatomical segmentation is essential for clinical diagnosis and surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.
Figures & tables
Figure 1: The overall workflow of iterative text-based refinement for 3D medical segmentation shapes. The green arrows and 6 steps represent the iterative workflow.
Figure 2: Centerline refinement and Segment Selection. A to E shows the steps for refining the centerline representation, while F illustrates the partial selection of a segment.
Figure 3: Examples of generated instructions in CoWTalk .
Figure 4: Overview of the proposed instruction-guided medical shape refinement pipeline.
Figure 5: Qualitative refinement results with shape-only input from CoWTalk , and only 3 corrections are highlighted for each sample. Some correction descriptions are omitted.
Figure 6: Refinement (F1 %) with the proposed method under different visual and language input, showing the difference between refined results and synthesized error ( ± ).
Figure 7: Example of refining a real, suboptimal segmentation [ 6 ] prediction using instructions given by a layperson and a radiologist.