Authors: Sam Church, Danyal Maqbool, Joshua D. Warner, Andrew Voter, Junjie Hu, Meghan G. Lubner, Tyler J. Bradshaw
Organizations: University of Wisconsin–Madison, Department of Computer Sciences, Madison, WI USA · University of Wisconsin–Madison, Department of Radiology, Madison, WI USA
Visual grounding models can enhance radiology workflows by linking report findings to image regions. This is particularly valuable for 3D CT, where findings often occupy a tiny fraction of the volume. Training 3D grounding models requires large sets of paired phrases and regions, and building such datasets is expensive, requiring radiologists to annotate images by hand. We posit that this supervision is already created implicitly during routine reporting, as radiologists frequently place 2D annotations (e.g., distance measurement, arrows) on key images to make measurements and to support report interpretation. We introduce an automated pipeline that converts these routine clinical annotations into large-scale phrase-region supervision for 3D visual grounding. The pipeline links each annotation to the corresponding finding in the report through metadata matching, then uses a promptable 3D segmentation model to convert the 2D annotation into a volumetric mask. This produces phrase-mask-volume datasets without requiring additional radiologist annotation. Applied to a single institution's clinical picture archiving and communication system (PACS), our approach generated 105K phrase-mask-volume triplets from 59K abdominal CT exams. We also introduce two abdominal CT grounding benchmarks, LocusBench-Onc and LocusBench-ED, which comprise 240 oncology and 260 emergency-department radiologist-reviewed phrase-mask-volume triplets, respectively, with the latter spanning 13 distinct categories such as appendicitis, hematoma, and hernia. We further introduce LocusCT, a 3D visual grounding model trained on this dataset, which achieves hit rates of 0.725 on LocusBench-Onc and 0.773 on LocusBench-ED, substantially outperforming comparator models. These results show that routine PACS annotations are a scalable, previously unused source of supervision for 3D visual grounding.
Figures & tables
Figure 1: Visual grounding of clinical findings in 3D abdominal CT. A radiology report contains multiple findings referring to distinct anatomical regions within the CT volume. Colored phrases in the report correspond to their respective localized regions in the 3D abdominal CT, illustrated by the matching colored boxes and volume crops.
Figure 2: Automated generation of phrase–mask–volume triplets from routine clinical data. An LLM extracts finding information from free-text reports, which are matched to 2D GSPS annotations using exam metadata. A promptable 3D segmentation model converts the matched 2D annotation into a volumetric mask, yielding grounding supervision without additional radiologist annotation.
Figure 3: Our method provides large-scale, report-linked supervision for 3D CT grounding. (a) Composition of the training data for VoxTell and LocusCT. We call a volume report-linked if its segmentation masks are paired with finding phrases from the exam’s own clinical report. Only 5% of VoxTell’s volumes (13% of its CT volumes) are report-linked, whereas all 65K volumes (from 59K exams) in LocusCT’s training set are. (b) Distribution of the 105K findings across the 20 most frequent categories, which together cover 88.1% of all findings.
LocusBench-Onc
LocusBench-ED
Model
Dice
Hit Rate
Dice
Hit Rate
VoxTell
0.117
0.188
0.118
0.209
SegVol
0.015
0.044
0.016
0.037
BiomedParse-v2
0.065
0.158
0.067
0.182
Table 1: Comparison of 3D visual grounding models under zero-shot settings.
LocusCT ConTEXTualNet
LocusCT VoxTell
Dice
Hit Rate
Dice
Hit Rate
LocusBench-Onc
0.462
0.596
0.495
0.725
LocusBench-ED
0.468
0.731
0.490
0.773
LocusBench-ED by category
Cyst
0.735
0.950
0.685
0.950
Aneurysm
0.608
0.900
0.654
0.900
Table 2: Dice and hit rate on LocusBench. Per-category results on LocusBench-ED are sorted by LocusCT VoxTell Dice. Diverticulitis is the most challenging category for both models, likely because its margins blend into adjacent inflammation. Aneurysm, cyst, and mass achieve the highest Dice.
LocusBench-Onc
LocusBench-ED
Ablation
Data
Ch.
Dice
Hit Rate
Dice
Hit Rate
Data scale
10%
3
0.267
0.458
0.282
0.562
30%
3
0.382
0.550
0.409
0.681
Input channels
30%
1
0.353
0.575
0.373
0.654
Training source
Curated only
3
0.391
0.596
0.386
0.658
LocusCT (full)
100%
3
0.495
0.725
0.490
0.773
Table 3: Ablations of LocusCT on LocusBench-Onc and LocusBench-ED. All models use the LocusCT architecture. Data is the fraction of the full training set used; Ch. is the number of CT window input channels (3 = soft tissue, lung, bone). The last row is our released model.
Figure 4: LocusCT predictions on LocusBench-Onc and LocusBench-ED. Each example shows a clinical finding description alongside a representative CT slice with the model prediction (red), radiologist-annotated ground truth (green), and Dice score.
Figure 5: LocusCT predictions on the external Merlin dataset. Report phrases (left) and predicted masks in red (right), across eight ED categories. Merlin has no mask annotations, so these examples are qualitative only.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Cohort
Onc
ED
OED
All
Report coverage
Total reports
25,987
31,351
33,803
91,141
with ≥ 1 matchable sentence
22,210
(85.5%)
27,234
(86.9%)
27,808
(82.3%)
77,252
(84.8%)
Matchable sentences
61,246
55,050
83,489
199,785
per matchable report
2.76
2.02
3.00
2.59
Appendix
Table 4: Report sentence matching by cohort. Cohorts are disjoint; All is their sum. Gray percentages give each count as a share of its parent: total reports, matchable sentences, or current-measurement sentences. Match types are not mutually exclusive, so a sentence may be counted under more than one type.
Matched to
Converted to
Annotation type
Total
sentence
3D segmentation
Total
302,868
129,134
(42.6%)
109,401
(36.1%)
Unidimensional line
117,118
72,884
(62.2%)
63,184
(53.9%)
Arrow
112,388
29,958
(26.7%)
25,106
(22.3%)
Bidimensional lines
34,303
24,685
(72.0%)
21,111
(61.5%)
Ellipse
37,662
1,504
(4.0%)
—
Appendix
Table 5: GSPS annotations by type, pooled across cohorts. Gray percentages are shares of all annotations of that type. Other combines Text, Angle, and NPoint annotations. Ellipse and other annotations were not converted.
Cohort
Onc
ED
OED
All
All GSPS annotations
Total
88,903
109,088
104,877
302,868
Unidimensional line
34,855
38,246
44,017
117,118
Arrow
36,109
44,177
32,102
112,388
Bidimensional lines
13,095
9,317
11,891
34,303
Appendix
Table 6: GSPS annotations converted to 3D segmentations after sentence matching, by cohort and annotation type. Cohorts are disjoint; All is their sum. Gray percentages are conversion rates: converted annotations as a share of all annotations of that type. Ellipse and other annotations were not converted.
Figure 6: Representative GSPS to 3D segmentation failure modes. Yellow marks show the radiologist’s GSPS annotation, magenta shows the predicted mask, and the matched report sentence (partial) appears below each panel. (a) Shifted, minor error : the mask covers the correct nodule but is offset slightly from the annotation. (b) Shifted, major error : the mask lands on nearby tissue and largely misses the annotated pancreatic mass. (c) Misread arrow : instead of the small lesion at the arrow tip, the mask segments a larger adjacent structure. (d) Off-body : the annotation falls outside the patient, and the mask fragments across background noise.
Figure 7: Dice score distributions for LocusCT on LocusBench-ED ( n=260 ) and LocusBench-Onc ( n=240 ) . The two splits have similar medians (0.59 ED, 0.58 Onc), but both are bimodal: most cases cluster at high Dice, with a second group near zero corresponding to complete localization failures. LocusBench-Onc has more of these failures, so its lower quartile falls near zero (0.04, vs. 0.17 for ED).
Figure 8: Per-category Dice scores for LocusCT on LocusBench-ED, sorted by median Dice. Each category contains 20 cases (shown in parentheses). Points are individual cases, boxes span the interquartile range, and black lines mark the median. Many categories are bimodal: most cases are segmented well, but some score near zero, meaning the model failed to localize the target finding at all rather than producing a poor boundary. Performance is strongest for mass, aneurysm, and cyst and weakest for diverticulitis and abscess.
Figure 9: Reader study of LocusCT grounding on the external Merlin dataset. For each of the 13 LocusBench-ED finding types, six abdominal CT scans from the Merlin dataset were grounded with LocusCT using the corresponding report phrase, and each predicted mask was rated by a board-certified radiologist on a 5-point scale (1 = Missing/Not near the object of interest; 2= Incorrect, but near the object of interest; 3 = Useful for localization, but not for information extraction; 4 = Useful, only minor edits required for clinical measurements; 5 = Very useful: could be used for clinical measurements). Bars show the distribution of ratings per finding type, sorted by mean rating. Hatched segments mark cases where the model produced no prediction; these are assigned a score of 1 and included in the means. The bottom row pools all 78 cases.
Accurate 3D abnormality segmentation in chest CT requires dense spatial supervision, but obtaining expert voxel-level labels is costly. Radiology reports, however, are routinely generated during clinical interpretation and contain instance-specific descriptions that can provide additional guidance without new dense annotation. Existing vision-language grounding methods typically require report-derived findings at inference, making localization dependent on paired text and limiting each forward pass to a queried finding. We propose Instance-Guided Report Anchoring (IGRA), a model-agnostic module that preserves the correspondence between each annotated abnormality instance and the report finding that describes it. IGRA pools each instance representation and anchors it to the corresponding finding embedding during training; all text-related components are discarded at inference. We further reformulate free-text grounding on ReXGroundingCT as multi-label volumetric segmentation by merging same-category instances, allowing all abnormality categories to be predicted in one image-only forward pass. IGRA improves Dice by 22.5% over the strongest image-only baseline (30.93 vs. 25.25) and is comparable to VoxTell on the single-finding subset (30.29 vs. 30.43). Applied unchanged to four standard 3D segmentation backbones, IGRA improves Dice and hit rate across all architectures. Zero-shot evaluation on LIDC-IDRI, PleThora, and a private in-house dataset further shows consistent gains over image-only baselines.
Zhenyu Bu, Haoyan Ding, Chushu Shen +7
Siemens Medical Solutions USA, Inc., Malvern, PA, USA · The Ohio State University, Columbus, OH, USA · University of California, Los Angeles, Los Angeles, CA, USA +2
Grounding radiology report descriptions to 3D CT volumes is essential for verifiable clinical interpretation, yet remains challenging due to the semantic-spatial gap between free-text narratives and volumetric anatomy. Existing report-assisted and vision-language grounding methods typically rely on phrase-level alignment or dense pixel supervision, resulting in limited lesion-wise correspondence and suboptimal localization accuracy. We propose GLeVE, a graph-guided lesion grounding framework with anatomical prior verification and octree-based autoregressive refinement. GLeVE treats each lesion description as an atomic semantic unit and encodes organ attribution, attributes, and inter-lesion relations through relation-aware graph reasoning to produce discriminative lesion-wise queries. Anatomy-aware proposal generation with region-level verification enforces one-to-one text-lesion alignment, while hierarchical octree refinement progressively improves boundary delineation. Experiments on AbdomenAtlas 3.0 demonstrate consistent gains over classical multimodal foundation models and report-supervised baselines in both segmentation accuracy and lesion-level localization.
Shuo Jiang, Yuhao Hong, Chunbo Jiang +9
Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University, Hangzhou, China · Zhejiang University, Hangzhou, China · Tsinghua University, Beijing, China +1
Automatic voxel-level grounding of free-text findings in 3D chest Computed Tomography (CT) is critical for clinical interpretability. However, this task remains highly challenging due to the intricate spatial complexity of large 3D volumes and the heterogeneity of free-text findings. Existing end-to-end approaches often struggle to simultaneously learn the localized feature representations required for accurate 3D segmentation and the complex semantic understanding needed for text alignment, leading to suboptimal grounding performance. To overcome this fundamental limitation, we propose a novel decoupled framework that disentangles the problem into two specialized stages: (1) class-agnostic lesion segmentation and (2) text-volume reasoning. This structural separation allows the model to first extract candidate sub-volumes by localizing potential abnormalities. Subsequently, intensive cross-modal reasoning is performed to align these localized sub-volumes with free-text medical findings. To resolve the spatial ambiguities inherent in local regions, the reasoning module is augmented with explicit anatomical guidance, utilizing relative spatial coordinates and lung lobe priors. Evaluated on the ReXGroundingCT benchmark, our method achieves state-of-the-art performance in overall grounding quality on the official leaderboard. These results demonstrate that decoupling detection from reasoning is a highly effective paradigm for handling the complexity of 3D medical visual grounding. Our code is publicly available at https://github.com/khuhm/DAGG.
Kwang-Hyun Uhm, Inhwa Son, Sung-Jea Ko
Department of Artificial Intelligence, Gachon University, Republic of Korea · MEDAI, Republic of Korea