Authors: Pedro R. A. S. Bassia, Wenxuan Li, Jakob Wasserthal, Jieneng Chen, Xinze Zhou, Zheren Zhu, Chuntung Zhuanga, Sergio Decherchi, +5 more
Organizations: Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Clinic of Radiology and Nuclear Medicine, University Hospital Basel, Basel, Switzerland · Department of Computer Science, Stanford University, Stanford, CA, USA · Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, USA · Computational and Chemical Biology, Italian Institute of Technology, Genoa, Italy · Department of Pharmacy and Biotechnology, University of Bologna, Bologna, Italy · Centre Européen de Calcul Atomique et Moléculaire (CECAM), École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland · Department of Oncology, Johns Hopkins Medicine, Baltimore, MD, USA
Segmentation models can surpass radiologists, classification models, and vision-language models in tumor detection. Importantly, segmentation models outline tumors, allowing radiologists to better verify and trust the AI output. Their main limitation is the scarcity of tumor masks: creating one 3D tumor mask takes up to 30 minutes, so most public CT datasets contain only a few hundred masks, and even the largest private datasets contain only a couple of thousand. Tumor masks are not produced in clinical routine, but radiology reports are. Public datasets contain tens of thousands of CT-Report pairs, and hospitals contain hundreds of thousands. These reports describe tumors in detail, providing large-scale, informative training data. Here, we introduce Report Supervision (R-Super), a training framework that uses reports to directly supervise and improve tumor segmentation. R-Super introduces new loss functions that teach segmentation models to segment tumors that match report descriptions of tumor count, sizes, and locations. Reports are only used for training. We evaluated R-Super on kidney and pancreatic tumor segmentation, exploring diverse training data sizes, up to 41,418 CT-Report plus 3,488 pancreatic tumor CT-Mask pairs. On external validation, R-Super increased tumor detection F1-Score and segmentation DSC by up to +15% with respect to mask-only training. It also surpassed alternative methods such as CLIP and multi-task learning. Leveraging numerous readily available reports to supplement scarce masks, R-Super strongly improves AI performance when very few training masks are available (e.g., 50), and when many masks are available (e.g., 3,488), unlocking scale in tumor segmentation.
Figures & tables
Figure 1: R-Super trains tumor segmentation models using information from radiology reports, with Volume Loss and Ball Loss guiding predictions to match reported tumor size, location, and number. (a) The Volume Loss (deep supervision) aligns segmented tumors with volumes and locations from reports. It consists of three steps. (I) A large language model (LLM) extracts tumor location (organ/sub-segment) and diameters from the report. Using ellipsoid approximations, we estimate the total tumor volume in each organ o as Vr,o . (II) From a deep layer of the segmenter, we compute tumor probabilities via a 1 × 1 × 1 convolution and sigmoid/softmax. We sum the probabilities inside the organ (using pre-saved organ masks) to estimate the segmented tumor volume, Vs,o . (III) The Volume Loss penalizes discrepancies between Vs,o and Vr,o . As shown in the right panel (for Vr,o=1000 mm 3 ), the loss is zero when Vs,o is close to Vr,o , allowing for some tolerance due to uncertainty in report-based volume estimates. (b) The Ball Loss enforces segmented tumors to match detailed tumor information from reports—tumor count, locations, diameters, and volumes. It consists of four steps. (I) A large language model (LLM) extracts this tumor information from the report. (II) A Ball Convolution—a simple convolution with a fixed spherical kernel matching the reported tumor diameter—slides over the segmenter output to find the highest-probability tumor region. (III) Within this region, the loss maximizes probabilities in the top- Ni most probable voxels, where Ni is the tumor volume estimated from the report (in voxels, reduced by 20%). These voxels can form any shape that stays within the ball. (IV) The process (II–III) is repeated for all reported tumors, largest to smallest. Already-assigned voxels are ignored in later iterations to prevent overlap. Finally, the loss minimizes tumor probabilities in unassigned voxels (background). It focuses more on central tumor voxels and avoids penalizing uncertain tumor borders.
Figure 2: Dataset overview (Secs. 4.1 to 4.3 ). Summary of all datasets, including CT-Mask and CT-Report cohorts from multiple institutions. In (I), (II) and (III), we show the number of tumor scans, tumor types (pancreatic and kidney), and distributions of tumor size, tumor count per scan, and patient demographics (age and sex) across training and test sets (UCSF, JHH, and public datasets). Radiology reports enable large-scale dataset construction ( Li et al., 2025a ) , which R-Super leverages for segmentation training.
Figure 3: By learning from reports, R-Super substantially outperforms five state-of-the-art methods on both internal (UCSF-Test) and external (JHH-Test) validation (all six compared methods are in Tab. 1 ). Green arrows show the gain of R-Super over the best competing method at each mask budget, up to +14% in tumor detection F1-Score and +11% in DSC; over standard segmentation (no reports) alone, the gain reaches +16% in F1-Score (Tab. 1 ). Each panel corresponds to one tumor type and one test set, and its x-axis shows the number of training masks (50 from AbdomenAtlas-Subset; 344 or 1.7K from AbdomenAtlas 3.0; the axis labels 350 and 1750 are rounded). R-Super and the other report-based methods shown (CLIP, MTL) supplement these masks with 2.2K pancreatic and 2.7K kidney tumor reports from UCSF-Train. Both test sets are challenging: JHH-Test is out-of-distribution relative to training, and UCSF-Test spans a wide range of CT resolutions and contrast phases, including non-contrast scans. DSC is only reported for JHH-Test because UCSF-Test does not include ground-truth tumor masks, needed for calculating DSC.
pancreas tumor
kidney tumor
JHH-Test
UCSF-Test
UCSF-Test
train paradigm
mask
rep.
DSC
NSD
F1
AUC
Se
Sp
F1
AUC
Se
Sp
mask
rep.
F1
AUC
Se
Sp
few training masks (50)
standard segmentation
50
0
38
41
62
63
62
62
51
63
47
77
50
0
64
68
68
63
R-Super (ours)
50
2.2K
49
52
67
75
68
64
61
70
73
59
50
2.7K
65
66
73
55
medium / many training masks (344 / 1.7K)
Table 1: R-Super outperforms state-of-the-art training methods on tumor detection and segmentation. On pancreatic tumors, R-Super achieves the highest DSC, NSD, F1-Score, and AUC across all baselines. On kidney tumors, it matches mask-based segmentation when few masks are available (50) and surpasses it when many are available (1.7K). DSC and NSD are computed on scans with tumors and reported only in datasets where masks exist (UCSF-Test has none). Sensitivity (Se) and Specificity (Sp) correspond to the threshold that maximizes F1-Score. Statistical tests were performed to compare F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC). Orange highlights statistically significant gains over standard segmentation ( p<0.05 ). Standard deviations and confidence intervals are available in Tab. 6 (Appendix C ).
Figure 4: Tumors segmented by R-Super. The examples show two cases of small pancreatic adenocarcinoma and one case of small renal carcinoma (bottom) correctly segmented by R-Super. Radiology reports are shown, but they are used only in training, not at inference.
pancreas tumor
kidney tumor
JHH-Test
UCSF-Test
UCSF-Test
train paradigm
mask
rep.
DSC
NSD
F1
AUC
Se
Sp
F1
AUC
Se
Sp
mask
rep.
F1
AUC
Se
Sp
small tumors (diameter ≤ 2 cm)
few training masks (50)
standard segmentation
50
0
7
17
38
61
53
62
41
59
40
77
50
0
43
59
53
63
R-Super (ours)
50
2.2K
15
25
35
68
47
64
54
72
76
59
50
2.7K
47
61
65
55
Table 2: R-Super outperforms state-of-the-art methods on both small (diameter ≤ 2 cm) and large tumors. For pancreatic tumors (344 training masks), R-Super achieves the highest DSC, NSD, and F1-Score across all baselines in both small ( ≤ 2 cm) and large ( > 2 cm) tumors, and the highest AUC except for small tumors in JHH-Test (CLIP-Like 90 vs. 89). For kidney tumors (1.7K training masks), R-Super leads in F1-Score and AUC on small tumors and in F1-Score on large tumors. DSC and NSD are computed on scans with tumors and reported only where masks exist (UCSF-Test has none). Sensitivity (Se) and Specificity (Sp) correspond to the threshold that maximizes F1-Score. Statistical tests were performed to compare F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC). Orange highlights statistically significant gains over standard segmentation ( p<0.05 ). Test set sizes: JHH-Test contains 15 small and 35 large pancreatic tumors; UCSF-Test contains 90 small and 28 large pancreatic tumors (21 CTs with tumors of unknown size were excluded from this table) and 77 small and 65 large kidney tumors (27 excluded). Tumor sizes were extracted from reports in UCSF-Test and from masks in JHH-Test. Standard deviations and confidence intervals are available in Tab. 7 (Appendix C ).
training
JHH-Small
JHH-Large
Merlin-Test
train paradigm
mask
rep.
DSC
NSD
F1
AUC
Se
Sp
DSC
NSD
F1
AUC
Se
Sp
F1
AUC
Se
Sp
nnU-Net
926
0
33
58
83
91
80
87
49
72
88
93
84
92
69
76
68
73
Multi-task learning
926
1,848
33
57
92
95
92
92
50
73
97
99
98
96
69
75
63
82
standard segmentation
926
0
31
55
91
94
86
98
50
73
99
100
98
100
67
74
60
82
R-Super (ours)
926
1,848
34
59
95
97
93
98
52
74
98
100
100
96
78
83
77
79
Table 3: R-Super sets a new state-of-the-art on the largest public pancreatic tumor dataset. Combining CT-Mask pairs from PanTS (the largest public pancreatic tumor CT-Mask dataset) with CT-Report pairs from Merlin (the largest public abdominal CT-Report dataset), R-Super outperforms a segmentation model trained on CT-Mask pairs alone (PanTS) by +7% sensitivity on small pancreatic tumors. R-Super also surpasses all alternative models on DSC and NSD, and on F1-Score and AUC in JHH-Small and Merlin-Test (in JHH-Large, all methods saturate at F1-Score 97–99 and AUC 99–100). All models were trained on the same 926 pancreatic tumor CT-Mask pairs from PanTS. Multi-task learning and R-Super additionally trained on Merlin-Train (1,848 pancreatic tumor and 1,848 no-tumor CT-Report pairs from Merlin). DSC and NSD are computed on scans with tumors and reported only where masks exist (Merlin-Test has none). Sensitivity (Se), Specificity (Sp), and F1-Score correspond to the threshold that maximizes F1-Score for each model. Statistical tests were performed to compare R-Super and standard segmentation for F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC); orange highlights statistically significant gains ( p<0.05 ). Standard deviations and confidence intervals are available in Tab. 8 (Appendix C ).
training
pancreas tumor
train paradigm
mask
rep.
F1
AUC
Se
Sp
Swiss-Test
nnU-Net
3,488
0
78
74
74
71
Multi-task learning
3,488
28,295
74
76
64
85
standard segmentation
3,488
0
80
82
72
86
R-Super (ours)
3,488
28,295
85
85
80
86
Table 4: R-Super surpasses mask-based training when large amounts of both reports and masks are available. On Swiss-Test, an external dataset which we publicly release, R-Super outperforms standard mask-based segmentation by +8% sensitivity at the same specificity, for pancreatic tumor detection. On UCSF-Test-Non-Contrast, an internal test set of non-contrast CT scans, R-Super outperforms the standard segmentation model by +19% sensitivity at the same specificity. DSC and NSD are absent because there are no ground-truth tumor masks for testing. All models were trained on the same 3,488 pancreatic tumor CT-Mask pairs from JHH (contrast-enhanced) ; multi-task learning and R-Super additionally trained on 28,295 tumor CT-Report pairs from UCSF-Huge (41,418 CT-Report pairs , 4,809 non-contrast ). Sensitivity (Se), Specificity (Sp), and F1-Score correspond to the threshold that maximizes F1-Score for each model. Statistical tests compare R-Super and standard segmentation (paired permutation test for F1-Score, DeLong’s test for AUC); orange highlights statistically significant gains ( p<0.05 ). Standard deviations and confidence intervals are available in Tab. 9 (Appendix C ).
pancreas tumor
kidney tumor
JHH-Test
UCSF-Test
UCSF-Test
train paradigm
mask
rep.
DSC
NSD
F1
AUC
Se
Sp
F1
AUC
Se
Sp
mask
rep.
F1
AUC
Se
Sp
R-Super (ours)
344
2.2K
59
69
91
92
94
88
83
90
83
89
1.7K
2.7K
75
78
80
70
loss functions
Volume Loss only
344
2.2K
59
61
94
97
90
98
76
89
80
82
1.7K
2.7K
75
74
86
63
Ball Loss only
344
2.2K
59
59
83
92
78
90
71
82
64
90
1.7K
2.7K
74
75
87
58
Table 5: Ablation studies of R-Super. Loss functions: combining the Volume Loss and Ball Loss achieved the best performance on UCSF-Test; on the smaller JHH-Test, the Volume Loss alone scored higher F1-Score and AUC. Training with the Volume Loss alone surpassed training with the Ball Loss alone, because the Volume Loss is easier to optimize. LLM errors (or report errors): R-Super remained highly accurate when errors were injected into up to 10% of the tumor reports in the training dataset. These errors can simulate either errors in the report or errors in the LLM analysis of the report. LLM size: training with small LLMs (Qwen 3 4B or 0.6B instead of Llama 3.1 70B) substantially reduced R-Super performance. Organ mask quality: organ masks generated by an nnU-Net trained on only 50 CT scans were sufficient to accurately train R-Super. Even when organ masks were fully removed, R-Super still outperformed the standard segmentation model (Tab. 1 ). Report information removed: tumor diameters are important for R-Super, particularly for DSC. Standard deviations and confidence intervals are available in Tab. 10 (Appendix C ). Results are reported at the operating point of maximum F1-Score for all models.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
pancreas tumor
kidney tumor
JHH-Test
UCSF-Test
UCSF-Test
train paradigm
mask
rep.
DSC
NSD
F1
AUC
Se
Sp
F1
AUC
Se
Sp
mask
rep.
F1
AUC
Se
Sp
few training masks (50)
standard segmentation
50
0
47
46
50–73
51–73
48–75
50–76
43–58
57–70
39–55
72–82
50
0
58–69
63–73
61–75
56–69
R-Super (ours)
50
2.2K
47
47
55–77
65–83
56–80
51–78
55–67
65–75
66–80
52–65
50
2.7K
60–70
60–71
66–80
48–62
medium / many training masks (344 / 1.7K)
Appendix
Table 6: 95% confidence intervals (lower–upper, percentage points) for the detection metrics and standard deviations for DSC/NSD in Tab. 1 . The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles). Statistical tests were performed to compare F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC). Orange highlights statistically significant gains over standard segmentation ( p<0.05 ).
pancreas tumor
kidney tumor
JHH-Test
UCSF-Test
UCSF-Test
train paradigm
mask
rep.
DSC
NSD
F1
AUC
Se
Sp
F1
AUC
Se
Sp
mask
rep.
F1
AUC
Se
Sp
small tumors (diameter ≤ 2 cm)
few training masks (50)
standard segmentation
50
0
13
29
18–56
43–78
27–78
48–76
32–50
51–66
31–51
72–83
50
0
34–51
52–67
42–64
56–70
R-Super (ours)
50
2.2K
24
32
15–52
51–83
21–73
50–77
46–61
65–78
66–84
52–65
50
2.7K
38–54
53–69
54–75
48–62
Appendix
Table 7: 95% confidence intervals (lower–upper, percentage points) for the detection metrics and standard deviations for DSC/NSD in Tab. 2 (size-stratified). The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles). Statistical tests were performed to compare F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC). Orange highlights statistically significant gains over standard segmentation ( p<0.05 ).
training
JHH-Small
JHH-Large
Merlin-Test
train paradigm
mask
rep.
DSC
NSD
F1
AUC
Se
Sp
DSC
NSD
F1
AUC
Se
Sp
F1
AUC
Se
Sp
nnU-Net
926
0
26
30
79–88
87–94
74–87
82–93
26
30
76–92
90–99
78–97
90–98
64–75
71–80
61–75
67–79
Multi-task learning
926
1,848
25
32
87–95
91–98
85–97
86–97
24
24
93–100
96–100
93–100
90–100
64–75
71–79
56–70
77–88
standard segmentation
926
0
24
31
86–95
90–97
78–93
94–100
22
21
96–100
99–100
93–100
100–100
61–73
69–78
52–67
77–87
R-Super (ours)
926
1,848
24
29
92–98
94–99
87–98
94–100
23
22
95–100
100–100
100–100
90–100
73–82
79–87
70–83
74–85
Appendix
Table 8: 95% confidence intervals (lower–upper, percentage points) for the detection metrics and standard deviations for DSC/NSD in Tab. 3 . The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles). Statistical tests were performed to compare R-Super and standard segmentation for F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC); orange highlights statistically significant gains ( p<0.05 ).
training
pancreas tumor
train paradigm
mask
rep.
F1
AUC
Se
Sp
Swiss-Test
nnU-Net
3,488
0
74–81
70–78
70–78
65–77
Multi-task learning
3,488
28,295
70–77
72–79
59–68
80–89
standard segmentation
3,488
0
76–84
79–85
67–77
80–91
R-Super (ours)
3,488
28,295
83–88
82–88
78–85
81–90
Appendix
Table 9: 95% confidence intervals (lower–upper, percentage points) for the detection metrics in Tab. 4 . The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles). Statistical tests compare R-Super and standard segmentation (paired permutation test for F1-Score, DeLong’s test for AUC); orange highlights statistically significant gains ( p<0.05 ).
pancreas tumor
kidney tumor
JHH-Test
UCSF-Test
UCSF-Test
train paradigm
mask
rep.
DSC
NSD
F1
AUC
Se
Sp
F1
AUC
Se
Sp
mask
rep.
F1
AUC
Se
Sp
R-Super (ours)
344
2.2K
41
38
85–96
85–97
87–100
78–96
78–87
87–94
76–89
85–93
1.7K
2.7K
69–79
73–82
74–85
64–77
loss functions
Volume Loss only
344
2.2K
41
40
88–98
94–100
81–98
94–100
70–81
85–93
73–86
77–86
1.7K
2.7K
70–80
69–79
81–91
56–70
Ball Loss only
344
2.2K
43
42
74–91
86–96
66–90
81–98
64–77
77–87
56–72
86–94
1.7K
2.7K
69–78
69–80
82–92
51–65
Appendix
Table 10: Uncertainty for Tab. 5 : 95% confidence intervals (lower–upper, percentage points) for Se/Sp/F1/AUC and standard deviations for DSC/NSD. The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles).
Multi-tumor segmentation is important for early cancer detection and allows radiologists to visualize, verify, and understand AI predictions. However, tumor segmentation masks are expensive, time-consuming, and unavailable for many tumor types in public data. Instead, hospitals have vast, readily available data that can guide segmentation: radiology reports, longitudinal images, and multi-phase images. We use this readily available data to substitute for tumor masks in training AI for tumor segmentation. To this end, we propose a new architecture, RT-Super. It has a teacher network, which analyzes the patient's longitudinal images and reports to create high-quality tumor masks. These masks train a student network, which sees a single image and no report. At inference, when longitudinal images and reports are unavailable, we use the student. RT-Super uses a new CNN-Transformer architecture and novel Consistency Losses that exploit tumor location consistency across longitudinal images. We train RT-Super to segment esophagus, uterus and spleen tumors, which have few or no public masks. Even without training masks, RT-Super can segment these tumors and surpass public AI models. Overall, we demonstrate that learning from longitudinal images, multi-phase images, and reports can overcome mask scarcity and advance multi-cancer detection and segmentation. Code: https://github.com/MrGiovanni/RT-Super
Pedro R. A. S. Bassi, Wenxuan Li, Hanxue Gu +11
Johns Hopkins University · Harvard Medical School · Massachusetts General Hospital +4
Multi-cancer segmentation in computed tomography (CT) is fundamentally limited by the scarcity of tumor masks across different organs. We present Merlin Plus, the first large-scale CT dataset with radiologist-created tumor masks across 9 organs. Merlin Plus extends the Merlin dataset by adding 1,153 per-voxel tumor masks and longitudinal metadata. To create these tumor masks, we developed a report-based active-learning framework in which radiology reports identify tumor cases for annotation and support training of a tumor segmentation model. The model generates initial masks, which radiologists review and correct to produce the final masks, reducing annotation burden while maintaining high-quality annotations. Besides tumor masks, the longitudinal metadata in Merlin Plus enables temporal modeling of cancer progression. By directly addressing the major bottleneck of limited multi-cancer segmentation masks, Merlin Plus supports scalable multi-organ cancer detection, segmentation, and longitudinal analysis in CT. Dataset is available at: https://github.com/MrGiovanni/MerlinPlus
Pedro R. A. S. Bassi, Wenxuan Li, Szymon Plotka +10
Johns Hopkins University · Harvard Medical School · Massachusetts General Hospital +5
We introduce a method for training a text-conditioned segmentation model for CT scans, which combines voxel-level supervision with coarse but scalable slice-level supervision from reports. We extract, from a large database of scan-report pairs, descriptions of findings with indices of slices where those findings occur. We then finetune a general-purpose 2D image segmentation model, SAM3, with standard segmentation losses from strongly labeled data and with a slice-level classification loss from the extracted weak supervision. Our results on the ReXGroundingCT dataset illustrate that this strategy improves the segmentation dice score: from an 8% relative gain when there are 1000 fully labeled volumes to 22% when there are 250 fully labeled volumes.
Sanjay Subramanian, Junwei Yu, Zirui Wang +5
Department of Electrical Engineering and Computer Science, UC Berkeley · Voio · Department of Radiology and Biomedical Imaging, UC San Francisco +1