Report Supervision

Pedro R. A. S. Bassia; Wenxuan Li; Jakob Wasserthal; Jieneng Chen; Xinze Zhou; Zheren Zhu; Chuntung Zhuanga; Sergio Decherchi

cs.CVAug 27, 2026

Report Supervision

Authors: Pedro R. A. S. Bassia, Wenxuan Li, Jakob Wasserthal, Jieneng Chen, Xinze Zhou, Zheren Zhu, Chuntung Zhuanga, Sergio Decherchi, +5 more

Organizations: Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Clinic of Radiology and Nuclear Medicine, University Hospital Basel, Basel, Switzerland · Department of Computer Science, Stanford University, Stanford, CA, USA · Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, USA · Computational and Chemical Biology, Italian Institute of Technology, Genoa, Italy · Department of Pharmacy and Biotechnology, University of Bologna, Bologna, Italy · Centre Européen de Calcul Atomique et Moléculaire (CECAM), École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland · Department of Oncology, Johns Hopkins Medicine, Baltimore, MD, USA

Abstract

Segmentation models can surpass radiologists, classification models, and vision-language models in tumor detection. Importantly, segmentation models outline tumors, allowing radiologists to better verify and trust the AI output. Their main limitation is the scarcity of tumor masks: creating one 3D tumor mask takes up to 30 minutes, so most public CT datasets contain only a few hundred masks, and even the largest private datasets contain only a couple of thousand. Tumor masks are not produced in clinical routine, but radiology reports are. Public datasets contain tens of thousands of CT-Report pairs, and hospitals contain hundreds of thousands. These reports describe tumors in detail, providing large-scale, informative training data. Here, we introduce Report Supervision (R-Super), a training framework that uses reports to directly supervise and improve tumor segmentation. R-Super introduces new loss functions that teach segmentation models to segment tumors that match report descriptions of tumor count, sizes, and locations. Reports are only used for training. We evaluated R-Super on kidney and pancreatic tumor segmentation, exploring diverse training data sizes, up to 41,418 CT-Report plus 3,488 pancreatic tumor CT-Mask pairs. On external validation, R-Super increased tumor detection F1-Score and segmentation DSC by up to +15% with respect to mask-only training. It also surpassed alternative methods such as CLIP and multi-task learning. Leveraging numerous readily available reports to supplement scarce masks, R-Super strongly improves AI performance when very few training masks are available (e.g., 50), and when many masks are available (e.g., 3,488), unlocking scale in tumor segmentation.

Figures & tables

Figure 1: R-Super trains tumor segmentation models using information from radiology reports, with Volume Loss and Ball Loss guiding predictions to match reported tumor size, location, and number. (a) The Volume Loss (deep supervision) aligns segmented tumors with volumes and locations from reports. It consists of three steps. (I) A large language model (LLM) extracts tumor location (organ/sub-segment) and diameters from the report. Using ellipsoid approximations, we estimate the total tumor volume in each organ $o$ as $V_{r,o}$ . (II) From a deep layer of the segmenter, we compute tumor probabilities via a 1 $\times$ 1 $\times$ 1 convolution and sigmoid/softmax. We sum the probabilities inside the organ (using pre-saved organ masks) to estimate the segmented tumor volume, $V_{s,o}$ . (III) The Volume Loss penalizes discrepancies between $V_{s,o}$ and $V_{r,o}$ . As shown in the right panel (for $V_{r,o}=1000$ mm 3 ), the loss is zero when $V_{s,o}$ is close to $V_{r,o}$ , allowing for some tolerance due to uncertainty in report-based volume estimates. (b) The Ball Loss enforces segmented tumors to match detailed tumor information from reports—tumor count, locations, diameters, and volumes. It consists of four steps. (I) A large language model (LLM) extracts this tumor information from the report. (II) A Ball Convolution—a simple convolution with a fixed spherical kernel matching the reported tumor diameter—slides over the segmenter output to find the highest-probability tumor region. (III) Within this region, the loss maximizes probabilities in the top- $N_{i}$ most probable voxels, where $N_{i}$ is the tumor volume estimated from the report (in voxels, reduced by 20%). These voxels can form any shape that stays within the ball. (IV) The process (II–III) is repeated for all reported tumors, largest to smallest. Already-assigned voxels are ignored in later iterations to prevent overlap. Finally, the loss minimizes tumor probabilities in unassigned voxels (background). It focuses more on central tumor voxels and avoids penalizing uncertain tumor borders.

Figure 2: Dataset overview (Secs. 4.1 to 4.3 ). Summary of all datasets, including CT-Mask and CT-Report cohorts from multiple institutions. In (I), (II) and (III), we show the number of tumor scans, tumor types (pancreatic and kidney), and distributions of tumor size, tumor count per scan, and patient demographics (age and sex) across training and test sets (UCSF, JHH, and public datasets). Radiology reports enable large-scale dataset construction ( Li et al., 2025a ) , which R-Super leverages for segmentation training.

Figure 3: By learning from reports, R-Super substantially outperforms five state-of-the-art methods on both internal (UCSF-Test) and external (JHH-Test) validation (all six compared methods are in Tab. 1 ). Green arrows show the gain of R-Super over the best competing method at each mask budget, up to +14% in tumor detection F1-Score and +11% in DSC; over standard segmentation (no reports) alone, the gain reaches +16% in F1-Score (Tab. 1 ). Each panel corresponds to one tumor type and one test set, and its x-axis shows the number of training masks (50 from AbdomenAtlas-Subset; 344 or 1.7K from AbdomenAtlas 3.0; the axis labels 350 and 1750 are rounded). R-Super and the other report-based methods shown (CLIP, MTL) supplement these masks with 2.2K pancreatic and 2.7K kidney tumor reports from UCSF-Train. Both test sets are challenging: JHH-Test is out-of-distribution relative to training, and UCSF-Test spans a wide range of CT resolutions and contrast phases, including non-contrast scans. DSC is only reported for JHH-Test because UCSF-Test does not include ground-truth tumor masks, needed for calculating DSC.

	pancreas tumor												kidney tumor
	JHH-Test								UCSF-Test				UCSF-Test
train paradigm	mask	rep.	DSC	NSD	F1	AUC	Se	Sp	F1	AUC	Se	Sp	mask	rep.	F1	AUC	Se	Sp
few training masks (50)
standard segmentation	50	0	38	41	62	63	62	62	51	63	47	77	50	0	64	68	68	63
R-Super (ours)	50	2.2K	49	52	67	75	68	64	61	70	73	59	50	2.7K	65	66	73	55
medium / many training masks (344 / 1.7K)

Table 1: R-Super outperforms state-of-the-art training methods on tumor detection and segmentation. On pancreatic tumors, R-Super achieves the highest DSC, NSD, F1-Score, and AUC across all baselines. On kidney tumors, it matches mask-based segmentation when few masks are available (50) and surpasses it when many are available (1.7K). DSC and NSD are computed on scans with tumors and reported only in datasets where masks exist (UCSF-Test has none). Sensitivity (Se) and Specificity (Sp) correspond to the threshold that maximizes F1-Score. Statistical tests were performed to compare F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC). Orange highlights statistically significant gains over standard segmentation (

p<0.05

). Standard deviations and confidence intervals are available in Tab. 6 (Appendix C ).

Figure 4: Tumors segmented by R-Super. The examples show two cases of small pancreatic adenocarcinoma and one case of small renal carcinoma (bottom) correctly segmented by R-Super. Radiology reports are shown, but they are used only in training, not at inference.

	pancreas tumor												kidney tumor
	JHH-Test								UCSF-Test				UCSF-Test
train paradigm	mask	rep.	DSC	NSD	F1	AUC	Se	Sp	F1	AUC	Se	Sp	mask	rep.	F1	AUC	Se	Sp
small tumors (diameter $\leq$ 2 cm)
few training masks (50)
standard segmentation	50	0	7	17	38	61	53	62	41	59	40	77	50	0	43	59	53	63
R-Super (ours)	50	2.2K	15	25	35	68	47	64	54	72	76	59	50	2.7K	47	61	65	55

Table 2: R-Super outperforms state-of-the-art methods on both small (diameter

\leq

2 cm) and large tumors. For pancreatic tumors (344 training masks), R-Super achieves the highest DSC, NSD, and F1-Score across all baselines in both small (

\leq

2 cm) and large (

>

2 cm) tumors, and the highest AUC except for small tumors in JHH-Test (CLIP-Like 90 vs. 89). For kidney tumors (1.7K training masks), R-Super leads in F1-Score and AUC on small tumors and in F1-Score on large tumors. DSC and NSD are computed on scans with tumors and reported only where masks exist (UCSF-Test has none). Sensitivity (Se) and Specificity (Sp) correspond to the threshold that maximizes F1-Score. Statistical tests were performed to compare F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC). Orange highlights statistically significant gains over standard segmentation (

p<0.05

). Test set sizes: JHH-Test contains 15 small and 35 large pancreatic tumors; UCSF-Test contains 90 small and 28 large pancreatic tumors (21 CTs with tumors of unknown size were excluded from this table) and 77 small and 65 large kidney tumors (27 excluded). Tumor sizes were extracted from reports in UCSF-Test and from masks in JHH-Test. Standard deviations and confidence intervals are available in Tab. 7 (Appendix C ).

	training		JHH-Small						JHH-Large						Merlin-Test
train paradigm	mask	rep.	DSC	NSD	F1	AUC	Se	Sp	DSC	NSD	F1	AUC	Se	Sp	F1	AUC	Se	Sp
nnU-Net	926	0	33	58	83	91	80	87	49	72	88	93	84	92	69	76	68	73
Multi-task learning	926	1,848	33	57	92	95	92	92	50	73	97	99	98	96	69	75	63	82
standard segmentation	926	0	31	55	91	94	86	98	50	73	99	100	98	100	67	74	60	82
R-Super (ours)	926	1,848	34	59	95	97	93	98	52	74	98	100	100	96	78	83	77	79

Table 3: R-Super sets a new state-of-the-art on the largest public pancreatic tumor dataset. Combining CT-Mask pairs from PanTS (the largest public pancreatic tumor CT-Mask dataset) with CT-Report pairs from Merlin (the largest public abdominal CT-Report dataset), R-Super outperforms a segmentation model trained on CT-Mask pairs alone (PanTS) by +7% sensitivity on small pancreatic tumors. R-Super also surpasses all alternative models on DSC and NSD, and on F1-Score and AUC in JHH-Small and Merlin-Test (in JHH-Large, all methods saturate at F1-Score 97–99 and AUC 99–100). All models were trained on the same 926 pancreatic tumor CT-Mask pairs from PanTS. Multi-task learning and R-Super additionally trained on Merlin-Train (1,848 pancreatic tumor and 1,848 no-tumor CT-Report pairs from Merlin). DSC and NSD are computed on scans with tumors and reported only where masks exist (Merlin-Test has none). Sensitivity (Se), Specificity (Sp), and F1-Score correspond to the threshold that maximizes F1-Score for each model. Statistical tests were performed to compare R-Super and standard segmentation for F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC); orange highlights statistically significant gains (

p<0.05

). Standard deviations and confidence intervals are available in Tab. 8 (Appendix C ).

	training		pancreas tumor
train paradigm	mask	rep.	F1	AUC	Se	Sp
Swiss-Test
nnU-Net	3,488	0	78	74	74	71
Multi-task learning	3,488	28,295	74	76	64	85
standard segmentation	3,488	0	80	82	72	86
R-Super (ours)	3,488	28,295	85	85	80	86

Table 4: R-Super surpasses mask-based training when large amounts of both reports and masks are available. On Swiss-Test, an external dataset which we publicly release, R-Super outperforms standard mask-based segmentation by +8% sensitivity at the same specificity, for pancreatic tumor detection. On UCSF-Test-Non-Contrast, an internal test set of non-contrast CT scans, R-Super outperforms the standard segmentation model by +19% sensitivity at the same specificity. DSC and NSD are absent because there are no ground-truth tumor masks for testing. All models were trained on the same 3,488 pancreatic tumor CT-Mask pairs from JHH (contrast-enhanced) ; multi-task learning and R-Super additionally trained on 28,295 tumor CT-Report pairs from UCSF-Huge (41,418 CT-Report pairs , 4,809 non-contrast ). Sensitivity (Se), Specificity (Sp), and F1-Score correspond to the threshold that maximizes F1-Score for each model. Statistical tests compare R-Super and standard segmentation (paired permutation test for F1-Score, DeLong’s test for AUC); orange highlights statistically significant gains (

p<0.05

). Standard deviations and confidence intervals are available in Tab. 9 (Appendix C ).

	pancreas tumor												kidney tumor
	JHH-Test								UCSF-Test				UCSF-Test
train paradigm	mask	rep.	DSC	NSD	F1	AUC	Se	Sp	F1	AUC	Se	Sp	mask	rep.	F1	AUC	Se	Sp
R-Super (ours)	344	2.2K	59	69	91	92	94	88	83	90	83	89	1.7K	2.7K	75	78	80	70
loss functions
Volume Loss only	344	2.2K	59	61	94	97	90	98	76	89	80	82	1.7K	2.7K	75	74	86	63
Ball Loss only	344	2.2K	59	59	83	92	78	90	71	82	64	90	1.7K	2.7K	74	75	87	58

Table 5: Ablation studies of R-Super. Loss functions: combining the Volume Loss and Ball Loss achieved the best performance on UCSF-Test; on the smaller JHH-Test, the Volume Loss alone scored higher F1-Score and AUC. Training with the Volume Loss alone surpassed training with the Ball Loss alone, because the Volume Loss is easier to optimize. LLM errors (or report errors): R-Super remained highly accurate when errors were injected into up to 10% of the tumor reports in the training dataset. These errors can simulate either errors in the report or errors in the LLM analysis of the report. LLM size: training with small LLMs (Qwen 3 4B or 0.6B instead of Llama 3.1 70B) substantially reduced R-Super performance. Organ mask quality: organ masks generated by an nnU-Net trained on only 50 CT scans were sufficient to accurately train R-Super. Even when organ masks were fully removed, R-Super still outperformed the standard segmentation model (Tab. 1 ). Report information removed: tumor diameters are important for R-Super, particularly for DSC. Standard deviations and confidence intervals are available in Tab. 10 (Appendix C ). Results are reported at the operating point of maximum F1-Score for all models.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

pancreas tumor

kidney tumor

JHH-Test

UCSF-Test

train paradigm

mask

rep.

DSC

NSD

AUC

mask

rep.

AUC

few training masks (50)

standard segmentation

50–73

51–73

48–75

50–76

43–58

57–70

39–55

72–82

58–69

63–73

61–75

56–69

R-Super (ours)

2.2K

55–77

65–83

56–80

51–78

55–67

65–75

66–80

52–65

2.7K

60–70

60–71

66–80

48–62

medium / many training masks (344 / 1.7K)

Appendix

Table 6: 95% confidence intervals (lower–upper, percentage points) for the detection metrics and standard deviations for DSC/NSD in Tab. 1 . The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles). Statistical tests were performed to compare F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC). Orange highlights statistically significant gains over standard segmentation (

p<0.05

pancreas tumor

kidney tumor

JHH-Test

UCSF-Test

train paradigm

mask

rep.

DSC

NSD

AUC

mask

rep.

AUC

small tumors (diameter

\leq

2 cm)

few training masks (50)

standard segmentation

18–56

43–78

27–78

48–76

32–50

51–66

31–51

72–83

34–51

52–67

42–64

56–70

R-Super (ours)

2.2K

15–52

51–83

21–73

50–77

46–61

65–78

66–84

52–65

2.7K

38–54

53–69

54–75

48–62

Appendix

Table 7: 95% confidence intervals (lower–upper, percentage points) for the detection metrics and standard deviations for DSC/NSD in Tab. 2 (size-stratified). The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles). Statistical tests were performed to compare F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC). Orange highlights statistically significant gains over standard segmentation (

p<0.05

training

JHH-Small

JHH-Large

Merlin-Test

train paradigm

mask

rep.

DSC

NSD

AUC

DSC

NSD

AUC

nnU-Net

926

79–88

87–94

74–87

82–93

76–92

90–99

78–97

90–98

64–75

71–80

61–75

67–79

Multi-task learning

926

1,848

87–95

91–98

85–97

86–97

93–100

96–100

93–100

90–100

64–75

71–79

56–70

77–88

standard segmentation

926

86–95

90–97

78–93

94–100

96–100

99–100

93–100

100–100

61–73

69–78

52–67

77–87

R-Super (ours)

926

1,848

92–98

94–99

87–98

94–100

95–100

100–100

90–100

73–82

79–87

70–83

74–85

Appendix

Table 8: 95% confidence intervals (lower–upper, percentage points) for the detection metrics and standard deviations for DSC/NSD in Tab. 3 . The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles). Statistical tests were performed to compare R-Super and standard segmentation for F1-Score and AUC (paired permutation test for F1-Score and DeLong’s test for AUC); orange highlights statistically significant gains (

p<0.05

	training		pancreas tumor
train paradigm	mask	rep.	F1	AUC	Se	Sp
Swiss-Test
nnU-Net	3,488	0	74–81	70–78	70–78	65–77
Multi-task learning	3,488	28,295	70–77	72–79	59–68	80–89
standard segmentation	3,488	0	76–84	79–85	67–77	80–91
R-Super (ours)	3,488	28,295	83–88	82–88	78–85	81–90

Appendix

Table 9: 95% confidence intervals (lower–upper, percentage points) for the detection metrics in Tab. 4 . The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles). Statistical tests compare R-Super and standard segmentation (paired permutation test for F1-Score, DeLong’s test for AUC); orange highlights statistically significant gains (

p<0.05

pancreas tumor

kidney tumor

JHH-Test

UCSF-Test

train paradigm

mask

rep.

DSC

NSD

AUC

mask

rep.

AUC

R-Super (ours)

344

2.2K

85–96

85–97

87–100

78–96

78–87

87–94

76–89

85–93

1.7K

2.7K

69–79

73–82

74–85

64–77

loss functions

Volume Loss only

344

2.2K

88–98

94–100

81–98

94–100

70–81

85–93

73–86

77–86

1.7K

2.7K

70–80

69–79

81–91

56–70

Ball Loss only

344

2.2K

74–91

86–96

66–90

81–98

64–77

77–87

56–72

86–94

1.7K

2.7K

69–78

69–80

82–92

51–65

Appendix

Table 10: Uncertainty for Tab. 5 : 95% confidence intervals (lower–upper, percentage points) for Se/Sp/F1/AUC and standard deviations for DSC/NSD. The 95% confidence intervals were estimated by nonparametric bootstrap over the test cases (1,000 resamples; 2.5th and 97.5th percentiles).

Explore similar work

RT-Super: Learning Tumor Segmentation from Longitudinal Images and Reports
Sep 28, 2026Pedro R. A. S. Bassi, Wenxuan Li, Hanxue Gu +11Weakly Supervised Image Segmentation Longitudinal Medical Image Analysis
Johns Hopkins University · Harvard Medical School · Massachusetts General Hospital +4
Merlin Plus: A Large-Scale, Multi-Cancer, Image-Mask-Report Dataset
Sep 29, 2026Pedro R. A. S. Bassi, Wenxuan Li, Szymon Plotka +10Medical Image Segmentation Computed Tomography
Johns Hopkins University · Harvard Medical School · Massachusetts General Hospital +5
Open-Ended CT Volume Segmentation with Weak Supervision from Language
Jul 28, 2026Sanjay Subramanian, Junwei Yu, Zirui Wang +5Weakly Supervised Image Segmentation Text-Guided Image Segmentation
Department of Electrical Engineering and Computer Science, UC Berkeley · Voio · Department of Radiology and Biomedical Imaging, UC San Francisco +1

Sep 28, 2026J/K move · Enter open · S save

RT-Super: Learning Tumor Segmentation from Longitudinal Images and Reports

Pedro R. A. S. Bassi, Wenxuan Li, Hanxue Gu, Jieneng Chen+10

Johns Hopkins University · Harvard Medical School · Massachusetts General Hospital+4

Weakly Supervised Image Segmentation Longitudinal Medical Image Analysis Medical Image Segmentation

Open paper AskPDF arXiv

Multi-tumor segmentation is important for early cancer detection and allows radiologists to visualize, verify, and understand AI predictions. However, tumor segmentation masks are expensive, time-consuming, and unavailable for many tumor types in public data. Instead, hospitals have vast, readily available data that can guide segmentation: radiology reports, longitudinal images, and multi-phase images. We use this readily available data to substitute for tumor masks in training AI for tumor segmentation. To this end, we propose a new architecture, RT-Super. It has a teacher network, which analyzes the patient's longitudinal images and reports to create high-quality tumor masks. These masks train a student network, which sees a single image and no report. At inference, when longitudinal images and reports are unavailable, we use the student. RT-Super uses a new CNN-Transformer architecture and novel Consistency Losses that exploit tumor location consistency across longitudinal images. We train RT-Super to segment esophagus, uterus and spleen tumors, which have few or no public masks. Even without training masks, RT-Super can segment these tumors and surpass public AI models. Overall, we demonstrate that learning from longitudinal images, multi-phase images, and reports can overcome mask scarcity and advance multi-cancer detection and segmentation. Code: https://github.com/MrGiovanni/RT-Super

Abstract

Explore similar work

RT-Super: Learning Tumor Segmentation from Longitudinal Images and Reports

Merlin Plus: A Large-Scale, Multi-Cancer, Image-Mask-Report Dataset

Open-Ended CT Volume Segmentation with Weak Supervision from Language