RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation

Seunghwan Lee; Sangjoon Park; Jongmin Lee; Yoon Jo Kim; Wonyoung Cho; Jin Sung Kim; Han Joo Chae

cs.CVOct 29, 2026

RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation

Authors: Seunghwan Lee, Sangjoon Park, Jongmin Lee, Yoon Jo Kim, Wonyoung Cho, Jin Sung Kim, Han Joo Chae

Organizations: Oncosoft Inc., Seoul, South Korea · Department of Radiation Oncology, Yonsei University College of Medicine, Seoul, South Korea

Abstract

Tumor delineation is required for radiotherapy planning, yet automatic segmentation is limited by errors, predominantly false positives, that distort dose and increase editing time. The radiology report accompanies the scan and localizes disease at the suborgan level, but existing text-promptable models condition on short referring phrases, and most operate in 2D. We present RepConTUR, a single native-3D early-fusion multimodal model that conditions volumetric abdominal tumor segmentation on the narrative report. We extend AbdomenAtlas 3.0 to a suborgan-level promptable segmentation dataset and generate report-based VQA pairs, yielding four report-derived tasks on the same annotations: referring segmentation (RES), report-grounded segmentation (GSEG), report generation (RRG), and visual question answering (VQA). We train RepConTUR, comprising a 3D CT encoder, an early-fusion language model, and a native-3D segmentation module, jointly on the four tasks. Across liver, kidney and pancreas, RepConTUR attains the highest Dice similarity coefficient (DSC), instance-wise precision, and detection F1 among the text-promptable baselines, and adding the report raises mean precision while leaving recall comparable. Against two vision-only specialists, nnU-Net and the report-supervised R-Super, RepConTUR attains the highest value on every in-domain metric for liver and pancreas and the highest mean DSC, recall, precision, and detection F1 over the four out-of-distribution public cohorts. Ablations indicate that conditioning on the narrative report raises detection F1 from 0.499 to 0.528 within the same model, that joint GSEG training raises RES F1 from 0.450 to 0.482, and that the native-3D segmentation module raises GSEG F1 from 0.139 to 0.528 over its 2D counterpart. Code is available at https://github.com/Oncosoft-Research/RepConTUR.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of RepConTUR. (a) Report-driven dataset construction from AbdomenAtlas 3.0 [ 4 ] . Report matching decomposes each organ tumor mask into suborgan-localized instances by intersection-over-component, yielding the RES and GSEG targets; the reports additionally yield verified VQA pairs. (b) The four tasks trained jointly on the shared backbone, namely RES, GSEG, RRG and VQA. (c) A single native-3D early-fusion model. A 3D CT encoder produces visual tokens fused with the text prompt from the first LLM layer; a [SEG] token drives a 3D segmentation module to a volumetric mask, while the same backbone answers VQA and generates RRG.

Liver
	DSC		R		P		F1
Model	Rs	Gs	Rs	Gs	Rs	Gs	Rs	Gs
VoxTell	0.406	0.339	0.455	0.387	0.019	0.010	0.037	0.020
UniBioMed	0.370	0.391	0.531	0.554	0.081	0.086	0.141	0.149
BiomedParse v2	0.341	—	0.693	—	0.082	—	0.147	—
RepConTUR	0.522	0.542	0.616	0.611	0.369	0.424	0.461	0.500

Table 1: Comparison with the text-promptable baselines on test split. Rs

=

RES, Gs

=

GSEG. DSC: Dice, R/P: instance-wise Recall/Precision, F1: harmonic mean. Bold : highest per column per section.

Panel of Figure 2 — Figure 2: Qualitative segmentation on the test split, same referring prompt per row, one axial slice. Cyan box marks the referred tumor, magnified in the lower-right inset; yellow arrows mark components outside it. GT green, prediction red. RepConTUR confines its prediction to the referred tumor, whereas the baselines add components elsewhere on the slice, namely other real but non-referred tumors for the multifocal liver and pancreas cases and the tumor-free contralateral kidney for the kidney case. These raise the false-positive count and are consistent with the low baseline precision in Tab. 1 .

Panel of Figure 3 — Figure 3: Prompt conditioning for a multifocal liver case, one axial slice. Switching the prompt between segments 5 and 6 moves the mask from anterior to posterior; querying segment 3, whose tumor lies on other slices, or the tumor-free kidney yields an empty mask, so the prompt controls both location and presence.

	DSC		R		P		F1
Vis./Seg.	Rs	Gs	Rs	Gs	Rs	Gs	Rs	Gs
2D / 2D	0.342	0.354	0.517	0.526	0.061	0.065	0.109	0.115
3D / 2D	0.331	0.378	0.526	0.535	0.075	0.080	0.129	0.139
3D / 3D	0.510	0.523	0.674	0.677	0.397	0.433	0.499	0.528

Table 2: Design ablation replacing the 2D visual and segmentation modules with 3D counterparts, 3-organ macro means. Rows are (vision / segmentation) module; 2D/2D is UniBioMed, 3D/2D replaces only the visual encoder. Metrics and abbreviations as in Tab. 1 . Bold : highest per column.

	DSC		R		P		F1
Train	Rs	Gs	Rs	Gs	Rs	Gs	Rs	Gs
RES	0.481	—	0.657	—	0.345	—	0.450	—
+GSEG	0.505	0.524	0.650	0.658	0.385	0.415	0.482	0.508
+RRG	0.503	0.524	0.656	0.655	0.396	0.435	0.493	0.523
+VQA	0.510	0.523	0.674	0.677	0.397	0.433	0.499	0.528

Table 3: Multitask ablation, tasks added cumulatively (RES, +GSEG, +RRG, +VQA), 3-organ macro means. A RES-only model has no Gs (–). Metrics and abbreviations as in Tab. 1 . Bold : highest per column.

Liver ( $n{=}134$ )
Model	DSC	R	P	F1
nnU-Net	0.602	0.566	0.392	0.463
R-Super	0.580	0.517	0.457	0.485
RES	0.610	0.659	0.550	0.599
GSEG	0.612	0.633	0.559	0.594
Pancreas ( $n{=}110$ )

Table 4: Report-based inference on the in-domain test set, per-organ RES/GSEG vs. two vision-only specialists, namely nnU-Net and the report-supervised R-Super [ 3 ] . nnU-Net and R-Super are restricted to the organs the agentic pipeline names, RES/GSEG are suborgan-decomposed. Metrics as in Tab. 1 . Bold : highest per column per organ.

HCC-TACE-Seg — HCC (liver), $n{=}130$
Model	DSC	R	P	F1
nnU-Net	0.650	0.679	0.134	0.223
R-Super	0.597	0.653	0.111	0.190
RES	0.719	0.689	0.225	0.340
GSEG	0.727	0.694	0.293	0.412
CPTAC-PDA — PDAC (pancreas), $n{=}108$

Table 5: Out-of-distribution generalization on four public tumor sets outside AbdomenAtlas 3.0, RES/GSEG (suborgan-decomposed, synthetic reports) vs. the vision-only nnU-Net and report-supervised R-Super [ 3 ] specialists. Sets are GT-positive only, so no organ is suppressed. Metrics as in Tab. 1 . Bold : highest per column per cohort.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	RES				GSEG
Backbone	DSC	R	P	F1	DSC	R	P	F1
InternVL3-2B (general)	0.399	0.572	0.340	0.421	0.471	0.644	0.352	0.450
InternVL3-8B (general)	0.485	0.661	0.360	0.466	0.506	0.661	0.397	0.495
RepConTUR ( $1$ B medical)	0.510	0.674	0.397	0.499	0.523	0.677	0.433	0.528

Appendix

Table A.1 : LLM backbone ablation with the same recipe and segmentation module. Metrics as in Tab. 1 , macro mean over three organs. Bold : highest per column.

	RES				GSEG
Encoder init	DSC	R	P	F1	DSC	R	P	F1
CT-RATE only	0.506	0.654	0.397	0.493	0.521	0.655	0.432	0.520
RepConTUR	0.510	0.674	0.397	0.499	0.523	0.677	0.433	0.528

Appendix

Table A.2 : Vision encoder initialization with the same recipe, CT-RATE

+

Merlin for RepConTUR vs. CT-RATE only [ 24 ] . Metrics as in Tab. 1 , macro mean over three organs. Bold : highest per column.

Liver
		Rs				Gs
Model	Size	DSC le	R	P	F1	DSC le	R	P	F1
VoxTell	$<$ 1	0.152	0.272	0.007	0.013	0.108	0.191	0.003	0.006
VoxTell	$\geq$ 1	0.384	0.559	0.400	0.466	0.339	0.499	0.337	0.402
UniBioMed	$<$ 1	0.230	0.423	0.033	0.062	0.245	0.448	0.036	0.067
UniBioMed	$\geq$ 1	0.361	0.592	0.265	0.367	0.380	0.614	0.281	0.385

Appendix

Table A.3 : Table 1 metrics split at a

1

cm tumor long axis; DSC le : GT-referenced tumor Dice. BiomedParse v2 has no GSEG mode. Bold : larger band per metric, organ and model.

Organ	Model	DSC	R	P	F1
Liver	nnU-Net	0.603	0.568	0.191	0.286
	nnU-Net $+$ agentic	0.602	0.566	0.392	0.463
	R-Super	0.580	0.518	0.230	0.319
	R-Super $+$ agentic	0.580	0.517	0.457	0.485
	RES	0.600	0.637	0.334	0.438
	RES $+$ agentic	0.610	0.659	0.550	0.599

Appendix

Table A.4 : In-domain decomposition of false-positive suppression. Base: RES/GSEG with one whole-organ prompt, specialists unrestricted;

+

agentic: suborgan prompts, or for nnU-Net and R-Super, predictions restricted to the reported organs. DSC: image-wise mean over GT-positive cases; R/P/F1 as in Tab. 1 ; Mean: macro mean over organs. Bold / underline : best/second per column.

Dataset	Model	DSC	R	P	F1
HCC-TACE liver ( $n{=}130$ )	nnU-Net	0.650	0.679	0.134	0.223
	R-Super	0.597	0.653	0.111	0.190
	RES	0.707	0.668	0.212	0.322
	RES $+$ agentic	0.719	0.689	0.225	0.340
	GSEG	0.725	0.689	0.302	0.420
	GSEG $+$ agentic	0.727	0.694	0.293	0.412

Appendix

Table A.5 : External-cohort version of Tab. A.4 ; metrics as in Tab. 1 . One GT-positive organ per cohort, so nnU-Net and R-Super appear once. Mean: macro mean over four cohorts. Bold / underline : best/second per column.

Figure A.1 : Report-word relevancy for three in-domain GSEG cases, one per organ, by gradient-weighted attention rollout [ 12 ] . Each word takes the maximum relevancy over its subword tokens; color is its ratio to the mean word relevancy of its report, on a log scale shared across rows from white at or below $1\times$ to $64\times$ .

Panel of Figure 15 — Figure A.2 : VQA on a multifocal liver case and RRG on a bilateral-kidney case from the in-domain cohort, each with one axial CT slice in radiological orientation and the reference tumor mask in red. Green / red : RepConTUR expressions that match/contradict the reference, which is lightly abridged.

	VQA accuracy				RRG
Model	Y/N	MC	Short	All	B-1	B-4	R-L	M	BS
UniBioMed	0.434	0.006	0.004	0.294	0.072	0.006	0.107	0.086	0.073
RepConTUR	0.848	0.640	0.406	0.732	0.329	0.168	0.377	0.299	0.593

Appendix

Table A.6 : Language tasks on the in-domain cohort,

n{=}373

reports and

7{,}011

VQA questions. MC: multiple choice; B-1/B-4: BLEU-1/4; R-L: ROUGE-L; M: METEOR [ 2 ] ; BS: BERTScore-F1 [ 69 ] . Bold : highest per column.

Model	Params	Input ( $W{\times}H{\times}D$ )	Inference Time (s)
nnU-Net (specialist)	$102$ M	$512\times 394\times 419$	$6.2$
R-Super (specialist)	$37.9$ M	$512\times 394\times 419$	$24.6$
RepConTUR (RES)	$1.41$ B	$512\times 394\times 419$	$23.7$
RepConTUR (GSEG)	$1.41$ B	$512\times 394\times 419$	$23.7$

Appendix

Table A.7 : Computational cost on one in-domain volume with

7

suborgan targets, NVIDIA RTX A5000, batch

1

, bfloat16, sliding-window pass only. Input: the original volume at

0.63\times 0.63\times 1.5

mm, which each model resamples to its own plan. Inference Time: mean over repeated runs, s.d. below

0.1

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 796 papers

Aug 27, 2026

Report Supervision

Segmentation models can surpass radiologists, classification models, and vision-language models in tumor detection. Importantly, segmentation models outline tumors, allowing radiologists to better verify and trust the AI output. Their main limitation is the scarcity of tumor masks: creating one 3D tumor mask takes up to 30 minutes, so most public CT datasets contain only a few hundred masks, and even the largest private datasets contain only a couple of thousand. Tumor masks are not produced in clinical routine, but radiology reports are. Public datasets contain tens of thousands of CT-Report pairs, and hospitals contain hundreds of thousands. These reports describe tumors in detail, providing large-scale, informative training data. Here, we introduce Report Supervision (R-Super), a training framework that uses reports to directly supervise and improve tumor segmentation. R-Super introduces new loss functions that teach segmentation models to segment tumors that match report descriptions of tumor count, sizes, and locations. Reports are only used for training. We evaluated R-Super on kidney and pancreatic tumor segmentation, exploring diverse training data sizes, up to 41,418 CT-Report plus 3,488 pancreatic tumor CT-Mask pairs. On external validation, R-Super increased tumor detection F1-Score and segmentation DSC by up to +15% with respect to mask-only training. It also surpassed alternative methods such as CLIP and multi-task learning. Leveraging numerous readily available reports to supplement scarce masks, R-Super strongly improves AI performance when very few training masks are available (e.g., 50), and when many masks are available (e.g., 3,488), unlocking scale in tumor segmentation.

Pedro R. A. S. Bassia, Wenxuan Li, Jakob Wasserthal +10

Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Clinic of Radiology and Nuclear Medicine, University Hospital Basel, Basel, Switzerland · Department of Computer Science, Stanford University, Stanford, CA, USA +5

Medical Image Segmentation Weakly Supervised Image Segmentation

Ask PDF

Sep 28, 2026

RT-Super: Learning Tumor Segmentation from Longitudinal Images and Reports

Multi-tumor segmentation is important for early cancer detection and allows radiologists to visualize, verify, and understand AI predictions. However, tumor segmentation masks are expensive, time-consuming, and unavailable for many tumor types in public data. Instead, hospitals have vast, readily available data that can guide segmentation: radiology reports, longitudinal images, and multi-phase images. We use this readily available data to substitute for tumor masks in training AI for tumor segmentation. To this end, we propose a new architecture, RT-Super. It has a teacher network, which analyzes the patient's longitudinal images and reports to create high-quality tumor masks. These masks train a student network, which sees a single image and no report. At inference, when longitudinal images and reports are unavailable, we use the student. RT-Super uses a new CNN-Transformer architecture and novel Consistency Losses that exploit tumor location consistency across longitudinal images. We train RT-Super to segment esophagus, uterus and spleen tumors, which have few or no public masks. Even without training masks, RT-Super can segment these tumors and surpass public AI models. Overall, we demonstrate that learning from longitudinal images, multi-phase images, and reports can overcome mask scarcity and advance multi-cancer detection and segmentation. Code: https://github.com/MrGiovanni/RT-Super

Pedro R. A. S. Bassi, Wenxuan Li, Hanxue Gu +11

Johns Hopkins University · Harvard Medical School · Massachusetts General Hospital +4

Weakly Supervised Image Segmentation Longitudinal Medical Image Analysis Medical Image Segmentation

Ask PDF

Jul 28, 2026

Open-Ended CT Volume Segmentation with Weak Supervision from Language

We introduce a method for training a text-conditioned segmentation model for CT scans, which combines voxel-level supervision with coarse but scalable slice-level supervision from reports. We extract, from a large database of scan-report pairs, descriptions of findings with indices of slices where those findings occur. We then finetune a general-purpose 2D image segmentation model, SAM3, with standard segmentation losses from strongly labeled data and with a slice-level classification loss from the extracted weak supervision. Our results on the ReXGroundingCT dataset illustrate that this strategy improves the segmentation dice score: from an 8% relative gain when there are 1000 fully labeled volumes to 22% when there are 250 fully labeled volumes.

Sanjay Subramanian, Junwei Yu, Zirui Wang +5

Department of Electrical Engineering and Computer Science, UC Berkeley · Voio · Department of Radiology and Biomedical Imaging, UC San Francisco +1

Weakly Supervised Image Segmentation Text-Guided Image Segmentation Promptable Image Segmentation Medical Image Segmentation Computed Tomography

Ask PDF

Abstract

Medical Image Segmentation

Explore similar work

Report Supervision

RT-Super: Learning Tumor Segmentation from Longitudinal Images and Reports

Open-Ended CT Volume Segmentation with Weak Supervision from Language