CORE-Seg: Reasoning-Driven Segmentation for Complex Lesions via Reinforcement Learning
Authors: Yuxin Xie, Yuming Chen, Yishan Yang, Yi Zhou, Tao Zhou, Zhen Zhao, Jiacheng Liu, Huazhu Fu
Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, 211189, China. · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China. · Zhongda Hospital, Southeast University, Nanjing, 210009, China. · Institute of High-Performance Computing, Agency for Science, Technology and Research, 138632, Singapore.
Medical image segmentation is undergoing a paradigm shift from conventional visual pattern matching to cognitive reasoning analysis. Although Multimodal Large Language Models (MLLMs) have shown promise in integrating linguistic and visual knowledge, significant gaps remain: existing general MLLMs possess broad common sense but lack the specialized visual reasoning required for complex lesions, whereas traditional segmentation models excel at pixel-level segmentation but lack logical interpretability. In this paper, we introduce ComLesion-14K, the first diverse Chain-of-Thought (CoT) benchmark for reasoning-driven complex lesion segmentation. To accomplish this task, we propose CORE-Seg, an end-to-end framework integrating reasoning with segmentation through a Semantic-Guided Prompt Adapter. We design a progressive training strategy from SFT to GRPO, equipped with an adaptive dual-granularity reward mechanism to mitigate reward sparsity. Our Method achieves state-of-the-art results with a mean Dice of 37.06% (14.89% higher than the second-best baseline), while reducing the failure rate to 18.42%. Project Page: https://xyxl024.github.io/CORE-Seg.github.io/
Figures & tables
Table 1
Figure 1: (a) Three major challenges in complex medical lesion segmentation, containing noisy and distorted imaging, lesion diversity, and pathological heterogeneity. (b) Supervised fine-tuned, end-to-end dense prediction architecture. (c) Reinforcement learning-driven, cascaded reasoning framework. (d) Our proposed end-to-end complex-lesion-centric reasoning segmentation framework, CORE-Seg.
Figure 2: Overview of the ComLesion-14K dataset construction pipeline. The pipeline includes difficulty-aware filtering, VQA & CoT generation, and automated quality assurance. Easy and hard samples are compared on the left to demonstrate the necessity of filtering mechanism. Bounding boxes are adopted to clearly visualize lesion morphology without obscuring details.
Figure 3: (a) Representative samples illustrating broad organ coverage across diverse imaging modalities. (b) Dataset statistics summarizing the distribution of imaging modalities, anatomical structures, and disease categories.
Table 1: Detailed Composition of the ComLesion-14K Dataset
Figure 4: Overview of the CORE-Seg framework. The architecture integrates an MLLM with a SAM via a Semantic-Guided Prompt Adapter. The training consists of two stages: (1) CoT-Based Semantic Alignment Stage: Supervised fine-tuning establishes the mapping between clinical reasoning and lesion localization using the <seg> token. (2) RL-Driven Reasoning Refinement Stage: The model is further optimized using GRPO. A multi-dimensional Reward Engine (evaluating format, detection, and segmentation) serves as the critic, where a Dual-Granularity Mask Reward penalizes disconnected regions and encourages precise boundary delineation.
Method
Overall Performance (%)
Anatomy-specific Dice (%)
mDice ↑
mIoU ↑
Fail ↓
Head
Abd.
Lung
Eye
Neck
Skin
Foot
Breast
Heart
General MLLMs
Qwen2.5-VL-72B
11.04
7.78
37.61
6.68
5.11
3.30
5.23
31.36
54.09
22.89
52.86
1.21
InternVL3-8B
4.18
2.72
73.90
2.36
0.50
0.79
0.96
11.34
31.44
5.55
14.07
0.86
Qwen2.5-VL-7B
8.30
5.56
42.76
3.62
4.47
2.62
2.89
23.79
48.12
5.18
41.43
3.06
Medical-Specific MLLMs
Table 2: Quantitative comparison on complex lesion segmentation. We report mDice, mIoU, and Failure Rate (Fail). Detailed Dice scores for 9 anatomical regions are provided to analyze local grounding capabilities. All values are in percentage (%). Best results are bolded , second best are underlined . † denotes the models fine-tuned with our ComLesion-14K.
Figure 5: Comparison of mDice scores across 8 medical modalities. CORE-Seg demonstrates superior generalization capabilities, achieving state-of-the-art performance particularly among MLLM-based methods in MRI, CT, X-Ray and Ultrasound.
Method
ID Performance
OOD Robustness (mDice ↑ )
mDice ↑
mIoU ↑
Fail ↓
TN-SCUI
CVC-ClinicDB
ISPY
nnU-Net (Binary)
22.05
16.92
59.27
63.06
5.75
0.00
nnU-Net (Multi-class)
40.89
31.65
25.36
58.21
–
–
LViT (Referring)
41.25
33.97
24.32
22.87
1.10
1.08
CORE-Seg (Reasoning)
37.06
27.79
18.42
74.82
48.20
38.71
Table 3: Comparison with specialist segmentation models. In-domain (ID) results are on ComLesion-14K; OOD results are mDice (%) on TN-SCUI, CVC-ClinicDB, and ISPY. All specialist models are trained on the same patient-level training split of ComLesion-14K. “–” indicates that the multi-class model cannot be evaluated on OOD data due to label-space mismatch.
Components
ID Performance
OOD Robustness (mDice ↑ /Fail ↓ )
SG-A
S1
S2
mDice ↑
mIoU ↑
Fail ↓
CVC-ClinicDB
TN-SCUI
ISPY
-
-
-
4.86
3.00
58.5
30.35/27.6
15.01/37.8
0.98/65.2
-
✓
-
23.60
16.57
45.2
37.67/26.2
50.26/14.4
32.26/21.6
✓
✓
-
30.37
22.20
26.02
34.16/17.20
64.16/ 1.60
29.31/15.40
✓
✓
✓
37.06
27.79
18.42
48.20/15.00
74.82 /2.20
38.71/13.00
Table 4: Ablation study on progressive framework. We report mDice, mIoU, and Failure Rate (Fail) on ID data. For OOD robustness, we report mDice and Fail across three challenging datasets: CVC-ClinicDB, TN-SCUI, and ISPY.
CoT
rfmt
rbbox
rmask
LDiceCE
mDice ↑
mIoU ↑
Fail ↓
–
–
–
–
–
31.24
23.15
20.08
✓
–
–
–
–
30.37
22.20
26.02
✓
✓
✓
–
–
26.87
18.91
30.08
✓
✓
–
✓
–
31.83
23.45
24.63
✓
✓
✓
✓
–
32.41
24.06
24.32
Rand.
✓
✓
✓
✓
33.85
24.91
23.64
Table 5: Ablation study of adaptive reward mechanism in CORE-Seg. We analyze the impact of training-time reasoning guidance, reward types, Dice loss, and explicitly quantify the effect of reasoning quality (Random vs. Description-only vs. Full CoT).
Semantic Probe
Training Stage
mDice ↑
mIoU ↑
Fail ↓
Whole Answer
S1
32.25
23.90
24.82
S1 + S2
33.01
24.54
24.32
<seg> Token
S1
30.37
22.20
26.02
S1 + S2
37.06
27.79
18.42
Table 6: Ablation study on the semantic probe selection. We compare using the specific <seg> token (Ours) versus the hidden states of the whole answer across training stages.
Figure 6: Qualitative visualization and reasoning analysis of our two-stage framework. The left part illustrates that Stage 2 refines the outputs of Stage 1 via image-grounded reasoning, yielding higher-quality masks and stronger robustness, especially for small lesions and cases with inaccurate bounding box localization. The right part presents typical correct and erroneous reasoning pipelines, demonstrating that both bounding box localization and mask prediction are explicitly influenced by visual reasoning descriptions.
Segmentation Model
mDice ↑
mIoU ↑
Fail ↓
SAMed2D
19.58
13.44
45.28
SAM 2
23.60
16.57
39.87
MedSAM2 (Ours)
30.37
22.20
26.02
Table 7: Comparison of different SAM backbones. We evaluate the impact of the SAM version in Stage1.
Figure 7: Visualization of our method with baselines.
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Nanjing, China · Nanjing University of Science and Technology, Nanjing, China