XFlow: A Workflow Model for Instruction-Guided Lesion Segmentation in Chest X-rays

Geon Choi; Hangyul Yoon; Hyunki Park; Sang Hoon Seo; Edward Choi

cs.CVSep 28, 2026

XFlow: A Workflow Model for Instruction-Guided Lesion Segmentation in Chest X-rays

Authors: Geon Choi, Hangyul Yoon, Hyunki Park, Sang Hoon Seo, Edward Choi

Organizations: KAIST · Korea University Guro Hospital · Samsung Medical Center

Abstract

Existing text-guided segmentation models in the medical domain cover only a narrow set of anatomical structures and lesions in chest X-rays (CXRs), and most of them assume that the queried target is always present in the image. Instruction-guided lesion segmentation (ILS) was introduced to overcome these limitations by segmenting diverse lesion types from simple user instructions while also recognizing when the queried lesion is absent, and ROSALIA was proposed as the first model for this task. However, the masks produced by ROSALIA remain of limited quality, often carrying scattered noise. Moreover, ROSALIA predicts the mask in a single shot, which differs fundamentally from how radiologists perceive and delineate lesions in practice. A radiologist first surveys the entire thorax, then localizes the approximate region of abnormality, and only then refines the lesion contour. Motivated by this coarse-to-fine, multi-level perception process, we present XFlow, a workflow model for ILS that combines box-based localization with multi-turn point refinement. XFlow detects the lungs, decides whether the queried finding is present in each of them, and prompts a fine-tuned SAM with the lesion box for an initial mask. It then corrects that mask through point prompts until its boundary follows the lesion, leaving every intermediate decision visible. Our experiments show that XFlow achieves the best segmentation quality on both internal and external evaluation. Notably, it surpasses ROSALIA in segmentation quality even when the two are trained on the same lesion annotations. Code and model weights will be made publicly available.

Figures & tables

Figure 1 : Overview of XFlow. (Top) In Stage 1, XFlow grounds the lung regions, localizes the finding as a bounding box, and prompts a fine-tuned SAM with that box to obtain an initial mask. In Stage 2, it inspects the mask and issues positive or negative point prompts that expand or shrink the predicted region, repeating this until the mask is satisfactory. (Bottom) If the queried finding is absent, XFlow terminates at Stage 1 without producing a mask.

Figure 2 : Training the two policies. We first apply SFT to $\pi_{\text{init}}$ and $\pi_{\text{rev}}$ on automatically constructed initial segmentation and segmentation revision scenarios, and then train each policy further with GRPO. $A_{\text{init}}$ and $A_{\text{rev}}$ are the advantages obtained by normalizing $R_{\text{init}}$ and $R_{\text{rev}}$ within each group of sampled generations. Text tokens are shown in gray and image tokens in light blue.

Model	CheXpercept			CheXlocalize
Model	gIoU	cIoU	F1	gIoU	cIoU	F1
Human benchmark	—	—	—	0.314	0.339	—
LISA-7B	0.014	0.024	0.401	0.082	0.119	0.491
PixelLM-7B	0.031	0.032	0.416	0.125	0.125	0.485
Text4Seg	0.035	0.038	0.402	0.119	0.132	0.475
IMIS-Net	0.080	0.087	0.413	0.005	0.007	0.423

Table 1 : Comparison with prior models. On CheXpercept the three question levels are averaged with equal weight, and on CheXlocalize every case is queried at the chest level. The radiologists behind the human benchmark were told which finding to draw, so F1 is not defined for them. The best score among the models is shown in bold and the second best is underlined .

Model	Stage 1		Stage 2		CheXpercept			CheXlocalize
Model	SFT	RL	SFT	RL	gIoU	cIoU	F1	gIoU	cIoU	F1
ROSALIA					0.621	0.688	0.945	0.233	0.289	0.634
XFlow	✓				0.668	0.709	0.932	0.268	0.325	0.631
	✓	✓			0.693	0.718	0.961	0.280	0.327	0.632
	✓	✓	✓		0.699	0.726	0.961	0.280	0.329	0.632
	✓	✓	✓	✓	0.703	0.731	0.961	0.283	0.331	0.632

Table 2 : Contribution of each stage and training. Every component is added on top of the one before it. Stage 2 refines boxes Stage 1 has already emitted and never changes a presence decision, so it inherits Stage 1’s F1. ROSALIA is shown for reference and columns are as in Table 1 . The best score is shown in bold and the second best is underlined .

Figure 3 : Qualitative comparison of XFlow against baseline models.

Model	$\Delta C$	Specks
Model	$\Delta C$	count $\downarrow$	outside GT $\downarrow$	mask area % $\downarrow$
GT	—	0.01	—	0.01
XFlow (ours)	$+1.71$	1.69	1.06	0.14
ROSALIA	$+5.63$	5.57	3.43	2.60

Table 3 : Mask fragmentation analysis over all 1,141 positive cases in the CheXpercept evaluation set.

\Delta C

and specks are defined in Section 6.1 . Outside GT counts the specks that do not overlap the GT mask, and Mask area is the fraction of the predicted mask area occupied by specks.

Model	Log-strength $\uparrow$		Mean rank $\downarrow$
Model	Davidson	Rao–Kupper	Mean rank $\downarrow$
GT	0.326 [0.17, 0.48]	0.294 [0.16, 0.44]	1.79
XFlow (ours)	0.185 [0.01, 0.38]	0.170 [0.01, 0.33]	1.88
ROSALIA	$-0.511$ [ $-0.72$ , $-0.33$ ]	$-0.464$ [ $-0.64$ , $-0.30$ ]	2.33

Table 4 : Expert preference study. Three medical experts ranked three blinded masks on 95 balanced positive cases, giving 285 completed rankings. Log-strengths come from two tie-aware extensions of the Bradley–Terry model, Davidson and Rao–Kupper, which differ in how they treat a tie but agree here. Values are centred to sum to zero, with

95\%

bootstrap intervals over cases, and a difference of

d

means the stronger of the two masks is preferred

e^{d}

times as often when the reader does not tie.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Drawn from	Radiographs	Selection
$D_{\text{SFT}}$	MIMIC-ILS train $\setminus$ CheXpercept	$185{,}488$	everything the review did not cover
$D_{\text{SFT}}$	MIMIC-ILS val	$1{,}508$	MIMIC-ILS’s own split, unchanged
$D_{\text{RL}}$	MIMIC-ILS train $\cap$ CheXpercept	$1{,}296$	$70\%$ of the agreed set, by patient
$D_{\text{eval}}$	MIMIC-ILS train $\cap$ CheXpercept	$560$	the remaining $30\%$
MIMIC-ILS train, before the exclusion		$187{,}588$
CheXpercept, all of it inside MIMIC-ILS train		$2{,}100$

Appendix

Table 5 : How the splits divide the two sources. CheXpercept is a curated subset of the MIMIC-ILS training split, so holding it out leaves the supervised split disjoint from the other two by construction. Counts are radiographs; MIMIC-ILS carries one frontal radiograph per study, so they are also study counts.

Stage	Split	Unit	Examples
$\pi_{\text{init}}$ SFT	$D_{\text{SFT}}$	instruction	$1{,}031{,}506$ train / $8{,}246$ val
$\pi_{\text{init}}$ RL	$D_{\text{RL}}$	instruction	$13{,}488$ ( $2{,}605$ positive, $10{,}883$ negative)
$\pi_{\text{rev}}$ SFT	$D_{\text{SFT}}$	trajectory	$469{,}011$ train / $4{,}423$ val
$\pi_{\text{rev}}$ RL	$D_{\text{RL}}$	starting state	$58{,}320$

Appendix

Table 6 : Examples built from each split. The unit differs by stage:

\pi_{\text{init}}

is trained one instruction at a time, while one

\pi_{\text{rev}}

example is a whole revision trajectory of up to four turns.

Figure 4 : Example of SFT data for initial segmentation. The image shown at the top is what the <image> placeholder stands for; the box overlay is drawn here for illustration and is not part of the input.

Figure 5 : Example of SFT data for segmentation revision. The images shown at the top are what the <image> placeholders stand for.

	$F_{\text{seg}}$	$\pi_{\text{init}}$		$\pi_{\text{rev}}$
	$F_{\text{seg}}$	SFT	RL	SFT	RL
Initialized from	SAM3	MedGemma	$\pi_{\text{init}}$ SFT	$\pi_{\text{init}}$ SFT	$\pi_{\text{rev}}$ SFT
Adapted weights	all	LoRA $+$ ViT	LoRA	LoRA $+$ ViT	LoRA
LoRA $r$ / $\alpha$	—	$128$ / $256$	$128$ / $256$	$128$ / $256$	$128$ / $256$
Optimizer	AdamW, bf16
Learning rate	$1\times 10^{-4}$	$1\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-6}$

Appendix

Table 7 : Hyperparameters. Each policy is trained in two phases, and the RL phase starts from the SFT checkpoint of the same policy. A dash marks a setting that does not apply to that column. Effective batch size is per-device batch

\times

gradient accumulation

\times

8

GPUs, counted in samples for SFT and in generated completions for GRPO.

Finding	Chest		Lung		Zone
Finding	positive	negative	positive	negative	positive	negative
Atelectasis	$61$	$77$	$65$	$204$	$60$	$644$
Cardiomegaly	$57$	$25$	—	—	—	—
Consolidation	$49$	$102$	$62$	$254$	$57$	$772$
Edema	$58$	$103$	$110$	$253$	$40$	$760$
Effusion	$60$	$23$	$61$	$53$	$61$	$220$

Appendix

Table 8 : Distribution of the evaluation set. A positive row carries a mask and is scored by IoU against it; a negative row is answered correctly only with an empty mask. Cardiomegaly appears at the chest level alone, being a mediastinal finding for which the side- and zone-level questions do not arise.

Figure 6 : Additional qualitative comparisons across models. The layout follows Figure 3 .

Figure 7 : Review panel and annotation sheet.

Explore similar work

Jul 8, 2026cs.CV

Seeing What Matters: Lesion-Aware High-Resolution Patch Discovery and Fusion for Chest X-ray Report Generation

Despite rapid advances in chest X-ray (CXR) foundation models, most radiology report generation (RRG) systems still rely on heavily downsampled inputs (e.g., 256x256) due to the fixed visual token budgets of pretrained vision encoders, suppressing subtle yet clinically important cues present in native-resolution images. However, enabling high-resolution (high-res) perception remains challenging: naive tiling causes prohibitive token inflation, while global compression suppresses subtle lesions and degrades diagnostic fidelity. Inspired by radiologists' workflow, localizing suspicious regions before detailed high-res assessment. We propose Lesion-Aware High-Resolution Patch Discovery and Fusion for Chest X-ray Reporting (LePaX), the first RRG framework that enables efficient high-res CXR perception (up to 1920x1920) without increasing the vision-token count. LePaX formulates high-res perception as a constrained spatial resolution allocation problem under a fixed token budget and introduces two key components: Learnable Spatial Resolution Allocation (LSRA), which learns a spatial utility map that adaptively allocates limited high-res capacity to diagnostically relevant regions, enabling targeted extraction of high-res patches from native CXRs; and Global-Regional Fusion (GRF), which performs token-preserving region-to-global refinement by projecting high-resolution regional evidence back onto the global feature grid through spatially aligned resolution write-back, avoiding token inflation. Experiments on multiple CXR benchmarks demonstrate that LePaX consistently improves both clinical and linguistic metrics while enabling native-resolution CXR perception with over 10x fewer visual tokens than naive high-res tiling.

Yingshu Li, Yunyi Liu, Zhenghao Chen +5

The University of Sydney, Sydney, NSW 2006, Australia · The University of Newcastle, Newcastle, NSW 2308, Australia · University of Wollongong, Wollongong, NSW 2522, Australia +1

Chest X-Ray Radiology Report Generation Lesion High-Resolution

Ask PDF

May 18, 2026cs.CV

Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis

Medical image segmentation is more clinically valuable when it supports diagnosis rather than merely producing lesion masks. However, diagnostically relevant lesion cues are often subtle and localized, while existing models may be distracted by background tissues, acoustic artifacts, and irrelevant visual correlations. To address this problem, we propose Rad-VLSM, a two-stage cross-modal framework for semantics-assisted lesion focusing, robust segmentation, and visually grounded diagnosis. In the first stage, a BLIP-2-based vision-language alignment module identifies lesion-related candidate regions under semantic guidance and converts them into box prompts. In the second stage, these prompts are fed into a SAM-based multitask network, where a multi-candidate region aggregation strategy improves prompt stability and guides lesion segmentation. The predicted masks are then used as spatial priors for diagnosis, and a visual-radiomics fusion head integrates lesion-aware visual features with selected radiomics descriptors. By using semantic information for localization rather than direct prediction, Rad-VLSM reduces text-to-diagnosis dependence and grounds diagnosis in lesion-level evidence. Experiments on a private clinical breast ultrasound dataset and public benchmarks show that Rad-VLSM achieves strong segmentation and diagnostic performance with favorable generalization.

Fengyi Zhang, Xujie Zeng, Mohan Liu +2

Diagnosis Visible Image Fusion Cross-Modal

Ask PDF

Jun 19, 2026cs.CV

CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

The evaluation of vision-language models (VLMs) for chest X-ray (CXR) analysis has largely been limited to disease-presence classification without visual grounding. Such evaluations fail to verify the expert-level lesion perception necessary to ensure the clinical reliability of VLMs. To address these limitations, we introduce CheXpercept, a sequential, multi-level perception benchmark that mirrors a radiologist's cognitive workflow across coarse-level detection, fine-level contour evaluation and revision, and semantic-level attribute extraction. To ensure high clinical fidelity at scale, we construct the dataset using a semi-automated generation pipeline paired with a review by six medical experts. CheXpercept contains 10,400 QA items derived from 2,100 CXRs, covering seven clinically critical pulmonary and cardiac lesions. To demonstrate the current landscape of VLM perception, we benchmark 14 general and medical VLMs on CheXpercept. The models achieve adequate performance only at the coarse level, with accuracy degrading precipitously on deeper visual tasks. Notably, medical VLMs show almost no perceptual advantage over their general-domain counterparts, highlighting a systemic flaw in current domain adaptation. The code and dataset will be publicly available.

Geon Choi, Hangyul Yoon, Nalee Kim +5

1KAIST · 2Samsung Medical Center · 3Konkuk University Medical Center

Chest X-Ray Medical Vision-Language Models Lesion Perception Disease

Ask PDF