Segment What It Can Draw: Querying a Text-to-Image Generator with Its Own Images for Skin Lesion Segmentation
Organizations: Clemson University, Clemson, SC, USA · Analog Devices, Inc., USA · Wake Forest University, USA
Abstract
Skin lesion segmentation without annotated masks requires the target to be specified through other cues. Existing approaches rely on dermoscopy-specific rules, annotation-derived model priors, or user-selected regions. A general text-to-image diffusion transformer offers a text interface to the target, but its concept-word attention is unreliable on dermoscopy images: in the model's attention-output space, the word-derived skin-to-lesion direction is poorly aligned with the direction separating real skin from real lesions. We show that the direction derived from generated healthy-skin and lesion images is substantially better aligned. Based on this finding, we propose GENIQ, a training-free method that constructs initial skin and lesion prototypes from generated images and refines them using the most skin-like and lesion-like patches from unlabelled dermoscopy images. It scores test patches along the refined direction and applies a gated threshold to suppress contextual spill-over around small lesions. Without inference-time annotations or additional models, GENIQ achieves Dice scores of 0.85-0.88 on three dermoscopy test sets. It outperforms zero-shot and word-based attention methods on all three sets, and a hand-designed dermoscopy pipeline in Dice and IoU on two.
Figures & tables
| ISIC-2018 (1,000) | PH2 (200) | HAM10000 (1,000) | |||||
| Method | Dice | IoU | Dice | IoU | Dice | IoU | |
| Mean-mask prior (no image) | 0.620 ±0.012 | 0.474 ±0.011 | 0.642 ±0.019 | 0.489 ±0.021 | 0.640 ±0.011 | 0.495 ±0.011 | |
| Zero- shot | MedCLIP-SAMv2 | 0.463 ±0.023 | 0.384 ±0.022 | 0.362 ±0.046 | 0.289 ±0.043 | 0.508 ±0.021 | 0.413 ±0.019 |
| ADZUS (DiffSeg) ‡ | 0.536 ±0.017 | 0.421 ±0.018 | 0.524 ±0.035 | 0.403 ±0.036 | 0.458 ±0.015 | 0.335 ±0.015 | |
| FLUX | ConceptAttention, layers 14–17 | 0.549 ±0.018 | 0.431 ±0.017 | 0.694 ±0.026 | 0.562 ±0.028 | 0.683 ±0.016 | 0.570 ±0.016 |
| ConceptAttention, layers 9–13 | 0.706 ±0.013 | 0.579 ±0.014 | 0.769 ±0.024 | 0.654 ±0.028 | 0.785 ±0.012 | 0.680 ±0.013 | |
Skin Lesion Segmentation
Skin lesion segmentation identifies lesion pixels in dermoscopic images to support measurement, diagnosis, and consistent analysis.
See all 18 papers