cs.CVSep 28, 2026
SaveVL-AcneSeg: A Vision-Language Framework for Region-Aware Acne Lesion Segmentation
Organizations: Department of Computer Science and Artificial Intelligence, Dongguk University, Seoul 04620, Republic of Korea · InSkin Lab Inc., Seoul 06193, Republic of Korea · Department of Dermatology, Seoul National University College of Medicine, and also with the Department of Dermatology, Seoul National University Hospital, Seoul 03080, Republic of Korea
Abstract
Acne assessment is crucial for clinical decision-making, yet traditional grading and counting are subjective and fail to account for lesion size. While area-based assessment has emerged as a promising alternative, acne segmentation has continued to rely on general-purpose architectures. To address this gap, we propose VL-AcneSeg, a multimodal framework for acne lesion segmentation that leverages CLIP and region-level text prompts to incorporate spatial priors, enabling lesions to be localized across the whole face. Because region-level prompts indicate which facial areas contain lesions, we report a single global prompt, which requires no such information, as our primary setting. On our internal clinical dataset, VL-AcneSeg achieves a Dice score of 0.5082 and an IoU of 0.3407 under this protocol, the highest among all compared methods, including recent vision-language segmentation methods that are themselves given region-level prompts; region-level prompting raises these to 0.5296 and 0.3602. Moreover, lesion area measurements derived from our segmentation correlate with IGA scores at a level comparable to expert annotations (Pearson r = 0.719 versus 0.658). Notably, our framework maintains consistent performance across external validation datasets, performing reliably even on uncontrolled smartphone images without requiring additional training or fine-tuning. By pairing a protocol that requires no lesion-location information with area-based severity estimation, this work provides a foundation for objective acne assessment outside the clinic. Our implementation is publicly available at: https://github.com/sukjuoh/VL-AcneSeg
Figures & tables
Fig. 1 : Conceptual framework of VL-AcneSeg. Given a regional text prompt, VL-AcneSeg localizes potential lesion sites through CLIP-based semantic alignment (Middle) and then performs refinement to produce the segmentation mask (Right).
Fig. 2 : Facial regions were defined using Mediapipe Face Mesh [ 46 ] , and lesion masks were assigned to each region according to their centroid location. Region-wise masks and text prompts (“acne lesion on the { region }”) were then generated to provide spatial guidance for segmentation.
Fig. 3 : Overview of VL-AcneSeg. Overlapping tiles are encoded by CLIP and reassembled into a unified feature map, whose patch-wise cosine similarity with the text embeddings yields the localization map; intermediate vision features and the CLS token provide additional guidance. A Swin Transformer, a CNN bottleneck and a decoder then refine this map into the lesion mask.
| Split | Patients | Images | Lesions |
| Internal (Seoul National University Hospital) | |||
| Train | 190 | 703 | 3,814 |
| Validation | 16 | 60 | 494 |
| Test | 16 | 58 | 347 |
| Total | 222 | 821 | 4,655 |
| External (AI Hub) | |||
TABLE I : Composition of the internal splits and the two external test sets, counted over the inflammatory lesion masks used in this study. The full annotated cohort comprises 258 patients, 1,213 images and 20,699 lesions across five types. The internal dataset was divided patient-wise, and each external image originates from a distinct subject.
| Model | Backbone | Internal Data | External Data (Evaluation Only) | ||||||||||||||||
| (Training & Evaluation) | Controlled Setting (1) | Real-world Setting (2) | |||||||||||||||||
| IoU | Dice | Precision | Recall | TP/img | FP/img | IoU | Dice | Precision | Recall | TP/img | FP/img | IoU | Dice | Precision | Recall | TP/img | FP/img | ||
| VL-AcneSeg (Region) ∗ | ViT-L | 0.3602 [0.309, 0.399] | 0.5296 [0.472, 0.570] | 0.5202 | 0.5393 | 4.29 | 2.48 | 0.4232 [0.376, 0.470] | 0.5948 [0.546, 0.640] | 0.6110 | 0.5794 | 1.64 | 0.60 | 0.3044 [0.2476, 0.3483] | 0.4667 [0.3969, 0.5166] | 0.4882 | 0.4471 | 1.50 | 0.82 |
| SED [ 35 ] ∗ | ConvNeXt-L | 0.3182 [0.255, 0.373] | 0.4828 [0.406, 0.544] | 0.6291 | 0.3917 | 4.09 | 3.74 | 0.3070 [0.263, 0.355] | 0.4698 [0.416, 0.524] | 0.7509 | 0.3419 | 1.97 | 1.51 | 0.1650 [0.121, 0.229] | 0.2833 [0.215, 0.373] | 0.6365 | 0.1822 | 0.98 | 0.32 |
| CAT-Seg [ 34 ] ∗ | ViT-L | 0.2909 [0.226, 0.374] | 0.4507 [0.368, 0.545] | 0.5832 | 0.3673 | 3.09 | 1.45 | 0.2287 [0.202, 0.255] | 0.3722 [0.336, 0.407] | 0.4101 | 0.3407 | 1.70 | 4.44 | 0.1300 [0.098, 0.160] | 0.2300 [0.179, 0.276] | 0.2050 | 0.2622 | 0.82 | 0.46 |
| EviVLM [ 45 ] ∗ | ViT-B | 0.3290 [0.261, 0.385] | 0.4952 [0.414, 0.556] | 0.5289 | 0.4654 | 4.57 | 5.72 | 0.2584 [0.213, 0.311] | 0.4106 [0.351, 0.474] | 0.7417 | 0.2839 | 1.00 | 0.24 | 0.2487 [0.182, 0.312] | 0.3984 [0.308, 0.476] | 0.5134 | 0.3254 | 0.66 | 0.82 |
TABLE II : Quantitative comparison of acne lesion segmentation on the internal and external datasets. Shaded rows ( ) are vision-language methods given region-level prompts derived from the reference masks, an oracle setting; CAT-Seg and SED use the same tiling scheme and input resolution as the proposed method, so that the CLIP-based methods are compared at matched resolution. All remaining rows, including VL-AcneSeg (Global), require no reference-derived information. TP/img and FP/img are the numbers of predicted lesion components per image that do or do not overlap a reference lesion; neither is ranked. Point estimates are computed over all test-set pixels; 95% confidence intervals are obtained by patient-level cluster bootstrap (1,000 resamples). Best values are in bold and second-best are underlined .
Fig. 4 : Qualitative comparison across the three datasets : (A) internal, (B) external controlled, and (C) external smartphone images. The left column shows the reference annotation on the full face; the green box marks the region enlarged in the remaining columns, so that lesions a few pixels across remain visible at print size. Predicted masks are overlaid in red, and both prompting conditions of the proposed method are shown. A comparison including all baselines and both prompting conditions is given in Figs. S3 – S5 of the Supplementary Material.
| Metrics | Full Model | w/o cls | w/o v-v | w/o region | all-layer |
| token | attention | mask loss | v-v | ||
| Internal | |||||
| IoU | 0.3602 [0.309, 0.399] | 0.3563 [0.294, 0.398] | 0.3420 [0.290, 0.380] | 0.3510 [0.290, 0.394] | 0.2832 [0.215, 0.336] |
| Dice | 0.5296 [0.472, 0.570] | 0.5254 [0.458, 0.568] | 0.5097 [0.451, 0.552] | 0.5196 [0.454, 0.565] | 0.4414 [0.357, 0.505] |
| Precision | 0.5202 | 0.5149 | 0.4936 | 0.4935 | 0.3700 |
| Recall | 0.5393 | 0.5363 | 0.5268 | 0.5486 | 0.5449 |
TABLE III : Ablation study on the model components of VL-AcneSeg. Confidence intervals and values are obtained by paired cluster bootstrap against the Full Model, at the patient level for the internal dataset and at the image level for the external datasets. Specificity exceeds 0.999 everywhere and is omitted. FP lesions/img counts predicted lesion components matching no reference lesion. Best values are in bold and second-best are underlined .
| Metrics | patches | patches | patches |
| IoU | 0.3602 | 0.3194 | 0.2700 |
| Dice | 0.5296 | 0.4841 | 0.4253 |
| Precision | 0.5202 | 0.4308 | 0.4070 |
| Recall | 0.5393 | 0.5524 | 0.4452 |
| TP lesions/img | 4.29 | 3.67 | 3.14 |
| FP lesions/img | 2.48 | 2.72 | 1.71 |
TABLE IV : Ablation study on different image patch resolutions for CLIP-based processing in VL-AcneSeg. Best values are in bold and second-best are underlined .
| Prompt representation | IoU | Dice | Precision | Recall |
| Internal | ||||
| Regional text prompt | 0.3602 [0.309, 0.399] | 0.5296 [0.472, 0.570] | 0.5202 | 0.5393 |
| Unseen synonyms | 0.3422 [0.291, 0.392] | 0.5100 [0.451, 0.563] | 0.5272 | 0.4938 |
| Shuffled text prompt | 0.3039 [0.251, 0.356] | 0.4661 [0.402, 0.525] | 0.5425 | 0.4085 |
| Learnable embedding | 0.3565 [0.301, 0.399] | 0.5256 [0.463, 0.570] | 0.4654 | 0.6037 |
| One-hot region encoding | 0.3111 [0.260, 0.357] | 0.4745 [0.413, 0.526] | 0.4313 | 0.5274 |
TABLE V : Where the performance originates. Each row removes one element of the vision–language pipeline. Unseen synonyms and Shuffled text prompt keep the CLIP text encoder but substitute synonyms absent from training or pair each image with the prompts of another image, testing whether the prompt is read as a reference to a particular area. Learnable embedding and One-hot region encoding replace the text embedding with a non-linguistic region code, separating language from spatial conditioning. CLIP vision encoder only discards the text branch altogether, and w/o CLIP removes the CLIP encoders as well, so that the interval between the two isolates the contribution of the pretrained visual representation. The last four settings were retrained.
Fig. 5 : Anatomical region discrimination through text-guided spatial priors. Top row: region names used during training. Bottom row: anatomical terms never seen during training, paired column-wise with the region above.
Fig. 6 : Segmentation accuracy under prompt perturbation on the internal test set. The 58 test images carry 165 lesion-bearing (positive) region prompts and 705 lesion-free (negative) regions. Each pool was shuffled once with a fixed seed, and the perturbation level denotes the fraction of that pool that is withheld (FN) or added (FP); the perturbed sets are nested. When every prompt of an image is withheld, it falls back to the single global prompt, marked by the dashed line.
Fig. 7 : Model response under adversarial prompting. Prompts for lesion-bearing regions are omitted and misleading prompts are issued for lesion-free regions.
Fig. 8 : Failure case analysis of VL-AcneSeg. Red regions denote acne masks. Blue boxes mark false positives, predominantly PIH, scars and prominent pores; orange boxes mark false negatives, typically small or low-contrast lesions.
| Source of area ratio | Pearson | Spearman | ||
| [95% CI] | [95% CI] | |||
| Expert annotation (GT) | 0.658 [0.441, 0.807] | — | 0.710 [0.541, 0.793] | — |
| VL-AcneSeg (region) | 0.719 [0.477, 0.848] | 0.430 | 0.684 [0.485, 0.799] | 0.719 |
| VL-AcneSeg (global) | 0.695 [0.442, 0.833] | 0.621 | 0.695 [0.465, 0.843] | 0.868 |
| EviVLM | 0.640 [0.361, 0.796] | 0.839 | 0.660 [0.392, 0.809] | 0.598 |
| CAT-Seg | 0.650 [0.407, 0.797] | 0.924 | 0.685 [0.479, 0.812] | 0.737 |
TABLE VI : Correlation between the predicted lesion-to-face area ratio and IGA severity on the internal test set (22 visits from 16 patients). values compare each coefficient with that of the expert annotations using a test for dependent correlations sharing one variable; confidence intervals are obtained by patient-level cluster bootstrap.
| Params (M) | FLOPs (G) | Time (ms) | |
| VL-AcneSeg (global prompt) | 432 | 1,417 | 858 |
| VL-AcneSeg (all regions) | 432 | 1,676 | 1,536 |
| SAM (ViT-L) | 312 | 1,312 | 594 |
| EviVLM | 130 | 567 | 176 |
| Swin-UMamba | 27 | 25 | 64 |
| U-Net | 17 | 428 | 74 |
TABLE VII : Computational cost, measured on a single NVIDIA A6000 at the input resolution used throughout. For VL-AcneSeg, all regions denotes prompts issued for all fifteen facial regions; the tiled CLIP encoding is shared across prompts.
| Seen (training) | Unseen 1 | Unseen 2 |
| forehead | supraorbital area | frontal region |
| left cheek | left malar region | left buccal area |
| right cheek | right malar region | right buccal area |
| chin | mentum | mental region |
| nose | nasal region | nasal bridge |
| left temple | left pterion area | left temporal region |
TABLE S1 : Anatomical synonyms used in the unseen-prompt experiment. Each training region name was replaced at inference by one of the two alternatives, chosen at random; none appears in training.
| Metrics | Region Prompt | Level Prompt | Num Prompt |
| IoU | 0.3602 | 0.3521 | 0.3594 |
| Dice | 0.5296 | 0.5208 | 0.5288 |
| Precision | 0.5202 | 0.4889 | 0.4940 |
| Recall | 0.5393 | 0.5572 | 0.5688 |
TABLE S2 : Ablation study on different text prompt strategies for VL-AcneSeg in internal data. Best values are in bold and second-best are underlined .
Fig. S1 : Ablation study on patch granularity and regional textual cues. Finer patch division improves localization precision. The model still localizes lesions under a global text prompt alone (e).
Fig. S2 : Localization heatmaps in a smartphone environment. (b) The full model identifies lesion-suspected areas with high precision and sharpness. (c) Although the global text prompt configuration exhibits broader localization extending beyond facial boundaries, it still captures the actual lesion sites.
| Ablation Model | External Datasets (Evaluation Only) | |||||||||||
| Controlled Setting (1) | Real-world Setting (2) | |||||||||||
| IoU | Dice | Precision | Recall | TP/img | FP/img | IoU | Dice | Precision | Recall | TP/img | FP/img | |
| VL-AcneSeg (Full Model) | 0.4232 | 0.5948 | 0.6110 | 0.5794 | 1.64 | 0.60 | 0.3044 | 0.4667 | 0.4882 | 0.4471 | 1.50 | 0.82 |
| Patches | 0.3812 | 0.5520 | 0.5664 | 0.5383 | 1.83 | 0.79 | 0.2443 | 0.3927 | 0.3787 | 0.4077 | 1.11 | 0.66 |
| Patches | 0.3019 | 0.4638 | 0.4593 | 0.4683 | 1.58 | 0.77 | 0.1213 | 0.2164 | 0.1354 | 0.5383 | 0.89 | 1.43 |
TABLE S3 : Effect of patch tiling resolution on the two external datasets. Component-wise ablations on the external datasets are reported in the main paper. Best values are in bold and second-best are underlined .
| Baseline | IoU [95% CI] | p | Dice [95% CI] | p |
| Vision-language methods, compared against VL-AcneSeg (region) | ||||
| EviVLM | 0.031 [0.001, 0.069] | 0.048 | 0.034 [0.001, 0.080] | 0.046 |
| CAT-Seg | 0.069 [0.012, 0.136] | 0.013 | 0.079 [0.011, 0.166] | 0.015 |
| SED | 0.042 [0.003, 0.097] | 0.034 | 0.047 [0.003, 0.112] | 0.038 |
| Remaining methods, compared against VL-AcneSeg (global) | ||||
| SAM | 0.019 [ 0.037, 0.078] | 0.580 | 0.021 [ 0.041, 0.091] | 0.592 |
TABLE S4 : Differences in IoU and Dice between VL-AcneSeg and each baseline on the internal test set, with 95% confidence intervals from paired patient-level cluster bootstrap over the 16 test patients. A positive value favours the proposed method. Vision-language baselines receive region-level prompts and are therefore compared against the region-level configuration; the remaining methods take no text input and are compared against the global prompt.
| Model | Small lesions ( px 2 ) | AUPRC | ||
| Recall | Precision | F1 | ||
| VL-AcneSeg (region) | 0.494 | 0.451 | 0.472 | 0.4138 |
| VL-AcneSeg (global) | 0.492 | 0.432 | 0.460 | 0.4068 |
| SAM | 0.457 | 0.432 | 0.444 | 0.3400 |
| SED | 0.431 | 0.426 | 0.428 | 0.4167 |
| EviVLM | 0.453 | 0.405 | 0.428 | 0.4037 |
TABLE S5 : Additional metrics on the internal test set. Small lesions are those in the lowest quartile of the lesion-area distribution over the whole annotated set, corresponding to an area below 500 px 2 ; recall, precision and F1 are computed over these lesions at the lesion level. AUPRC is the area under the precision–recall curve, computed from the predicted probability maps and therefore independent of the binarization threshold. Rows are ordered by small-lesion F1. Best values are in bold and second-best are underlined .
Fig. S3 : Comprehensive qualitative comparison on the internal dataset (Part I). The left panel shows the input with the reference annotation; the green box marks the region enlarged in the remaining panels. Predicted masks are overlaid in red.
Fig. S4 : Comprehensive qualitative comparison on the external controlled dataset (Part II).
Fig. S5 : Comprehensive qualitative comparison on the external smartphone dataset (Part III).