Scribble-supervised segmentation is typically evaluated with a fixed annotation set, leaving robustness to distinct semantic and geometric imperfections poorly understood. Motivated by an audit of 12,031 manual PASCAL Visual Object Classes (VOC) 2012 scribbles and prior work, we introduce six controlled perturbation operators. Class corruption, spatial offset, and directional perturbation directly operationalize audit signals. Stroke-length reduction is motivated by prior scribble studies, while random dropout and fragmentation serve as diagnostic controls. Each operator reports requested and achieved changes. We evaluate manual and automatic scribbles across two image domains, two principal backbone settings, and multiple training pipelines, with an additional MiT-B1 PFA anchor evaluation. In the five-seed Tree Energy Loss (TEL)-ResNet101 VOC study, mean intersection over union (mIoU) drops from 70.08 to 1.46 under class corruption, compared with 35.57 for spatial offset, 52.20 for directional perturbation, and 63.24 for stroke-length reduction at their strongest settings. At the evaluated endpoints, semantic-anchor corruption produces the largest degradation, while geometric and support responses depend on the pipeline and dataset. Our benchmark provides a reproducible protocol for diagnosing annotation robustness beyond clean-set accuracy. The operator code is available at https://github.com/Wmhiram99/scribble-operator-benchmark.
Figures & tables
Figure 1: Noisy scribble semantic segmentation benchmark. Audit signals motivate controlled annotation perturbations, whose achieved properties are verified before matched training and robustness evaluation. The inset reports maximum mIoU drops in percentage points for TEL–ResNet101 across the nine ratios. VOC2012 uses the single-run full-ratio path; S4Pascal and S4Cityscapes are descriptive single-run paths. C, O, D, and S denote Class, Offset, Direction, and Shrink
Screening signal
Count
Share (%)
Image-level signals ( n=12,031 )
Foreground-class disagreement
1,556
12.93
Foreground scribble on GT background
7,272
60.44
Background scribble on GT foreground
1,328
11.04
Spatial-disagreement candidate
1,831
15.22
Scribble class absent from GT
15
0.12
Table 1: Screening results for manual PASCAL VOC2012 scribbles. The first five rows are image-level signals and may co-occur in one image. The direction analysis uses 17,937 eligible foreground components of 28,227 total.
Operator
Basis
Target property
Primary operators
Class
Corpus audit
Semantic identity
Offset
Corpus audit
Spatial alignment
Direction
Corpus audit
Trajectory direction
Shrink
Prior work
Coherent support
Diagnostic operators
Table 2: The six perturbation operators and their target properties. The primary set spans four annotation properties. Dropout and Fragmentation are diagnostic controls. We distinguish isolated from coupled operational control.
Figure 2: Six manual-VOC operations on image 2007_000256 ( Everingham et al., 2010 ) . The leftmost column shows the GT for visual reference; the remaining columns show generated labels at r=0.9,0.5,0.1 . In the class row, relabeling denotes replacement of a selected whole component’s class. Per-operator generation success and achieved-value distributions are reported in the supplementary material
Figure 3: Requested and achieved operator scores and image-level target attainment. Each panel is one dataset–operator pair. The left axis shows the requested score and achieved mean; the right axis shows the percentage of images with absolute score error at most 0.05 . The clean/identity reference at r=1.0 is shown for orientation. Its perfect attainment is expected by construction and it is excluded from perturbed-level summaries. Scores remain operator-specific
Dataset
Operator
Images
Mean level abs. error
Mean within-0.05 (%)
r=.1 achieved
r=.1 within-0.05 (%)
VOC2012 (manual)
Class
12,031
0.039
52.1
0.073
54.0
VOC2012 (manual)
Offset
12,031
0.052
73.8
0.203
46.2
VOC2012 (manual)
Direction
12,031
0.007
89.9
0.062
92.9
VOC2012 (manual)
Shrink
12,031
0.013
89.9
0.130
87.6
S4Pascal (auto)
Class
12,031
0.030
54.2
0.067
55.0
S4Pascal (auto)
Offset
12,031
0.050
72.0
0.197
39.4
Table 3: Generator target reachability over the nine perturbed levels r=0.1 – 0.9 . Mean level absolute error is the mean across the nine conditions of the absolute difference between achieved and requested mean scores; mean target attainment is the mean image-level fraction with absolute score error no greater than 0.05. The r=1.0 identity/clean reference is excluded. Scores are operator-specific and are not physical translations or angles.
Figure 4: Cross-property score ratios in each target-operator output over r=0.1 – 0.9 . Each column identifies the target operation and each row a dataset. Only non-target properties are plotted; the dashed horizontal line marks ratio one. The vertical axis is restricted to the observed near-neutral range to magnify small deviations. Distances from the reference should therefore be interpreted using the numeric values, not as a full-scale visual effect size. Score ratios use the report-specific fixed-binding or native score definitions described in the text
Dataset
Target operator
Changed pixels (%)
Changed/support (%)
Rclass
Roffset
Rdirection
Original-position overlap
VOC2012 (manual) ( n=12,031 )
Class
2.90
92.43
0.076
1.000
1.000
1.000
VOC2012 (manual) ( n=12,031 )
Offset
4.16
132.20
0.897
0.283
1.001
0.337
VOC2012 (manual) ( n=12,031 )
Direction
2.20
66.97
0.995
0.908
0.095
0.654
VOC2012 (manual) ( n=12,031 )
Shrink
2.74
87.07
0.998
0.768
1.067
0.130
S4Pascal (auto) ( n=12,031 )
Class
2.41
93.28
0.067
1.000
1.000
1.000
S4Pascal (auto) ( n=12,031 )
Offset
2.75
106.59
0.960
0.304
0.995
0.465
Table 4: Cross-property measurements on the r=0.1 target-operator outputs. Changed pixels use the full image area as denominator, and changed/support uses the original annotated support area as denominator. The latter can exceed 100% because edits may affect pixels outside original support. Rclass is same-position label retention, Roffset is the report-defined Offset score ratio (except Cityscapes Class, which uses the fixed-binding geometric control), and Rdirection is the report-defined fixed-binding ratio for side effects or the generator-native target ratio for Direction outputs. Values are computed per image and averaged equally over images with valid denominators. The final column reports original-position support overlap, ∣U∩V∣/∣U∣ , where U and V denote the input and output labeled support. This measures spatial overlap with the input support, not total output support area; translation can reduce it without deleting support. Ratios above one are retained and are not probabilities. Direction side-effect scores use fixed original component–GT binding with pixel-weighted component aggregation for S4Pascal. These score ratios are not interchangeable physical units.
Figure 5: Five-seed Manual-VOC TEL–ResNet101 response curves for the four retained operations. Lower r denotes a stronger within-operator intervention. Shaded bands show ±1 standard deviation across five seeds
Figure 6: TEL–ResNet101 cross-dataset response paths. Each panel shows clean-normalized deviation C(r) . VOC2012 shows the five-seed mean path, while S4Pascal and S4Cityscapes show descriptive single-run full paths. Lower r denotes stronger perturbation within each operator
VOC2012
S4Pascal
S4Cityscapes
Operation
R Cavg/Cend
S Cavg/Cend
R Cavg/Cend
S Cavg/Cend
R Cavg/Cend
S Cavg/Cend
Class
0.631 / 0.979
0.632 / 0.967
0.636 / 0.976
0.643 / 0.979
0.555 / 0.976
0.597 / 0.980
Offset
0.328 / 0.493
0.378 / 0.513
0.239 / 0.374
0.268 / 0.383
0.215 / 0.603
0.204 / 0.585
Direction
0.209 / 0.255
0.280 / 0.328
0.159 / 0.190
0.178 / 0.214
0.032 / 0.063
0.023 / 0.050
Shrink
0.031 / 0.098
0.034 / 0.041
0.055 / 0.116
0.034 / 0.056
0.091 / 0.215
0.042 / 0.109
Table 5: Descriptive TEL response across datasets and backbones. R and S denote ResNet-101 and SegFormer-B4. Cavg/Cend are the average clean-normalized response over r∈{0.9,…,0.1} and the response at r=0.1 . Higher values indicate larger deviation from the clean reference. The table provides numerical summaries for Figure 6 . VOC2012–ResNet101 uses five-seed mean paths. The remaining full-path extensions are descriptive single-run summaries rather than statistical backbone comparisons.
Figure 7: Pipeline sensitivity across matched TEL and DTS settings, measured by Cavg over the nine perturbed ratios. TEL–ResNet101 on VOC2012 uses the five-seed mean curve. All other settings, including DTS, use single-run full-ratio sweeps. Darker cells indicate greater clean-normalized response
Pipeline
Backbone
Dataset
Noise
n
Validation mIoU (mean ± sample SD)
r=1.0
r=0.9
r=0.5
r=0.1
TEL
ResNet101
VOC2012
Class
5
70.08±0.42
65.04±0.61
7.96±0.25
1.46±0.12
Offset
5
70.08±0.42
64.03±0.39
46.03±0.68
35.57±0.98
Direction
5
70.08±0.42
59.66±0.59
54.94±0.52
52.20±0.44
Shrink
5
70.08±0.42
69.83±0.27
68.77±0.82
63.24±1.25
TEL
ResNet101
S4Pascal
Class
4
75.76±0.17
69.43±0.19
13.07±0.54
1.46±0.10
Table 6: Repeated-seed results for the three settings with four reported anchor ratios (percentages; mean ± sample SD). Pipeline, backbone, and dataset are separated explicitly. TEL entries use the tail- 25% validation average, whereas DTS entries use the best validation checkpoint under its logged protocol. Repeated r=1.0 values are the shared clean reference within each setting.
Pipeline
Backbone
Dataset
Noise
n
Validation mIoU (mean ± sample SD)
r=1.0
r=0.1
PFA
MiT-B1
VOC2012
Class
3
78.32±0.10
4.46±0.06
Offset
3
78.32±0.10
39.94±0.05
Direction
3
78.32±0.10
58.83±0.03
Shrink
3
78.32±0.10
73.25±0.37
TEL
SegFormer-B4
VOC2012
Class
3
73.75±0.12
3.87±0.12
Table 7: Repeated-seed anchor-only results for the two settings evaluated at clean and strong noise (percentages; mean ± sample SD). TEL entries use the tail- 25% validation average, whereas PFA entries use the best validation checkpoint. Only clean and endpoint conditions were evaluated; intermediate response shapes are therefore not assessed.
Curating fully annotated datasets for medical image segmentation is labour-intensive and expertise-demanding. To alleviate this problem, prior studies have explored scribble annotations for weakly supervised segmentation. Existing solutions mainly compute losses on annotated areas and generate pseudo labels by propagating annotations to adjacent regions. However, these methods often suffer from inaccurate and unrealistic segmentations due to insufficient supervision and incomplete shape information. In contrast, we first investigate the principle of good scribble annotations, which leads to efficient scribble forms via supervision maximization and randomness simulation. We further introduce regularization terms to encode the spatial relationship and the shape constraints, where the EM algorithm is utilized to estimate the mixture ratios of label classes. These ratios are critical in identifying the unlabeled pixels for each class and correcting erroneous predictions, thus the accurate estimation lays the foundation for the incorporation of spatial prior. Finally, we integrate the efficient scribble supervision with the prior into a framework, referred to as ZScribbleSeg, and apply it to multiple scenarios. Leveraging only scribble annotations, ZScribbleSeg achieves competitive performance on six segmentation tasks including ACDC, MSCMRseg, BTCV, MyoPS, Decathlon-BrainTumor and Decathlon-Prostate. Our code will be released via https://github.com/DLwbm123/ZScribbleSeg.
Ke Zhang, Bomin Wang, Hangqi Zhou +1
School of Data Science, Fudan University, Shanghai, 200433, China · Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, USA
Scribble annotations offer an efficient alternative to costly pixel-wise labeling for medical image segmentation, yet in real clinical scenarios, scribble-annotated samples are often still limited, imposing the dual challenges of sparse supervision and annotated sample scarcity. These compounded constraints severely deprive models of the structural evidence needed for complete region recovery and precise boundary delineation. To break this bottleneck, we propose a bi-level collaborative learning framework for few-shot scribble-supervised medical image segmentation. Specifically, an upper-level learnable superpixel model is introduced to provide region-structural priors for lower-level segmentation, while superpixel-based region-wise pseudo-label propagation and a spatial-prior-guided filtering strategy are performed to generate reliable dense pseudo-labels for segmentation learning. Meanwhile, the anatomical semantics learned by the lower-level segmentation model under the guidance of the current superpixels are fed back to the upper level, further driving it to learn region-structural representations better aligned with the segmentation task. Through bidirectional interaction and collaborative learning between the upper and lower levels, the proposed framework significantly outperforms existing state-of-the-art scribble-supervised methods on the ACDC and Prostate datasets under the few-shot scribble-supervised setting.
Xiang-Xiang Su, Yufan Ye, Yihang Zheng +2
Fuzhou University Fuzhou, China · Qingdao University Qingdao, China
Accurate segmentation of tissues and instruments in surgical scenes is annotation-intensive due to irregular shapes, thin structures, specularities, and frequent occlusions. While SAM models support point, box, and mask prompts, points are often too sparse and boxes too coarse to localize such challenging targets. We present SCISSR, a scribble-promptable framework for interactive surgical scene segmentation. It introduces a lightweight Scribble Encoder that converts freehand scribbles into dense prompt embeddings compatible with the mask decoder, enabling iterative refinement for a target object by drawing corrective strokes on error regions. Because all added modules (the Scribble Encoder, Spatial Gated Fusion, and LoRA adapters) interact with the backbone only through its standard embedding interfaces, the framework is not tied to a single model: we build on SAM 2 in this work, yet the same components transfer to other prompt-driven segmentation architectures such as SAM 3 without structural modification. To preserve pre-trained capabilities, we train only these lightweight additions while keeping the remaining backbone frozen. Experiments on EndoVis 2018 demonstrate strong in-domain performance, while evaluation on the out-of-distribution CholecSeg8k further confirms robustness across surgical domains. SCISSR achieves 95.41% Dice on EndoVis 2018 with five interaction rounds and 96.30% Dice on CholecSeg8k with three interaction rounds, outperforming iterative point prompting on both benchmarks.
Haonan Ping, Jian Jiang, Cheng Yuan +3
Global College, Shanghai Jiao Tong University, Shanghai, China