Scribble-supervised segmentation is typically evaluated with a fixed annotation set, leaving robustness to distinct semantic and geometric imperfections poorly understood. Motivated by an audit of 12,031 manual PASCAL Visual Object Classes (VOC) 2012 scribbles and prior work, we introduce six controlled perturbation operators. Class corruption, spatial offset, and directional perturbation directly operationalize audit signals. Stroke-length reduction is motivated by prior scribble studies, while random dropout and fragmentation serve as diagnostic controls. Each operator reports requested and achieved changes. We evaluate manual and automatic scribbles across two image domains, two principal backbone settings, and multiple training pipelines, with an additional MiT-B1 PFA anchor evaluation. In the five-seed Tree Energy Loss (TEL)-ResNet101 VOC study, mean intersection over union (mIoU) drops from 70.08 to 1.46 under class corruption, compared with 35.57 for spatial offset, 52.20 for directional perturbation, and 63.24 for stroke-length reduction at their strongest settings. At the evaluated endpoints, semantic-anchor corruption produces the largest degradation, while geometric and support responses depend on the pipeline and dataset. Our benchmark provides a reproducible protocol for diagnosing annotation robustness beyond clean-set accuracy. The operator code is available at https://github.com/Wmhiram99/scribble-operator-benchmark.
Figures & tables
Figure 1: Noisy scribble semantic segmentation benchmark. Audit signals motivate controlled annotation perturbations, whose achieved properties are verified before matched training and robustness evaluation. The inset reports maximum mIoU drops in percentage points for TEL–ResNet101 across the nine ratios. VOC2012 uses the single-run full-ratio path; S4Pascal and S4Cityscapes are descriptive single-run paths. C, O, D, and S denote Class, Offset, Direction, and Shrink
Screening signal
Count
Share (%)
Image-level signals ( n=12,031 )
Foreground-class disagreement
1,556
12.93
Foreground scribble on GT background
7,272
60.44
Background scribble on GT foreground
1,328
11.04
Spatial-disagreement candidate
1,831
15.22
Scribble class absent from GT
15
0.12
Table 1: Screening results for manual PASCAL VOC2012 scribbles. The first five rows are image-level signals and may co-occur in one image. The direction analysis uses 17,937 eligible foreground components of 28,227 total.
Operator
Basis
Target property
Primary operators
Class
Corpus audit
Semantic identity
Offset
Corpus audit
Spatial alignment
Direction
Corpus audit
Trajectory direction
Shrink
Prior work
Coherent support
Diagnostic operators
Table 2: The six perturbation operators and their target properties. The primary set spans four annotation properties. Dropout and Fragmentation are diagnostic controls. We distinguish isolated from coupled operational control.
Figure 2: Six manual-VOC operations on image 2007_000256 ( Everingham et al., 2010 ) . The leftmost column shows the GT for visual reference; the remaining columns show generated labels at r=0.9,0.5,0.1 . In the class row, relabeling denotes replacement of a selected whole component’s class. Per-operator generation success and achieved-value distributions are reported in the supplementary material
Figure 3: Requested and achieved operator scores and image-level target attainment. Each panel is one dataset–operator pair. The left axis shows the requested score and achieved mean; the right axis shows the percentage of images with absolute score error at most 0.05 . The clean/identity reference at r=1.0 is shown for orientation. Its perfect attainment is expected by construction and it is excluded from perturbed-level summaries. Scores remain operator-specific
Dataset
Operator
Images
Mean level abs. error
Mean within-0.05 (%)
r=.1 achieved
r=.1 within-0.05 (%)
VOC2012 (manual)
Class
12,031
0.039
52.1
0.073
54.0
VOC2012 (manual)
Offset
12,031
0.052
73.8
0.203
46.2
VOC2012 (manual)
Direction
12,031
0.007
89.9
0.062
92.9
VOC2012 (manual)
Shrink
12,031
0.013
89.9
0.130
87.6
S4Pascal (auto)
Class
12,031
0.030
54.2
0.067
55.0
S4Pascal (auto)
Offset
12,031
0.050
72.0
0.197
39.4
Table 3: Generator target reachability over the nine perturbed levels r=0.1 – 0.9 . Mean level absolute error is the mean across the nine conditions of the absolute difference between achieved and requested mean scores; mean target attainment is the mean image-level fraction with absolute score error no greater than 0.05. The r=1.0 identity/clean reference is excluded. Scores are operator-specific and are not physical translations or angles.
Figure 4: Cross-property score ratios in each target-operator output over r=0.1 – 0.9 . Each column identifies the target operation and each row a dataset. Only non-target properties are plotted; the dashed horizontal line marks ratio one. The vertical axis is restricted to the observed near-neutral range to magnify small deviations. Distances from the reference should therefore be interpreted using the numeric values, not as a full-scale visual effect size. Score ratios use the report-specific fixed-binding or native score definitions described in the text
Dataset
Target operator
Changed pixels (%)
Changed/support (%)
Rclass
Roffset
Rdirection
Original-position overlap
VOC2012 (manual) ( n=12,031 )
Class
2.90
92.43
0.076
1.000
1.000
1.000
VOC2012 (manual) ( n=12,031 )
Offset
4.16
132.20
0.897
0.283
1.001
0.337
VOC2012 (manual) ( n=12,031 )
Direction
2.20
66.97
0.995
0.908
0.095
0.654
VOC2012 (manual) ( n=12,031 )
Shrink
2.74
87.07
0.998
0.768
1.067
0.130
S4Pascal (auto) ( n=12,031 )
Class
2.41
93.28
0.067
1.000
1.000
1.000
S4Pascal (auto) ( n=12,031 )
Offset
2.75
106.59
0.960
0.304
0.995
0.465
Table 4: Cross-property measurements on the r=0.1 target-operator outputs. Changed pixels use the full image area as denominator, and changed/support uses the original annotated support area as denominator. The latter can exceed 100% because edits may affect pixels outside original support. Rclass is same-position label retention, Roffset is the report-defined Offset score ratio (except Cityscapes Class, which uses the fixed-binding geometric control), and Rdirection is the report-defined fixed-binding ratio for side effects or the generator-native target ratio for Direction outputs. Values are computed per image and averaged equally over images with valid denominators. The final column reports original-position support overlap, ∣U∩V∣/∣U∣ , where U and V denote the input and output labeled support. This measures spatial overlap with the input support, not total output support area; translation can reduce it without deleting support. Ratios above one are retained and are not probabilities. Direction side-effect scores use fixed original component–GT binding with pixel-weighted component aggregation for S4Pascal. These score ratios are not interchangeable physical units.
Figure 5: Five-seed Manual-VOC TEL–ResNet101 response curves for the four retained operations. Lower r denotes a stronger within-operator intervention. Shaded bands show ±1 standard deviation across five seeds
Figure 6: TEL–ResNet101 cross-dataset response paths. Each panel shows clean-normalized deviation C(r) . VOC2012 shows the five-seed mean path, while S4Pascal and S4Cityscapes show descriptive single-run full paths. Lower r denotes stronger perturbation within each operator
VOC2012
S4Pascal
S4Cityscapes
Operation
R Cavg/Cend
S Cavg/Cend
R Cavg/Cend
S Cavg/Cend
R Cavg/Cend
S Cavg/Cend
Class
0.631 / 0.979
0.632 / 0.967
0.636 / 0.976
0.643 / 0.979
0.555 / 0.976
0.597 / 0.980
Offset
0.328 / 0.493
0.378 / 0.513
0.239 / 0.374
0.268 / 0.383
0.215 / 0.603
0.204 / 0.585
Direction
0.209 / 0.255
0.280 / 0.328
0.159 / 0.190
0.178 / 0.214
0.032 / 0.063
0.023 / 0.050
Shrink
0.031 / 0.098
0.034 / 0.041
0.055 / 0.116
0.034 / 0.056
0.091 / 0.215
0.042 / 0.109
Table 5: Descriptive TEL response across datasets and backbones. R and S denote ResNet-101 and SegFormer-B4. Cavg/Cend are the average clean-normalized response over r∈{0.9,…,0.1} and the response at r=0.1 . Higher values indicate larger deviation from the clean reference. The table provides numerical summaries for Figure 6 . VOC2012–ResNet101 uses five-seed mean paths. The remaining full-path extensions are descriptive single-run summaries rather than statistical backbone comparisons.
Figure 7: Pipeline sensitivity across matched TEL and DTS settings, measured by Cavg over the nine perturbed ratios. TEL–ResNet101 on VOC2012 uses the five-seed mean curve. All other settings, including DTS, use single-run full-ratio sweeps. Darker cells indicate greater clean-normalized response
Pipeline
Backbone
Dataset
Noise
n
Validation mIoU (mean ± sample SD)
r=1.0
r=0.9
r=0.5
r=0.1
TEL
ResNet101
VOC2012
Class
5
70.08±0.42
65.04±0.61
7.96±0.25
1.46±0.12
Offset
5
70.08±0.42
64.03±0.39
46.03±0.68
35.57±0.98
Direction
5
70.08±0.42
59.66±0.59
54.94±0.52
52.20±0.44
Shrink
5
70.08±0.42
69.83±0.27
68.77±0.82
63.24±1.25
TEL
ResNet101
S4Pascal
Class
4
75.76±0.17
69.43±0.19
13.07±0.54
1.46±0.10
Table 6: Repeated-seed results for the three settings with four reported anchor ratios (percentages; mean ± sample SD). Pipeline, backbone, and dataset are separated explicitly. TEL entries use the tail- 25% validation average, whereas DTS entries use the best validation checkpoint under its logged protocol. Repeated r=1.0 values are the shared clean reference within each setting.
Pipeline
Backbone
Dataset
Noise
n
Validation mIoU (mean ± sample SD)
r=1.0
r=0.1
PFA
MiT-B1
VOC2012
Class
3
78.32±0.10
4.46±0.06
Offset
3
78.32±0.10
39.94±0.05
Direction
3
78.32±0.10
58.83±0.03
Shrink
3
78.32±0.10
73.25±0.37
TEL
SegFormer-B4
VOC2012
Class
3
73.75±0.12
3.87±0.12
Table 7: Repeated-seed anchor-only results for the two settings evaluated at clean and strong noise (percentages; mean ± sample SD). TEL entries use the tail- 25% validation average, whereas PFA entries use the best validation checkpoint. Only clean and endpoint conditions were evaluated; intermediate response shapes are therefore not assessed.
School of Data Science, Fudan University, Shanghai, 200433, China · Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, USA