Sfumato Supervision: Learning Instance Segmentation with Noisy Labels
Organizations: Technion · Ben-Gurion University of the Negev
Abstract
Instance segmentation requires precise object masks, yet both human annotators and promptable models can produce unreliable contours. A high prediction loss near a contour does not necessarily indicate an annotation error: valid edges can also be uncertain. Inspired by Leonardo da Vinci's sfumato, which softens transitions between a subject and its surroundings, we introduce CB-LNL (Copula-Boundary Learning with Noisy Labels). Rather than judging each contour pixel alone, CB-LNL combines foreground evidence inside the mask with background evidence outside it to guide local supervision and annotation repair. We ground this approach in an audit of paired COCO and LVIS annotations, and introduce benchmarks covering human annotations, weak-prompt SAM2 masks, and controlled corruptions in natural and driving scenes. Controlled comparisons show improved training on procedural contour noise and fewer harmful edits than direct teacher-mask replacement. Paired-seed and frozen-prediction evaluations reveal how these benefits depend on the annotation source, supporting separate assessment of category and contour reliability.
Figures & tables
| Study | Task | Audit | Contour | Scale | Drift | Omit | WAN | Clean |
|---|---|---|---|---|---|---|---|---|
| Yang et al . [ 24 ] | Inst. | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Liu et al . [ 21 ] | Sem. | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| Yao et al . [ 36 ] | Sem. | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| Cityscapes-C [ 3 ] | Sem/Inst | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| SAM SA-1B [ 13 ] | Prompt | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| Ours | Inst. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| ResNet-50 | ConvNeXt-Tiny | Swin-Tiny | |||||
| Dataset | Supervision | Mask AP | Bnd. AP | Mask AP | Bnd. AP | Mask AP | Bnd. AP |
| COCO-RealNoise | Reference | 30.89 (28.31) | 17.56 (16.41) | 31.26 (28.76) | 17.85 (16.58) | 31.34 (28.71) | 17.86 (16.57) |
| Original COCO | 30.45 (29.83) | 17.05 (16.82) | 30.69 (30.22) | 17.00 (16.90) | 30.94 (30.59) | 17.16 (17.07) | |
| COCO-WAN | Tight box | 30.87 | 18.21 | 31.71 | 18.51 | 31.69 | 18.76 |
| Perturbed box | 28.63 | 16.57 | 28.98 | 16.98 | 29.17 | 17.05 | |
| Center point | 26.21 | 15.04 | 26.76 | 15.56 | 26.47 | 15.32 | |
| Isolated Operator | |||||||
|---|---|---|---|---|---|---|---|
| Clean Reference | 30.10 | 17.10 | 31.86 | 16.38 | 35.18 | 42.18 | 19.00 |
| Polygon Simplification | 27.07 | 14.03 | 30.15 | 11.45 | 33.51 | 41.66 | 18.95 |
| Rigid Spatial Drift | 25.14 | 11.43 | 28.83 | 8.24 | 31.72 | 40.56 | 18.31 |
| Vertex Jitter | 29.67 | 16.36 | 27.91 | 15.26 | 34.62 | 41.13 | 18.66 |
| Morphological Erosion | 20.85 | 8.59 | 25.17 | 5.73 | 24.94 | 38.47 | 20.52 |
| Morphological Dilation | 22.88 | 8.63 | 28.34 | 6.24 | 26.47 | 39.31 | 20.05 |
| Method | RealNoise | WAN-Box | WAN-Point | Easy | Medium | Hard |
|---|---|---|---|---|---|---|
| Mask AP / Boundary AP (single seed) | ||||||
| Standard | 30.45 / 17.05 | 28.63 / 16.57 | 26.21 / 15.04 | 29.09 / 16.29 | 26.70 / 14.34 | 24.09 / 10.48 |
| SCE [ 33 ] | 29.52 / 15.92 | 27.17 / 15.10 | 25.30 / 14.41 | 28.91 / 16.14 | 26.22 / 13.73 | 23.16 / 10.09 |
| GCE [ 37 ] | 30.01 / 16.40 | 28.53 / 16.29 | 25.94 / 15.26 | 29.68 / 16.86 | 27.21 / 14.34 | 23.07 / 10.13 |
| DivideMix-Cls [ 15 ] | 27.06 / 15.09 | 25.10 / 14.17 | 22.75 / 13.38 | 27.24 / 15.15 | 25.25 / 13.02 | 21.92 / 9.86 |
| DivideMix-Mask | 28.32 / 15.33 | 27.44 / 15.53 | 25.78 / 14.99 | 29.40 / 16.57 | 26.01 / 13.92 | 23.39 / 10.39 |
| Method | Mask AP | Boundary AP |
|---|---|---|
| COCO-RealNoise | ||
| Standard | ||
| Boundary-LNL | ||
| CB-LNL | ||
| COCO-N Medium | ||
| Standard | ||
| Frozen Scoring Rule (Same Checkpoint) | COCO-N Med ( ) | RealNoise ( ) | Edge FPR | ||
|---|---|---|---|---|---|
| AUROC | PR-AUC | AUROC | PR-AUC | Med / Real | |
| Whole-Mask BCE ( ) | 0.7143 | 0.7625 | 0.6412 | 0.6745 | 36.7% / 71.2% |
| Boundary-Band BCE ( on ) | 0.7915 | 0.8518 | 0.6290 | 0.6621 | 28.5% / 69.3% |
| Linear Contrast ( ) | 0.6856 | 0.7287 | 0.6334 | 0.6616 | 40.0% / 55.8% |
| CB-LNL Fréchet Copula (Ours) | 0.7880 | 0.8511 | 0.6469 | 0.6776 | 23.3% / 61.9% |
| Training Ablation (COCO-N Med) | Copula | ||||
| Dataset | Supervision / Repair Method | Beneficial / Harmful % | |||||||
|---|---|---|---|---|---|---|---|---|---|
| COCO-RealNoise | Original + Standard | – | 0.811 0.956 | 70.8% (88.2%) | 30.45 | 17.05 | 32.03 | 15.61 / 43.21 | 19.12 |
| Direct Teacher Replacement + Standard | 50.7% / 44.8% | (Frozen 0.629 0.647 ) | 70.8% 68.7% | 30.31 | 16.61 | 31.82 | 15.94 / 43.73 | 19.70 | |
| CB-LNL Cleaned + Standard | 51.4% / 38.6% | 70.8% 71.5% ( 90.1% ) | 30.52 | 16.81 | 31.49 | 16.48 / 43.83 | 18.87 | ||
| WAN-Box (Noisy) | Original / CB-LNL Cleaned + Std | 81.2% / 14.1% | 0.767 0.909 | 70.6% 76.5% | 28.63 29.27 | 16.57 / 16.25 | 31.12 31.20 | 16.04 / 41.94 | 16.10 |
| COCO-N Medium | Original + Standard | – | 0.815 0.925 | 64.4% (68.3%) | 26.70 | 14.34 | 28.46 | 13.39 / 39.11 | 15.70 |
| Direct Teacher Replacement + Standard | 74.6% / 24.1% | (Frozen 0.792 / 0.788) | 64.4% 71.6% | 27.17 | 14.42 | 27.78 | 13.67 / 40.16 | 16.27 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Supervision | Backbone | ||||||
| COCO-RealNoise | Clean Reference | ResNet-50 / ConvNeXt-T / Swin-T | 30.89 (28.31) / 31.26 (28.76) / 31.34 (28.71) | 17.56 (16.41) / 17.85 (16.58) / 17.86 (16.57) | 32.02 / 32.16 / 32.65 | 16.53 / 17.59 / 17.61 | 44.33 / 45.28 / 45.80 | 19.20 / 18.88 / 17.80 |
| Noisy COCO | ResNet-50 / ConvNeXt-T / Swin-T | 30.45 ( 29.83 ) / 30.69 ( 30.22 ) / 30.94 ( 30.59 ) | 17.05 ( 16.82 ) / 17.00 ( 16.90 ) / 17.16 ( 17.07 ) | 32.03 / 32.25 / 32.62 | 15.61 / 16.64 / 17.19 | 43.21 / 44.37 / 44.57 | 19.12 / 18.70 / 17.44 | |
| COCO-WAN | WAN-Box (Clean) | ResNet-50 / ConvNeXt-T / Swin-T | 30.87 / 31.71 / 31.69 | 18.21 / 18.51 / 18.76 | 32.00 / 32.89 / 32.72 | 16.84 / 16.92 / 17.16 | 44.62 / 45.82 / 45.55 | 18.92 / 18.94 / 17.54 |
| WAN-Box (Noisy) | ResNet-50 / ConvNeXt-T / Swin-T | 28.63 / 28.98 / 29.17 | 16.57 / 16.98 / 17.05 | 31.12 / 31.42 / 31.66 | 15.82 / 16.17 / 16.33 | 41.45 / 42.73 / 42.63 | 16.10 / 16.05 / 14.87 | |
| WAN-Point (Center) | ResNet-50 / ConvNeXt-T / Swin-T | 26.21 / 26.76 / 26.47 | 15.04 / 15.56 / 15.32 | 26.64 / 27.14 / 26.82 | 14.53 / 15.00 / 15.04 | 37.98 / 38.94 / 38.49 | 19.22 / 19.15 / 18.14 | |
| WAN-Point (Random) | ResNet-50 / ConvNeXt-T / Swin-T | 25.00 / 25.43 / 25.56 | 14.23 / 14.57 / 14.79 | 25.63 / 25.99 / 26.24 | 13.79 / 14.12 / 14.26 | 36.68 / 37.24 / 37.54 | 18.88 / 18.79 / 17.85 |
| Dataset Group | Supervision Regime | Mean IoU | Med. IoU | Mean BIoU | Med. BIoU | % | / | Class Flip / Omission |
|---|---|---|---|---|---|---|---|---|
| Real Human | COCO-RealNoise (MS-COCO Crowdsourced) | 0.823 | 0.853 | 0.730 | 0.761 | 47.0% | 0.768 / 0.632 | 3.1% (88.9% super) / 24.6% |
| COCO-WAN | WAN-Box (Clean) (SAM2 + Clean GT Box) | 0.796 | 0.843 | 0.741 | 0.786 | 41.0% | 0.735 / 0.657 | 0.0% / 0.0% |
| COCO-WAN | WAN-Box (Noisy) (SAM2 + Noisy Detector Box) | 0.754 | 0.810 | 0.698 | 0.749 | 50.2% | 0.691 / 0.600 | 8.0% (super) / 0.0% |
| COCO-WAN | WAN-Point (Center) (SAM2 + Center Click) | 0.617 | 0.712 | 0.570 | 0.651 | 62.4% | 0.581 / 0.430 | 0.0% / 0.0% |
| COCO-WAN | WAN-Point (Random) (SAM2 + Random Click) | 0.585 | 0.682 | 0.546 | 0.623 | 64.1% | 0.563 / 0.410 | 0.0% / 10.0% |
| COCO-N | COCO-N Easy (Calibrated to Human Noise) | 0.839 | – | 0.808 | – | – | Corrupted BIoU: 0.728 | 5.0% / 5.0% |
| Study Group | Ablation Variant | Copula | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Baseline | Standard (CE + BCE, Seed 1) | – | – | 26.70 | 14.34 | 28.46 | 13.39 | 39.11 | 15.70 |
| Baseline | Boundary-LNL (No Copula, Seed 1) | – | – | 28.30 | 15.48 | 29.74 | 15.24 | 41.61 | 15.87 |
| Part A: Evidence | Isotropic Band Neighbors ( Only) | 23.4% 29.3% | 27.66 | 14.85 | 29.36 | 14.88 | 40.61 | 15.96 | |
| Part A: Evidence | Interior Core Only ( ) | 23.6% 31.9% | 28.10 | 15.44 | 29.79 | 15.52 | 41.31 | 15.72 | |
| Part A: Evidence | Inverse-Exterior Only ( ) | 23.5% 31.9% | 27.46 | 14.60 | 29.27 | 14.32 | 39.33 | 15.95 | |
| Part A: Evidence | Directional Spatial ( ) | 23.4% 30.1% | 27.69 | 15.28 | 29.69 | 15.44 | 40.12 | 16.03 |
| Evaluated Split | Annotation Source | Audited Inst. | Inward Cut % | Outward Bleed % | Class Flip % | Prec@100 | ||
|---|---|---|---|---|---|---|---|---|
| MS-COCO val2017 | Official Validation Benchmark | 7,999 (3,042 LVIS) | 3.26% | 5.15% | 0.83% | 0.8032 0.9604 | 100.0% | 71.1% 72.0% (88.6% 90.5% ) |
| MS-COCO train2017 | Official Crowdsourced Training | 8,090 (3,088 LVIS) | 2.72% | 5.59% | 0.80% | 0.8108 0.9564 | 100.0% | 70.8% 71.5% (88.3% 90.1% ) |
| WAN-Box (Noisy) | SAM2 + Jittered Detector Boxes | 8,903 | 4.93% | 22.94% | 9.15% | 0.7674 0.9087 | 100.0% | 70.59% 76.46% (3,296 fixed) |
| WAN-Point (Center) | SAM2 + Single Center Click (Unseen) | 8,903 | 20.59% | 13.47% | 3.74% | 0.8398 0.9467 | 100.0% | 65.62% 72.09% (3,860 fixed) |
| COCO-N Medium | Multi-Factor Procedural Stress | 8,167 | 18.08% | 16.59% | 7.48% | 0.8146 0.9251 | 100.0% | 64.4% 72.4% (Multi: 77.7% ) |
| Regime Type | Error Mode & Geometric Condition | Governing Copula Mechanism | Detection AUROC | Repair Success Rate | Gain | Primary Bottleneck / Behaviour |
|---|---|---|---|---|---|---|
| Positive 1 | Inward Polygon Shortcut (Concavities / Limbs) | Foreground Continuation | 0.962 | 91.8% | Both sides of cut lie in foreground ( ) | |
| Positive 2 | Outward Box Bleed (Loose SAM2 / Dilated Contour) | Inverse-Background | 0.941 | 88.5% | Both sides of bleed lie in background ( ) | |
| Positive 3 | Within-Supercategory Semantic Flip ( ) | Class–Spatial Copula | 0.985 | 91.4% | True class verified by inner mask core | |
| Failure Mode 1 | Extreme Part Truncation ( Missing, WAN-Point ) | True contour lies far outside band ( ) | 0.812 | 24.6% | Local band cannot bridge large missing parts | |
| Failure Mode 2 | Crowded Same-Class Merge (Overlapping Instances) | Shared semantic class on occlusion border | 0.618 | 14.2% | Class-specific mask head predicts on both | |
| Failure Mode 3 | Sub-RoI Thin Structures ( in RoI) | Eroded inner core vanishes ( ) | 0.684 | 19.5% | Quantization removes inner anchor ; defaults to no-op |
Instance Segmentation
Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.
See all 176 papers