cs.CVOct 29, 2026

Single-Annotation, Multi-Hypothesis Segmentation with Semantic-Spatial Residual Banks

Authors: Armin Dadras, Robert Sablatnig, Franziska Proksa, Markus Seidl

Organizations: Computer Vision Lab, TU Wien, Vienna, Austria · UAS St. Pölten, St. Pölten, Austria

Abstract

Multi-rater segmentation datasets show that a single image can admit several valid masks, yet most training sets expose only one annotation per image. We study single-annotation multi-hypothesis segmentation: each training image exposes one fixed mask, while the model must return a finite bank that covers held-out human annotations at test time. Our experiments reveal a representation tradeoff. Banks learned directly from images recover variations in local boundaries and extent but lose structural hypotheses, whereas frozen semantic features preserve structural grouping but underrepresent spatial variation. We introduce the Semantic-Spatial Residual Bank (SSRB), which freezes a semantic bank trained with balanced assignments and learns a residual from the image for each hypothesis. Across five medical endpoints, SSRB improves coverage of held-out masks at K=8K=8 over a Probabilistic U-Net under the same one-mask constraint. On LIDC and LIDC-hard, it also produces substantially more distinct useful masks. On BSDS, it preserves the semantic bank's performance in grouping regions. Its advantage grows as more masks are returned, showing that residuals learned for corresponding heads improve coverage of local annotation variation while preserving how regions are grouped.

Figures & tables

Explore similar work

CardsList
  1. Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation

    Aug 3, 2026Jiazhen Liu, Mingkuan Feng, Long ChenInstance SegmentationImage Segmentation

  2. SemiSAM-O1: How far can we push the boundary of annotation-efficient medical image segmentation?

    Apr 27, 2026Yichi Zhang, Le Xue, Bichun Xu +6Image SegmentationMedical Imaging Foundation Models