Leveraging existing sparse point annotations for benthic imagery dense segmentation

Cesar Borja; Breck A. McCollum; Jarret E. Byrnes; Kenneth Sebens; Ana C. Murillo

cs.CVAug 18, 2026

Leveraging existing sparse point annotations for benthic imagery dense segmentation

Authors: Cesar Borja, Breck A. McCollum, Jarret E. Byrnes, Kenneth Sebens, Ana C. Murillo

Organizations: DIIS - I3A, Universidad de Zaragoza, Zaragoza, Spain · Berklee College of Music, Boston, USA · University of Massachusetts: Boston, Boston, USA · University of Washington, Seattle, USA

Abstract

The health of marine ecosystems is a critical indicator of global environmental change, yet the physical constraints of underwater observation and the intrinsic challenges of processing marine imagery severely limit the scalability of systematic monitoring. While recent visual foundation models such as the Segment Anything Model (SAM) series show great promise, they still struggle with the fine-grained recognition required in these complex scenarios and still require expert supervision. Our work addresses this gap by bridging state-of-the-art foundation models with existing sparse supervision. Because historical benthic surveys are typically annotated with only a few sparse expert points per image, we utilize these legacy point-labels as visual prompts for SAM2. Our primary contribution is a novel mechanism to automatically identify which of these points are suitable, and which are actively harmful, when used for propagation. By filtering out unreliable points, we extract high-quality pseudo-ground-truth masks capable of training more accurate, fine-grained semantic segmentation models. We demonstrate the effectiveness of our approach on public benthic data and introduce a new, challenging benchmark featuring real-world sparse expert annotations, paving the way for scalable ecological analysis.

Figures & tables

Panel of Figure 1 — Figure 1: Expanding sparse expert point-labels into dense masks is a common strategy to enable automated benthic image analysis. Since sparse point-label locations were usually placed random or uniformly, they often fall on ambiguous locations. Our framework automatically identifies and filters unsuitable points to significantly improve downstream segmentation performance. a) Ideal segmentation mask for class droebachiensis with two point-labels of the same class. b) Unsuitable point-label whose propagation bleeds across class boundaries, introducing severe label noise. c) Suitable point-label that propagates correctly.

Panel of Figure 2 — Figure 2: Visual comparison of the benthic imagery datasets: (a) Ours, (b) CoralScapes [ 21 ] , and (c) UCSD Mosaics [ 10 , 1 ] .

Figure 3: Overview of our pipeline. Given an image with sparse point-labels, we propagate (1) each point-label $i$ with SAM2. Using DINOv3 features, we calculate per-image class prototypes $\mu_{c}$ , and with them, we prune (2) the point-labels whose mask leak into other classes ( $i_{2}$ ), and trim (3) the patches that disagree with their mask’s class ( $p1$ , $p4$ and $p5$ ). The surviving masks are merged (4) into the final dense labels by ground-truth voting.

	Our dataset		Coralscapes [ 21 ]			UCSD Mosaics [ 10 , 1 ]
Strategy	mPA ∗	used%	mPA	mIoU	used%	mPA	mIoU	used%
None ( keep all )	$27.53$	100	33.71	25.57	100	53.75	44.41	100
GCE [ 35 ]	$27.86$	100	33.54	25.52	100	53.87	41.66	100
Blind random drop †	$22.60$	66.1	33.76	25.39	75.4	54.77	45.12	88.3
Class-balanced random †	$23.46$	66.1	33.95	25.79	75.4	52.11	42.73	88.3
Ours	$\mathbf{32.00}$	66.1	37.21	27.60	75.4	53.60	44.05	88.3

Table 1: Filtering propagated labels by reliability across three datasets. All strategies propagate the same expert point-labels into dense pseudo-masks with SAM2 and train Segformer on them. Keep all trusts every propagated label; GCE also keeps all labels but trains with a noise-robust loss; the random baselines discard labels without any reliability estimate; Ours prunes unreliable points before propagation and trims unreliable regions afterwards. Our benthic dataset has no dense annotation and is evaluated only at its expert-labeled points. Mean over 5 seeds for our benthic dataset, a single seed for Coralscapes and UCSD Mosaics.

Method	mPA
keep all	$27.53\pm 1.0$
trim only	$29.82\pm 0.9$
prune only	$31.00\pm 0.7$
prune + trim (ours)	$\mathbf{32.00\pm 1.1}$

Table 2: Pruning and trimming ablation on our benthic dataset.

Panel of Figure 4 — Figure 4: Qualitative examples on our benthic dataset. Each row shows, (a) the input image with the available the point-labels overlaid as class-colored dots (with yellow edge for visibility); (b) the masks obtained after propagating all points; (c) the masks obtained with our approach. (a) and (b) show all the point-labels while (c) only shows the points that survive our pruning . All examples show situations where expert correct point-labels get incorrectly covered by other class propagation. Red dashed boxes and arrows highlight some of the interesting cases. Pruning discards points whose expansion is unreliable and trimming removes mislabeled regions. In some cases removing an over-extended mask is key to enable neighbouring mask supply the correct label, recovering pixels that keep-all had mislabeled (e.g., the example in row 1).

Panel of Figure 5 — Figure 5: Qualitative examples on Coralscapes. Each row shows, (a) the input image with the available the point-labels overlaid as class-colored dots (with yellow edge for visibility); (b) the Ground truth; (c) the masks obtained after propagating all points; (d) the masks obtained with our approach. (a) and (c) show all the point-labels while (d) only shows the points that survive our pruning . These examples show two situations our method corrects: expert-correct point-labels whose SAM2 mask bleeds into a neighbouring class, and correct labels whose mask over-expands far beyond the object. Pruning discards the point when its whole expansion is unreliable, and trimming removes only the regions of a surviving mask that disagree with its class. As a result, ours often leaves a region unlabelled rather than keep the confidently-wrong, over-extended mask that keep-all propagates.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 399 papers

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity
Nov 17, 2025Junwei Yu, Trevor Darrell, XuDong WangImage Segmentation Promptable Image Segmentation
UC Berkeley
HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation
Mar 13, 2026Pingping Zhang, Tianyu Yan, Yuhao Wang +7Image Segmentation Promptable Image Segmentation
School of Future Technology, Dalian University of Technology. · Central Hospital of Dalian University of Technology. · Affiliated Zhongshan Hospital of Dalian University. +1
Exploring the Underwater World Segmentation without Extra Training
Nov 11, 2025Bingyu Li, Tao Huo, Da Zhang +3Image Segmentation Open-Vocabulary Semantic Segmentation
Institute of Artificial Intelligence (TeleAI), China Telecom, China · University of Science and Technology of China, China · Northwestern Polytechnical University, China

Nov 17, 2025J/K move · Enter open · S save

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

Junwei Yu, Trevor Darrell, XuDong Wang

UC Berkeley

Image Segmentation Promptable Image Segmentation Unsupervised Image Segmentation Video Object Segmentation

Open paper AskPDF arXiv

The Segment Anything Model (SAM) family has become a widely adopted vision foundation model, but its ability to control segmentation granularity remains limited. Users often need to refine results manually - by adding more prompts or selecting from pre-generated masks - to achieve the desired level of detail. This process can be ambiguous, as the same prompt may correspond to several plausible masks, and collecting dense annotations across all granularities is prohibitively expensive, making supervised solutions infeasible. To address this limitation, we introduce UnSAMv2, which can segment anything at any granularity without using any human annotations. UnSAMv2 extends the divide-and-conquer strategy of UnSAM by discovering abundant mask-granularity pairs and introducing a novel granularity control embedding that enables precise, continuous control over segmentation scale. Remarkably, with only

6

K unlabeled images and

0.02\%

additional parameters, UnSAMv2 substantially enhances SAM-2, enabling segmentation of anything at any granularity across interactive, whole-image, and video segmentation tasks. Furthermore, we introduce a language-guided segmentation pipeline that treats granularity as a predictable variable by grounding it in text semantics via an external vision-language model, improving upon the previous state of the art in 1-IoU (40.4

\rightarrow

47.3). Across 11+ benchmarks, UnSAMv2 achieves consistent gains over SAM/SAM-2 baselines, improving

\text{NoC}_{90}

(5.69

\rightarrow

4.75), 1-IoU (58.0

\rightarrow

73.1), and

\text{AR}_{1000}

(49.6

\rightarrow

68.3) by a large margin. Our work demonstrates that a small amount of unlabeled data can deliver large, practical improvements to strong supervised baselines. Our code, dataset, and demo are available at https://github.com/yujunwei04/UnSAMv2.

Abstract

Image Segmentation

Explore similar work

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation

Exploring the Underwater World Segmentation without Extra Training