Progressively Refining SAM for Accurate Phytoplankton Segmentation

Jialin Li; Jinkun Jiang; Yuezun Li; Junyu Dong

cs.CVOct 27, 2026

Progressively Refining SAM for Accurate Phytoplankton Segmentation

Authors: Jialin Li, Jinkun Jiang, Yuezun Li, Junyu Dong

Organizations: School of Computer Science and Technology, Ocean University of China, Qingdao, China

Abstract

Accurate phytoplankton segmentation is essential for ecological monitoring, yet it remains largely underexplored due to the subtle visual characteristics of phytoplankton and the scarcity of pixel-level annotations. To address these challenges, we propose PhySeg, a SAM-based segmentation framework equipped with a lightweight refinement module that progressively restores fine-grained details through multi-scale feature fusion. In addition, we construct a phytoplankton segmentation dataset by manually annotating instance-level masks for an existing dataset, providing a valuable resource for future research. Experimental results demonstrate that PhySeg improves segmentation accuracy, validating the effectiveness of the proposed module. Code is available at https://github.com/lijialin404/PhySeg.

Figures & tables

Figure 1: Detailed structure of the proposed PhySeg.

Figure 2: Overview of the Feature Fusion module. $\mathrm{CGG}_{k\times k}^{d}$ denotes a $k\times k$ convolution with dilation rate $d$ , followed by GroupNorm and GELU. The dilation rate is 1 when omitted.

Figure 3: Visual examples of the phytoplankton segmentation dataset.

ID	Genus	Images	Instances
1	bacteriastrum	677	703
2	biddulphia	356	360
3	ceratium	2267	2332
4	chaetoceros	1095	1184
5	coscinodiscus	1123	1140
6	dinophysis	413	417

Table 1: Category statistics of the dataset.

Method	Venue	Prompt	mIoU $\uparrow$	Dice $\uparrow$	BF1 $\uparrow$	BF2 $\uparrow$	BF4 $\uparrow$
CascadePSP ∗ [ 17 ]	CVPR’20	Mask	87.08	92.77	53.45	72.41	90.57
BPR ∗ [ 18 ]	CVPR’21	Mask	88.51	93.53	59.61	77.99	92.88
CRM ∗ [ 19 ]	CVPR’22	Mask	68.20	76.08	34.94	49.04	65.39
SAM2 ∗ [ 12 ]	ICLR’25	Box	81.03	87.82	42.86	60.29	80.46
HQ-SAM2 ∗ [ 20 ]	NeurIPS’23	Box	82.99	89.35	47.57	65.83	84.64
CascadePSP [ 17 ]	CVPR’20	Mask	89.72	94.37	62.69	79.85	93.36

Table 2: Quantitative performance comparison of different methods on the test set. “Box” indicates that a bounding box is used as the model prompt, whereas “Mask” indicates that the SAM2+LoRA prediction is provided as the initial mask.

Figure 4: Visual examples of our method and others: (a) original image, (b) ground truth, (c) SAM2, (d) SAM2+LoRA, (e) HQ-SAM2, and (f) PhySeg (Ours).

Method	mIoU $\uparrow$	Dice $\uparrow$	BF1 $\uparrow$	BF2 $\uparrow$	BF4 $\uparrow$
PhySeg	92.07	95.80	67.75	84.11	95.47
w/o $H$	91.97	95.74	67.92	84.35	95.61
w/o $S_{0}$	92.03	95.77	67.65	83.95	95.49
w/o Step 0	92.00	95.76	67.45	84.04	95.55
w/o Step 0 & Step 1	91.96	95.74	67.36	83.88	95.45

Table 3: Ablation study on the val-set.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

PlankFormer: Robust Plankton Instance Segmentation via MAE-Pretrained Vision Transformers and Pseudo Community Image Generation
Apr 20, 2026Masaharu Miyazaki, Yurie Otake, Koichi Ito +3Instance Segmentation Synthetic Data Generation
Graduate School of Information Sciences, Tohoku University 6-6-05, Aramaki Aza Aoba, Sendai, 9808579, Japan. · The Center for Ecological Research, Kyoto University, 2–509–3, Hirano, Otsu-shi, Shiga-ken, 5202113, Japan. · Graduate School of Life Sciences, Tohoku University, 6–3, Aramaki Aza Aoba, Aoba-ku, Sendai-shi, 9808578, Japan.
ZMIS-SAM: Segment Anything Model Enhanced with Wavelet Transform for Zooplankton Microscopy Image Instance Segmentation
Jul 30, 2026Dekun Yuan, Zhongwei Li, Zheng Qiao +1Instance Segmentation Segment Anything Models
College of Oceanography and Space Informatics, China University of Petroleum (East China), Qingdao, 266580, China
HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation
Mar 13, 2026Pingping Zhang, Tianyu Yan, Yuhao Wang +7Image Segmentation Promptable Image Segmentation
School of Future Technology, Dalian University of Technology. · Central Hospital of Dalian University of Technology. · Affiliated Zhongshan Hospital of Dalian University. +1

Apr 20, 2026J/K move · Enter open · S save

PlankFormer: Robust Plankton Instance Segmentation via MAE-Pretrained Vision Transformers and Pseudo Community Image Generation

Masaharu Miyazaki, Yurie Otake, Koichi Ito, Wataru Makino+2

Graduate School of Information Sciences, Tohoku University 6-6-05, Aramaki Aza Aoba, Sendai, 9808579, Japan. · The Center for Ecological Research, Kyoto University, 2–509–3, Hirano, Otsu-shi, Shiga-ken, 5202113, Japan. · Graduate School of Life Sciences, Tohoku University, 6–3, Aramaki Aza Aoba, Aoba-ku, Sendai-shi, 9808578, Japan.

Instance Segmentation Synthetic Data Generation Microscopy Image Analysis

Open paper AskPDF arXiv

Plankton monitoring is essential for assessing aquatic ecosystems but is limited by the labor-intensive nature of manual microscopic analysis. Automating the segmentation of plankton from crowded images is crucial, however, it faces two major challenges: (i) the scarcity of pixel-level annotated datasets and (ii) the difficulty of distinguishing plankton from debris and overlapping individuals using conventional CNN-based methods. To address these issues, we propose PlankFormer, a novel framework for plankton instance segmentation. First, to overcome the data shortage, we introduce a method to generate labeled Pseudo Community Images (PCI) by synthesizing individual plankton images onto diverse backgrounds, including those created by generative models. Second, we propose a segmentation model utilizing a Vision Transformer (ViT) backbone with a Mask2Former decoder. To robustly capture the global structural features of plankton against occlusion and debris, we employ a Masked Autoencoder (MAE) for self-supervised pre-training on unlabeled individual images. Experimental results on real-world datasets demonstrate that our method significantly outperforms conventional methods, such as Mask R-CNN, particularly in challenging environments with high debris density. We demonstrate that our synthetic training strategy and MAE-based architecture enable high-precision segmentation with requiring less manual annotations for individual plankton images.

Abstract

Instance Segmentation

Explore similar work

PlankFormer: Robust Plankton Instance Segmentation via MAE-Pretrained Vision Transformers and Pseudo Community Image Generation

ZMIS-SAM: Segment Anything Model Enhanced with Wavelet Transform for Zooplankton Microscopy Image Instance Segmentation

HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation