HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation

Pingping Zhang; Tianyu Yan; Yuhao Wang; Yang Liu; Tongdan Tang; Yili Ma; Long Lv; Feng Tian

cs.CVMar 13, 2026

HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation

Authors: Pingping Zhang, Tianyu Yan, Yuhao Wang, Yang Liu, Tongdan Tang, Yili Ma, Long Lv, Feng Tian,

Organizations: School of Future Technology, Dalian University of Technology. · Central Hospital of Dalian University of Technology. · Affiliated Zhongshan Hospital of Dalian University. · School of Information and Communication Engineering, Dalian University of Technology.

Abstract

Marine Animal Segmentation (MAS) aims at identifying and segmenting marine animals from complex marine environments. Most of previous deep learning-based MAS methods struggle with the long-distance modeling issue. Recently, Segment Anything Model (SAM) has gained popularity in general image segmentation. However, it lacks of perceiving fine-grained details and frequency information. To this end, we propose a novel learning framework, named Hierarchical Frequency Prompted SAM (HFP-SAM) for high-performance MAS. First, we design a Frequency Guided Adapter (FGA) to efficiently inject marine scene information into the frozen SAM backbone through frequency domain prior masks. Additionally, we introduce a Frequency-aware Point Selection (FPS) to generate highlighted regions through frequency analysis. These regions are combined with the coarse predictions of SAM to generate point prompts and integrate into SAM's decoder for fine predictions. Finally, to obtain comprehensive segmentation masks, we introduce a Full-View Mamba (FVM) to efficiently extract spatial and channel contextual information with linear computational complexity. Extensive experiments on four public datasets demonstrate the superior performance of our approach. The source code is publicly available at https://github.com/Drchip61/TIP-HFP-SAM.

Figures & tables

Panel of Figure 1 — Fig. 1: Our motivations and advantages. The first row shows the input image, ground truth, the prediction mask obtained by point prompt and box prompt, respectively. The second row shows the component maps obtained from the wavelet transform and the result of our combined frequency map. The third row displays the existing point prompt methods and our proposed method.

Panel of Figure 2 — Fig. 2: The framework of our proposed HFP-SAM. It consists of three main components: Frequency Guided Adapter (FGA), Frequency-aware Point Selection (FPS), and Full View Mamba (FVM). HFP-SAM leverages frequency domain information to automatically obtain efficient prompts. Additionally, HFP-SAM utilizes the FVM to fully combine frequency and spatial domain information.

Panel of Figure 3 — Fig. 3: Illustration of the Frequency Guided Adapter.

Panel of Figure 4 — Fig. 4: Illustration of the Full View Mamba.

		MAS3K					RMAS
Method	Backbone	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{mE}_{\phi}$	MAE	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{mE}_{\phi}$	MAE
UNet++ [ 49 ]	–	0.506	0.726	0.552	0.790	0.083	0.558	0.763	0.644	0.835	0.046
BASNet [ 50 ]	ResNet-34	0.677	0.826	0.724	0.862	0.046	0.707	0.847	0.771	0.907	0.032
PFANet [ 51 ]	VGG-16	0.405	0.690	0.471	0.768	0.086	0.556	0.767	0.582	0.810	0.051
SCRN [ 52 ]	ResNet-50	0.693	0.839	0.730	0.869	0.041	0.695	0.842	0.731	0.878	0.030
U2Net [ 53 ]	–	0.654	0.812	0.711	0.851	0.047	0.676	0.830	0.762	0.904	0.029

TABLE I: Performance comparisons on MAS3K and RMAS. Each group represents methods based on different architectures.

		UFO120					RUWI
Method	Backbone	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{mE}_{\phi}$	MAE	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{mE}_{\phi}$	MAE
UNet++ [ 49 ]	–	0.412	0.459	0.433	0.451	0.409	0.586	0.714	0.678	0.790	0.145
BASNet [ 50 ]	ResNet-34	0.710	0.809	0.793	0.865	0.097	0.841	0.871	0.895	0.922	0.056
PFANet [ 51 ]	VGG-16	0.677	0.752	0.723	0.815	0.129	0.773	0.765	0.811	0.867	0.096
SCRN [ 52 ]	ResNet-50	0.678	0.783	0.760	0.839	0.106	0.830	0.847	0.883	0.925	0.059
U2Net [ 53 ]	–	0.680	0.792	0.709	0.811	0.134	0.841	0.873	0.861	0.786	0.074

TABLE II: Performance comparisons on UFO120 and RUWI. Each group represents methods based on different architectures.

Panel of Figure 7 — Fig. 5: Pairwise dataset distances measured by W1 and MMD-RBF.

Panel of Figure 8 — Fig. 6: Visual comparison of predicted segmentation masks with different methods. From left to right: (a) Input Images ; (b) Ground Truth ; (c) Ours ; (d) MASNet ; (e) SETR ; (f) TransUNet ; (g) SAM ; (h) MAS-SAM ; (i) Dual-SAM ; (j) SAM2 .

Panel of Figure 9 — Fig. 7: Visual comparison of predicted segmentation masks with different modules from Tab. III .

	Module					MAS3K
	Adapter	FGA	FPS	FVM	SL	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{m}\textbf{E}_{\phi}$	MAE
(A)	✕	✕	✕	✕	✕	0.566	0.763	0.656	0.807	0.059
(B)	✓	✕	✕	✕	✕	0.739	0.860	0.810	0.917	0.031
(C)	✓	✓	✕	✕	✕	0.754	0.865	0.818	0.923	0.030
(D)	✓	✓	✓	✕	✕	0.771	0.872	0.827	0.929	0.028
(E)	✓	✓	✓	✓	✕	0.792	0.881	0.839	0.932	0.026

TABLE III: Performance with different key modules on MAS3K.

Panel of Figure 11 — Fig. 8: Visual effect of using FGA. (a) Input Images; (b) Windows selected by FGA; (c) Images after applying the window masks.

Panel of Figure 12 — Fig. 9: (a) shows the regions with significant frequency domain changes captured by the FGA. (b) and (c) are the feature maps with the FGA and a standard adapter. (d) is the ground truth. Best view by zooming in.

Panel of Figure 13 — Fig. 10: Visual comparison of different amounts and sizes of sliding windows in FPS. The ablation settings from left to right: (a) Window=10, Size=16 ; (b) Window=10, Size=32 ; (c) Window=15, Size=32 ; (d) Window=10, Size=64 .

Window	Size	Point	mIoU	$\textbf{F}^{w}_{\beta}$	$\textbf{mE}_{\phi}$	Time (s)
5	32	2	0.767	0.825	0.927	0.0087
10			0.771	0.827	0.929	0.0093
20			0.772	0.827	0.930	0.0117
10	16	2	0.762	0.823	0.926	0.0090
	32		0.771	0.827	0.929	0.0093
	64		0.766	0.824	0.927	0.0106

TABLE IV: Performance with different settings in FPS.

	mIoU	$\textbf{F}^{w}_{\beta}$	$\textbf{mE}_{\phi}$	Time (s)
Random Sampling	0.760	0.823	0.925	0.0013
Global Sampling	0.764	0.824	0.926	0.3148
FPS	0.771	0.827	0.929	0.0093

TABLE V: Performance with different point prompt methods.

Method	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}^{w}_{\beta}$	$\textbf{mE}_{\phi}$	MAE
P only	0.789	0.879	0.839	0.937	0.026
N only	0.782	0.876	0.829	0.932	0.027
Both	0.797	0.888	0.845	0.938	0.024

TABLE VI: Performance with different point prompts in FPS.

$\tau$	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}^{w}_{\beta}$	$\textbf{mE}_{\phi}$	MAE
0.3	0.794	0.882	0.842	0.938	0.025
0.5	0.797	0.888	0.845	0.938	0.024
0.7	0.786	0.877	0.832	0.933	0.027

TABLE VII: Performance with different

\tau

values.

Panel of Figure 18 — Fig. 11: Visualization of the effective receptive field brought by FVM.

Panel of Figure 19 — Fig. 12: (a) and (b) are the segmentation results with and without using FVM against the ground truth, where green indicates correct predictions and red indicates redundant predictions. (c) is a heatmap showing the differences between the predictions in (a) and (b). Best view by zooming in.

Method	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}^{w}_{\beta}$	$\textbf{mE}_{\phi}$	MAE
Channel SSM	0.771	0.868	0.823	0.924	0.029
Spatial SSM	0.775	0.872	0.825	0.926	0.029
VSSM	0.778	0.872	0.826	0.926	0.029
FC-SSM	0.790	0.885	0.838	0.933	0.026
BC-SSM	0.783	0.874	0.828	0.932	0.028
FVM	0.797	0.888	0.845	0.938	0.024

TABLE VIII: Performance with different SSMs in FVM.

Prompt	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}^{w}_{\beta}$	$\textbf{mE}_{\phi}$	MAE
Point	0.767	0.870	0.825	0.926	0.028
Mask	0.759	0.866	0.822	0.925	0.029
Point+Mask	0.771	0.872	0.827	0.929	0.028

TABLE IX: Performance with different input prompts.

Fig. 13: Window selection in FPS. The image resolution increases from top to bottom. The window size decreases from left to right. Red rectangles show the selected top- $k$ windows with $k=10$ .

Panel of Figure 23 — Fig. 14: Resolution-aware window size selection with $k=10$ . Left: The mean foreground ratio. Right: Selected window sizes by matching the reference ratio from the default setting.

$\lambda$	mIoU	$\textbf{S}_{\alpha}$	$\textbf{F}^{w}_{\beta}$	$\textbf{mE}_{\phi}$	MAE
0.5	0.790	0.884	0.838	0.935	0.026
1	0.797	0.888	0.845	0.938	0.024
2	0.789	0.886	0.838	0.932	0.025

TABLE X: Effect of the loss weight on MAS3K.

	Gaussian		Speckle
$\sigma$	SAM	SAM+FGA	SAM	SAM+FGA
0.00	0.566	0.754	0.566	0.754
0.05	0.546	0.670	0.549	0.708
0.10	0.529	0.663	0.532	0.683
0.20	0.518	0.598	0.526	0.665

TABLE XI: Noise stress test on MAS3K. mIoU is reported.

Grid-level
Metric	FPS	FPS + gating ( $\gamma=0.8$ )
Grid-level error	9.01%	6.92%
Image-level: at least one failing grid	35.41%	29.01%
Image-level: failing grids $\geq 2$	20.86%	13.67%
Image-level: failing grids $\geq 50\%$	5.00%	4.38%
Point-level

TABLE XII: Error analysis of confidence-gated FPS.

Panel of Figure 27 — Fig. 15: Visual effect of our frequency domain priors. (a) Input Images; (b) The result by using canny edge detection; (c) The result by our frequency domain method. Best view by zooming in.

Variant	En	De1	FGA	FPS	De2	FVM	Overall	Peak Mem
SAM	21.04	15.23	0.00	0.00	0.00	0.00	36.38	618
SAM+FGA	27.06	15.00	0.54	0.00	0.00	0.00	42.71	649
SAM+FGA+FPS	27.06	15.13	0.45	1.18	15.65	0.00	59.77	654
HFP-SAM	27.06	15.25	0.44	1.19	15.80	0.15	59.92	654

TABLE XIII: Comparison of the latency and memory cost.

	COD10K [ 54 ]
Method	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{m}\textbf{E}_{\phi}$	MAE
SINet [ 54 ]	0.771	0.551	0.806	0.051
RankNet [ 56 ]	0.767	0.611	0.861	0.045
PFNet [ 55 ]	0.800	0.660	0.868	0.040
FBNet [ 76 ]	0.809	0.684	0.889	0.035
ECDNet [ 22 ]	0.783	0.701	0.798	0.050

TABLE XIV: Performance comparison on COD task.

	Kvasir [ 77 ]
Method	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{m}\textbf{E}_{\phi}$	MAE
SANet [ 79 ]	0.915	0.892	0.953	0.028
C2FNet [ 80 ]	0.905	0.870	0.889	0.035
LDNet [ 81 ]	0.905	0.869	0.945	0.031
FAPNet [ 82 ]	0.919	0.894	0.953	0.027
CFANet [ 83 ]	0.924	0.903	0.962	0.023

TABLE XV: Performance comparison on PS task.

	DUTS [ 78 ]
Method	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{m}\textbf{E}_{\phi}$	MAE
CANet [ 79 ]	0.878	0.838	0.889	0.044
ICON [ 80 ]	0.809	0.684	0.889	0.035
VST [ 81 ]	0.896	0.818	0.892	0.037
EDN [ 82 ]	0.883	0.845	0.911	0.041
MENet [ 83 ]	0.905	0.893	0.937	0.028

TABLE XVI: Performance comparison on SOD task.

	COD10K
Method	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{m}\textbf{E}_{\phi}$	MAE
SAM [ 11 ]	0.765	0.633	0.835	0.050
HFP-SAM	0.786	0.668	0.853	0.046
	Kvasir
Method	$\textbf{S}_{\alpha}$	$\textbf{F}_{\beta}^{w}$	$\textbf{m}\textbf{E}_{\phi}$	MAE
SAM [ 11 ]	0.356	0.246	0.339	0.515

TABLE XVII: Performance comparison of zero-shot capability.

Panel of Figure 33 — Fig. 16: Visualizations of failure cases and the corresponding point prompts obtained by our FPS. Best view by zooming in.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 399 papers

Jul 8, 2026

HPR-SAM: Hierarchical Probabilistic Representation Learning for Prompt-free SAM-based Medical Image Segmentation

Prompt-free adaptation of the Segment Anything Model (SAM) has emerged as a promising paradigm for automatic medical image segmentation. Existing methods mainly focus on prompt generation, while overlooking that prompt quality is fundamentally constrained by the expressiveness of anatomical representations. However, deterministic prototypes or semantic tokens are insufficient to jointly capture global anatomical priors, intra-structure diversity, and local structural reliability. To address this limitation, we propose the Hierarchical Probabilistic Representation (HPR) framework, which learns complementary anatomical representations through Distributional Anatomical Representation (DAR), Multi-component Anatomical Representation (MAR), and Local Reliability Representation (LRR), and integrates their predictions via Hierarchical Prediction Fusion (HPF) while remaining compatible with the original SAM decoder. Experiments on the Synapse, LA, and PROMISE12 datasets demonstrate that HPR-SAM achieves state-of-the-art performance on Synapse and the best performance under few-shot settings on LA and PROMISE12, validating the effectiveness of the proposed hierarchical probabilistic representation learning framework for prompt-free medical image segmentation. Code is available at https://anonymous.4open.science/r/HPR-SAM-E4AF.

Yingzhen Hu, Yiheng Zhong, Keying Zhu +5

Yale University, United States

Medical Imaging Foundation Models Representation Learning Hierarchical Representation Learning Medical Image Segmentation

Ask PDF

Nov 17, 2025

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

The Segment Anything Model (SAM) family has become a widely adopted vision foundation model, but its ability to control segmentation granularity remains limited. Users often need to refine results manually - by adding more prompts or selecting from pre-generated masks - to achieve the desired level of detail. This process can be ambiguous, as the same prompt may correspond to several plausible masks, and collecting dense annotations across all granularities is prohibitively expensive, making supervised solutions infeasible. To address this limitation, we introduce UnSAMv2, which can segment anything at any granularity without using any human annotations. UnSAMv2 extends the divide-and-conquer strategy of UnSAM by discovering abundant mask-granularity pairs and introducing a novel granularity control embedding that enables precise, continuous control over segmentation scale. Remarkably, with only

6

K unlabeled images and

0.02\%

additional parameters, UnSAMv2 substantially enhances SAM-2, enabling segmentation of anything at any granularity across interactive, whole-image, and video segmentation tasks. Furthermore, we introduce a language-guided segmentation pipeline that treats granularity as a predictable variable by grounding it in text semantics via an external vision-language model, improving upon the previous state of the art in 1-IoU (40.4

\rightarrow

47.3). Across 11+ benchmarks, UnSAMv2 achieves consistent gains over SAM/SAM-2 baselines, improving

\text{NoC}_{90}

(5.69

\rightarrow

4.75), 1-IoU (58.0

\rightarrow

73.1), and

\text{AR}_{1000}

(49.6

\rightarrow

68.3) by a large margin. Our work demonstrates that a small amount of unlabeled data can deliver large, practical improvements to strong supervised baselines. Our code, dataset, and demo are available at https://github.com/yujunwei04/UnSAMv2.

Junwei Yu, Trevor Darrell, XuDong Wang

UC Berkeley

Image Segmentation Promptable Image Segmentation Unsupervised Image Segmentation Video Object Segmentation

Ask PDF

Sep 21, 2025

SAM-DCE: Decoupled Context Enhancement for Prompt-Free Medical Image Segmentation

The Segment Anything Model (SAM) is a promising foundation model for medical image segmentation, but its prompt-driven design is not directly suitable for fully automatic multi-class anatomical segmentation. Without manual point, box, or mask prompts, SAM faces a dual over-smoothing problem: deep image features may lose fine local structural evidence, while mask tokens may become semantically ambiguous. We propose SAM-DCE, a decoupled context enhancement framework for prompt-free medical image segmentation. SAM-DCE introduces Spatial Representation Compensation (SRC) to recover anatomical evidence from intermediate ViT features and Semantic Token Calibration (STC) to improve the category consistency of foreground mask tokens. Rather than replacing the pretrained decoder or fusing output masks, SAM-DCE refines the spatial and semantic representations before recombining them through SAM's original token-to-mask mechanism. On the Synapse multi-organ CT segmentation dataset, SAM-DCE achieved a mean Dice score of 83.86% and an HD95 of 16.04 mm, outperforming the representative CNN-, Transformer-, and SAM-based baselines evaluated here. Ablation experiments further showed that SRC and STC provide complementary improvements, supporting the proposed spatial--semantic factorization for prompt-free SAM adaptation.

Yingzhen Hu, Yiheng Zhong, Aijing Yang +2

Mohamed bin Zayed University of AI, Abu Dhabi, UAE · Xi’an Jiaotong-Liverpool University, Suzhou, China · University of Liverpool, Liverpool, United Kingdom

Medical Image Segmentation Segment Anything Models Multi-Organ Segmentation

Ask PDF

Abstract

Image Segmentation

Explore similar work

HPR-SAM: Hierarchical Probabilistic Representation Learning for Prompt-free SAM-based Medical Image Segmentation

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

SAM-DCE: Decoupled Context Enhancement for Prompt-Free Medical Image Segmentation