Interactive segmentation is commonly used in medical image analysis to obtain precise, pixel-level labeling, typically involving iterative user input to correct mislabeled regions. However, existing approaches often fail to fully utilize user knowledge from interactive inputs and achieve comprehensive feature extraction. Specifically, these methods tend to treat all mislabeled regions equally, selecting them randomly for refinement without evaluating each region's potential impact on segmentation quality. Additionally, most models rely solely on spatial domain features, overlooking frequency domain information that could enhance feature extraction and improve performance. To address these limitations, we propose ActiveFreq, a novel interactive segmentation framework that integrates active learning and frequency domain analysis to minimize human intervention while achieving high-quality labeling. ActiveFreq introduces AcSelect, an autonomous module that prioritizes the most informative mislabeled regions, ensuring maximum performance gain from each click. Moreover, we develop FreqFormer, a segmentation backbone incorporating a Fourier transform module to map features from the spatial to the frequency domain, enabling richer feature extraction. Evaluations on the ISIC-2017 and OAI-ZIB datasets demonstrate that ActiveFreq achieves high performance with reduced user interaction, achieving 3.74 NoC@90 on ISIC-2017 and 9.27 NoC@90 on OAI-ZIB, with 23.5% and 12.8% improvements over previous best results, respectively. Under minimal input conditions, such as two clicks, ActiveFreq reaches mIoU scores of 85.29% and 75.76% on ISIC-2017 and OAI-ZIB, highlighting its efficiency and accuracy in interactive medical segmentation.
Figures & tables
Figure 1: Comparison between our proposed method and other methods. While other methods randomly select a mislabeled region to click, our approach uses an active learning selection module (AcSelect) to evaluate all mislabeled areas and select the most valuable one for annotation.
Figure 2: The overall pipeline of ActiveFreq. (a) Coarse segmentation is generated using the proposed FreqFormer. (b) Mislabeled regions within the coarse mask. (c) The AcSelect module selects the most informative region for refinement. This process evaluates each mislabeled region using three metrics: Maximum Pixel Entropy (MPE), Average Pixel Entropy (APE), and Regional Group Uncertainty (RGU). A Region Score (RS) is calculated for each region, and the region with the highest score is chosen for further refinement to produce the refined mask.
Figure 3: The diagram of FreqFormer. (a) SegFormer encoder extracts multi-scale features from the input image and user clicks. (b) FreqNet decoder, composed of four Freq layers, refines these features to produce a coarse mask. (c) Transformer Block architecture, where FFN means feed-forward network. (d) MLP Layer aligns feature maps for final concatenation. (e) Freq Block architecture in the Freq layer, in which FreqM represents the proposed frequency analysis module.
Figure 4: The architecture of frequency analysis module (FreqModule). The input feature map is split into four sub-feature maps: one is processed with depth-wise convolution (DW Conv) for spatial information, while the other three pass through 2D Discrete Fourier Transform (DFT) and 2D Inverse DFT (IDFT) for frequency information. The outputs are then concatenated to produce the final result.
Layer Type
Layer Index
Embed Dim
Attn Heads
Block Num
Transformer
1
32
1
2
2
64
2
2
3
128
4
2
4
256
8
2
FreqLayer
1
256
-
1
2
128
-
1
Table 1: Configuration of the FreqFormer architecture, including transformer layers in the encoder and Freq layers in the decoder. Embed dim means the channel C of feature maps, and attn heads are the number of heads in self-attention.
ISIC-2017 [ 28 ]
OAI-ZIB [ 29 ]
Method
Venue
Backbone
NoC@80
NoC@85
NoC@90
NoC@80
NoC@85
NoC@90
CDNet [ 68 ]
ICCV2021
ResNet-34
4.18
5.18
6.79
6.20
8.56
9.61
RITM [ 11 ]
ICIP2022
HRNet-18
2.90
3.70
5.69
8.41
14.53
19.25
RITM [ 11 ]
ICIP2022
HRNet-32
5.99
7.77
11.28
8.93
14.84
19.56
iSegFormer [ 69 ]
MICCAI2022
HRNet-32
2.73
3.66
5.91
8.37
12.99
18.81
FocalClick [ 15 ]
CVPR2022
HRNet-18
2.83
3.63
5.01
5.75
8.17
9.83
Table 2: Quantitative comparison of NoC performance with other methods. Results are reported on two medical benchmarks: ISIC-2017 [ 28 ] and OAI-ZIB [ 29 ] . The best scores are highlighted in bold, and the second-best are underlined.
Figure 5: Comparison of mIoU@2 performance between our proposed AcSelect and other state-of-the-art methods on two datasets: (a) Transformer-based approaches on ISIC-2017 [ 28 ] and (b) CNN-based methods on OAI-ZIB [ 29 ] .
Figure 6: mIoU scores with varying numbers of user clicks on two medical datasets. (a) and (b) are CNN-based methods and Transformer-based methods on ISIC-2017 [ 28 ] . (c) and (d) show CNN-based and Transformer-based models on OAI-ZIB [ 29 ] .
Backbone
Params/M
FLOPs/G
SPC/ms
ResNet34 [ 74 ]
23.47
113.60
34
Resnet50 [ 74 ]
40.36
78.82
331
HRNet18 [ 72 ]
10.03
30.99
56
HRNet32 [ 72 ]
30.95
83.12
86
Swin-Base [ 75 ]
87.44
138.21
36
Swin-Large [ 75 ]
195.90
302.78
44
Table 3: Computation comparison for model parameters, FLOPs, and speed (measured by seconds per click). The best scores are highlighted in bold, and the second-best are underlined.
Figure 7: Qualitative results on ISIC-2017 [ 28 ] . Each row shows (a) ground truth, and segmentation results for (b) iSegFormer [ 69 ] using Swin-B, (c) FocalClick [ 15 ] with SegFormerB0, (d) GraCo [ 71 ] based on ViT-B, (e) SimpleClick [ 33 ] with SegFormerB0, (f) the proposed ActiveFreq. Green points indicate positive clicks and red points indicate negative clicks. The red masks represent segmentation results, with numerical values indicating the Intersection over Union (IoU) between the predicted mask and the ground truth.
Figure 8: Visualization of the mIoU performance for interactive segmentation on four different medical samples using our proposed ActiveFreq with varying click counts. The results demonstrate how the performance improves as the number of clicks increases from 1 to 10.
MPE
APE
RGU
ISIC-2017 [ 28 ]
OAI-ZIB [ 29 ]
NoC@85
NoC@90
NoC@85
NoC@90
3.03
5.11
8.31
9.62
✓
2.75
4.62
7.83
9.49
✓
✓
2.43
4.19
7.37
9.37
✓
✓
✓
2.21
3.74
6.91
9.27
Table 4: Ablation study results of AcSelect. We use ActiveFreq (AcSelect with FreqFormer) trained on the ISIC-2017 [ 28 ] and OAI-ZIB [ 29 ] datasets. The experiments incrementally added the three key metrics (MPE, APE, and RGU) to assess their individual and combined contributions to segmentation performance. Results are reported in terms of NoC@85 and NoC@90.
Sampling Strategy
ISIC-2017
OAI-ZIB
NoC@85
NoC@90
NoC@85
NoC@90
Random
3.03
5.11
8.31
9.62
Entropy
2.71
4.68
7.85
9.50
Least Confidence
2.69
4.45
7.72
9.45
AcSelect (Ours)
2.21
3.74
6.91
9.27
Table 5: Ablation study results on the sampling strategy. This table compares different active learning sampling strategies, including Random Sampling, Entropy-Based Sampling, Least Confidence Sampling. The best results are highlighted in bold.
F(H,W)
F(H,C)
F(W,C)
ISIC-2017 [ 28 ]
OAI-ZIB [ 29 ]
NoC@85
NoC@90
NoC@85
NoC@90
3.14
5.22
8.42
9.73
✓
2.85
4.74
7.94
9.61
✓
✓
2.52
4.28
7.48
9.48
✓
✓
✓
2.21
3.74
6.91
9.27
Table 6: Ablation study results on the FreqModule. We use ActiveFreq (AcSelect with FreqFormer) trained on ISIC-2017 [ 28 ] and OAI-ZIB [ 29 ] . Experiments include progressively adding 2D DFT transformations ( F(H,W) , F(H,C) , and F(W,C) ) to evaluate their effectiveness.
Frequency Module
ISIC-2017
OAI-ZIB
NoC@85
NoC@90
NoC@85
NoC@90
Wavelet Transform
2.66
3.83
7.09
9.40
Fourier Transform
2.21
3.74
6.91
9.27
Table 7: Ablation study results on the Frequency Module. This table presents a comparison between Fourier Transform (FT) and Wavelet Transform (WT) in the frequency module across the ISIC-2017 and OAI-ZIB datasets. While WT demonstrates some capability in frequency domain representation, FT consistently outperforms WT across all metrics. The better results are highlighted in bold.
Interactive medical image segmentation uses clinician prompts and iterative model feedback to refine anatomical masks and reduce manual annotation effort.
IRCAD France, Strasbourg, France · Division of Hepatobiliary and General Surgery, Department of Surgery, IRCCS Humanitas Research Hospital, Milan, Italy · Department of Biomedical Sciences, Humanitas University, Milan, Italy +2
School of Computer Science and Technology, University of Science and Technology of China · School of Artificial Intelligence and Data Science, University of Science and Technology of China