Organizations: School of Artificial Intelligence and the National Engineering Laboratory of Big Data System Computing Technology, Shenzhen University, Shenzhen 518060, China · School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China · Department of Computer Science, University of Nottingham Ningbo China, Ningbo 315100, China
Few-shot semantic segmentation has attracted growing interest for its ability to generalize to novel object categories using only a few annotated samples. To address data scarcity, recent methods incorporate multiple foundation models to improve feature transferability and segmentation performance. However, they often rely on dual-branch architectures that combine pre-trained encoders to leverage complementary strengths, a design that limits flexibility and efficiency. This raises a fundamental question: can we build a unified model that integrates knowledge from different foundation architectures? Achieving this is, however, challenging due to the misalignment between class-agnostic segmentation capabilities and fine-grained discriminative representations. To this end, we present UINO-FSS, a novel framework built on the key observation that early-stage DINOv2 features exhibit distribution consistency with SAM's output embeddings. This consistency enables the integration of both models' knowledge into a single-encoder architecture via coarse-to-fine multimodal distillation. In particular, our segmenter consists of three core components: a bottleneck adapter for embedding alignment, a meta-visual prompt generator that leverages dense similarity volumes and semantic embeddings, and a mask decoder. Using hierarchical cross-model distillation, we effectively transfer SAM's knowledge into the segmenter, further enhanced by Mamba-based 4D correlation mining on support-query pairs. Extensive experiments on PASCAL-5i and COCO-20i show that UINO-FSS achieves new state-of-the-art results under the 1-shot setting, with mIoU of 80.6 (+3.8%) on PASCAL-5i and 64.5 (+4.1%) on COCO-20i, demonstrating the effectiveness of our unified approach.
Figures & tables
Fig. 1: Introduction of our framework. Our framework (b) is unified and efficient compared to the existing dual-modal architectures [ 1 , 2 , 3 ] shown in (a). Our lightweight segmenter (c) only contains 5.6M parameters, but gains knowledge from the powerful SAM via the procedure in (d).
Fig. 2: Framework comparison with prior FSS works. Existing SAM-based FSS methods, such as Matcher [ 1 ] and VRP-SAM [ 3 ] , typically adopt a dual-encoder design, using SAM’s encoder and mask decoder alongside a separate DINOv2 extractor for support-query feature extraction. SegIC adopts a single-encoder framework but requires CLIP for meta prompts. In contrast, UINO-FSS employs a single DINOv2 encoder with a lightweight bottleneck adapter, extracting features only once without any additional extractor. For prompt generation, prior methods rely on sparse prompts (points/boxes) derived from 2D distance maps between support prototypes and query features. UINO-FSS provides a unified framework with Meta-Visual Prompt Generator (MVPG) that eliminates sparse points and boxes entirely, relying instead on dense prompts primarily. Specifically, we model 4D correlations between support feature map and the query correspondence via contrastive enhancement and a Mamba-HyperCorrelation Module (MHCM) followed by Dense Prompt Generation (DPG), while introducing semantic-aware sparse prompts that replace explicit point/box inputs.
Fig. 3: Analyzation on the embeddings from DINOv2 and SAM . We compute the self-similarity map (SSM) of feature vectors on red dots to analyze the embedding distribution of both models and visualize them in the figure. Here, (a) shows the SSM of features from the last layer of SAM’s encoder, while (b) displays the SSM of features from all layers of DINOv2-Base. Unlike SAM’s holistic semantic focus, DINOv2’s high-level embeddings concentrate on various local regions, offering richer representation. Despite this, we found that embeddings from DINOv2’s 3rd layer are most similar to SAM’s encoder embeddings. This observation enables efficient cross-model distillation for our lightweight segmenter.
Fig. 4: CKNNA values between features from various DINOv2 layers and the SAM Encoder. The star markers denote the CKNNA results after distillation. Specifically, w/o Distillation indicates the CKNNA obtained on original DINOv2 embeddings, while Distillation presents the scores on DINOv2 embeddings learned with distillation.
Fig. 5: The proposed UINO-FSS architecture. Our architecture consists of a DINOv2 encoder and a lightweight segmenter that includes a bottleneck adapter (BA), a Meta-Visual Prompt Generator (MVPG) and a mask decoder. Upper part is the coarse-to-fine cross-model distillation procedure, with only the adapter trainable for feature matching. Below is the overall architecture of our few-shot semantic segmentation model. Our MVPG includes two modules for the Semantic-aware Visual Prompts (SVP) and the Mamba-based dense prompts, respectively.
Fig. 6: Network structure for the Mamba-HyperCorrelation Module (MHCM). MHCM stacks two Hierarchical Global Modeling Blocks (HGMB), which process 4D volumetric correlation while maintaining high efficiency.
Method
Backbone
Trainable Params(M)
Total Params(M)
1-shot
5-shot
Fold0
Fold1
Fold2
Fold3
Mean
Fold0
Fold1
Fold2
Fold3
Mean
BAM CVPR′22 [ 71 ]
ResNet50
4.9
51.7
69.0
73.6
67.6
61.1
67.8
70.6
75.1
70.8
67.2
70.9
HDMNet CVPR′23 [ 16 ]
4.2
50.9
71.0
75.4
68.9
62.1
69.4
71.3
76.2
71.3
68.5
71.8
AENet ECCV’24 [ 72 ]
4.4
51.2
71.3
75.9
68.6
65.4
70.3
73.9
77.8
73.3
72.0
74.2
OCNet ICCV′25 [ 73 ]
-
-
73.5
75.9
71.1
64.9
71.4
75.9
77.1
74.1
70.9
74.5
MSI ICCV’23 [ 45 ]
ResNet101
3.2
47.8
73.1
73.9
64.7
68.8
70.1
73.6
76.1
68.0
71.3
72.2
TABLE I: Performance comparisons on PASCAL-5 i under 1-shot and 5-shot settings. Results in bold indicate the best performance, and results with underlining represent the second-best performance.
Method
Backbone
Trainable Params(M)
Total Params(M)
1-shot
5-shot
Fold0
Fold1
Fold2
Fold3
Mean
Fold0
Fold1
Fold2
Fold3
Mean
BAM CVPR′22 [ 71 ]
ResNet50
4.9
51.7
43.4
50.6
47.5
43.4
46.2
49.3
54.2
51.6
49.6
51.2
HDMNet CVPR′23 [ 16 ]
4.2
50.9
43.8
55.3
51.6
49.4
50.0
50.6
61.6
55.7
56.0
56.0
AENet ECCV’24 [ 72 ]
4.4
51.2
45.4
57.1
52.6
50.0
51.3
52.7
62.6
56.8
56.1
57.1
OCNet ICCV′25 [ 73 ]
-
-
45.9
56.9
52.9
50.4
51.5
52.7
63.1
57.4
54.8
57.0
MSI ICCV’23 [ 45 ]
ResNet101
3.2
47.8
44.8
54.2
52.3
48.0
49.8
49.3
58.0
56.1
52.7
54.0
TABLE II: Performance comparisons on COCO-20 i under 1-shot and 5-shot settings. Our method consistently achieves the best performance, outperforming other approaches based on DINOv2-Base (DINOv2-B) as well as heavier architectures combining DINOv2-Large (DINOv2-L) and SAM-ViT-H (SAM-H).
Metric
Painter [ 76 ]
SegGPT [ 77 ]
PerSAM [ 25 ]
PerSAM-F [ 25 ]
SEGIC [ 26 ]
Ours
mIoU
61.7
85.6
71.2
75.6
86.8
89.0
TABLE III: Performance on FSS-1000 under one-shot setting.
Method
Train Params
Total Params
mIoU
Matcher
0
945.5M
52.7
GF-SAM
0
945.5M
58.7
FCP
2.9M
667.6M
52.5
SEGIC
5.2M
214.9M
53.6
VRP-SAM (ResNet50)
1.6M
666.4M
53.9
UINO-FSS-Comp*
0.07M
92.0M
57.2
TABLE IV: Additional Complexity ablation study on variants of UINO-FSS and prior methods on COCO-20 i under 1-shot setting. UINO-FSS-Comp(*) refers to our compact model without decoder training.
Method
Latency (ms)
Peak Memory (M)
mIoU
Matcher [ 1 ]
6218.0
6936.4
52.7
GF-SAM [ 2 ]
726.4
6910.2
58.7
FCP [ 75 ]
941.9
5975.1
52.5
VRP-SAM [ 3 ]
472.6
5942.3
53.9
UINO-FSS
93.4
5558.0
64.5
TABLE V: Comparison of inference efficiency across methods, including inference time and peak GPU memory usage during testing.
BA
PS
SVP
MHCM
CE
mIoU (%)
✓
✓
58.2
✓
✓
✓
59.6
✓
✓
62.0
✓
✓
✓
✓
62.3
✓
✓
✓
✓
✓
64.5
TABLE VI: Ablation study results on COCO-20 i under the one-shot setting. Here, we denote BA as the Bottleneck Adapter, PS the Prototype-based Similarity, SVP the Semantic-aware Visual Prompt, MHCM the Mamba-HyperCorrelation Module and CE the Contrastive Enhancement Module.
Dataset
Method
Fold0
Fold1
Fold2
Fold3
mIoU
COCO-20 i
UINO-FSS w/ CP4D Conv † [ 17 ]
61.9
65.9
64.9
63.1
64.0
UINO-FSS
62.2
66.0
65.5
64.2
64.5
PASCAL-5 i
UINO-FSS w/ CP4D Conv † [ 17 ]
79.6
82.9
77.0
79.3
79.7
UINO-FSS
80.7
83.2
77.8
80.5
80.6
TABLE VII: More ablation studies on HGMB under the one-shot setting, where † denotes the abbreviation for Center-pivot 4D Convolution. We replace HGMB with CP4D Conv, keeping all other modules unchanged. Our HGMB demonstrates significant improvements over CP4D Conv on both COCO-20 i and PASCAL-5 i .
Fig. 7: Qualitative results of UINO-FSS on COCO-20 i under the one-shot setting. From top to bottom in each row are the support image with its corresponding mask, the query image with ground-truth annotation, the output of UINO-FSS without CE and MHCM modules, the output of UINO-FSS without CE, and the output of the complete UINO-FSS model. Red circles indicate inaccurate segmentation.
Method
Fold0
Fold1
Fold2
Fold3
mIoU
UINO-FSS-Comp w/o distillation
50.5
55.2
53.5
52.1
52.8
UINO-FSS-Comp
55.5
63.8
59.4
59.5
59.6
TABLE VIII: More studies on cross-model distillation from coarse to fine.
Method
Fold0
Fold1
Fold2
Fold3
mIoU
UINO-FSS-Comp w/ DINOv2-Small
49.2
58.5
55.3
54.3
54.3
UINO-FSS-Comp w/ DINOv2-Base
55.5
63.8
59.4
59.5
59.6
UINO-FSS-Comp w/ DINOv2-Large
57.6
63.8
61.6
59.9
60.7
TABLE IX: Performance comparison of our compact model with different DINOv2 backbones (small/base/large) on COCO-20 i under 1-shot setting.
Sanya Science and Education Innovation Park, Wuhan University of Technology Sanya, Hainan, China · School of Artificial Intelligence, Wuhan University of Technology Wuhan, Hubei, China · Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology Wuhan, Hubei, China +1