Organizations: School of Artificial Intelligence and the National Engineering Laboratory of Big Data System Computing Technology, Shenzhen University, Shenzhen 518060, China · School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China · Department of Computer Science, University of Nottingham Ningbo China, Ningbo 315100, China
Few-shot semantic segmentation has attracted growing interest for its ability to generalize to novel object categories using only a few annotated samples. To address data scarcity, recent methods incorporate multiple foundation models to improve feature transferability and segmentation performance. However, they often rely on dual-branch architectures that combine pre-trained encoders to leverage complementary strengths, a design that limits flexibility and efficiency. This raises a fundamental question: can we build a unified model that integrates knowledge from different foundation architectures? Achieving this is, however, challenging due to the misalignment between class-agnostic segmentation capabilities and fine-grained discriminative representations. To this end, we present UINO-FSS, a novel framework built on the key observation that early-stage DINOv2 features exhibit distribution consistency with SAM's output embeddings. This consistency enables the integration of both models' knowledge into a single-encoder architecture via coarse-to-fine multimodal distillation. In particular, our segmenter consists of three core components: a bottleneck adapter for embedding alignment, a meta-visual prompt generator that leverages dense similarity volumes and semantic embeddings, and a mask decoder. Using hierarchical cross-model distillation, we effectively transfer SAM's knowledge into the segmenter, further enhanced by Mamba-based 4D correlation mining on support-query pairs. Extensive experiments on PASCAL-5i and COCO-20i show that UINO-FSS achieves new state-of-the-art results under the 1-shot setting, with mIoU of 80.6 (+3.8%) on PASCAL-5i and 64.5 (+4.1%) on COCO-20i, demonstrating the effectiveness of our unified approach.
Figures & tables
Fig. 1: Introduction of our framework. Our framework (b) is unified and efficient compared to the existing dual-modal architectures [ 1 , 2 , 3 ] shown in (a). Our lightweight segmenter (c) only contains 5.6M parameters, but gains knowledge from the powerful SAM via the procedure in (d).
Fig. 2: Framework comparison with prior FSS works. Existing SAM-based FSS methods, such as Matcher [ 1 ] and VRP-SAM [ 3 ] , typically adopt a dual-encoder design, using SAM’s encoder and mask decoder alongside a separate DINOv2 extractor for support-query feature extraction. SegIC adopts a single-encoder framework but requires CLIP for meta prompts. In contrast, UINO-FSS employs a single DINOv2 encoder with a lightweight bottleneck adapter, extracting features only once without any additional extractor. For prompt generation, prior methods rely on sparse prompts (points/boxes) derived from 2D distance maps between support prototypes and query features. UINO-FSS provides a unified framework with Meta-Visual Prompt Generator (MVPG) that eliminates sparse points and boxes entirely, relying instead on dense prompts primarily. Specifically, we model 4D correlations between support feature map and the query correspondence via contrastive enhancement and a Mamba-HyperCorrelation Module (MHCM) followed by Dense Prompt Generation (DPG), while introducing semantic-aware sparse prompts that replace explicit point/box inputs.
Fig. 3: Analyzation on the embeddings from DINOv2 and SAM . We compute the self-similarity map (SSM) of feature vectors on red dots to analyze the embedding distribution of both models and visualize them in the figure. Here, (a) shows the SSM of features from the last layer of SAM’s encoder, while (b) displays the SSM of features from all layers of DINOv2-Base. Unlike SAM’s holistic semantic focus, DINOv2’s high-level embeddings concentrate on various local regions, offering richer representation. Despite this, we found that embeddings from DINOv2’s 3rd layer are most similar to SAM’s encoder embeddings. This observation enables efficient cross-model distillation for our lightweight segmenter.
Fig. 4: CKNNA values between features from various DINOv2 layers and the SAM Encoder. The star markers denote the CKNNA results after distillation. Specifically, w/o Distillation indicates the CKNNA obtained on original DINOv2 embeddings, while Distillation presents the scores on DINOv2 embeddings learned with distillation.
Fig. 5: The proposed UINO-FSS architecture. Our architecture consists of a DINOv2 encoder and a lightweight segmenter that includes a bottleneck adapter (BA), a Meta-Visual Prompt Generator (MVPG) and a mask decoder. Upper part is the coarse-to-fine cross-model distillation procedure, with only the adapter trainable for feature matching. Below is the overall architecture of our few-shot semantic segmentation model. Our MVPG includes two modules for the Semantic-aware Visual Prompts (SVP) and the Mamba-based dense prompts, respectively.
Fig. 6: Network structure for the Mamba-HyperCorrelation Module (MHCM). MHCM stacks two Hierarchical Global Modeling Blocks (HGMB), which process 4D volumetric correlation while maintaining high efficiency.
Method
Backbone
Trainable Params(M)
Total Params(M)
1-shot
5-shot
Fold0
Fold1
Fold2
Fold3
Mean
Fold0
Fold1
Fold2
Fold3
Mean
BAM CVPR′22 [ 71 ]
ResNet50
4.9
51.7
69.0
73.6
67.6
61.1
67.8
70.6
75.1
70.8
67.2
70.9
HDMNet CVPR′23 [ 16 ]
4.2
50.9
71.0
75.4
68.9
62.1
69.4
71.3
76.2
71.3
68.5
71.8
AENet ECCV’24 [ 72 ]
4.4
51.2
71.3
75.9
68.6
65.4
70.3
73.9
77.8
73.3
72.0
74.2
OCNet ICCV′25 [ 73 ]
-
-
73.5
75.9
71.1
64.9
71.4
75.9
77.1
74.1
70.9
74.5
MSI ICCV’23 [ 45 ]
ResNet101
3.2
47.8
73.1
73.9
64.7
68.8
70.1
73.6
76.1
68.0
71.3
72.2
TABLE I: Performance comparisons on PASCAL-5 i under 1-shot and 5-shot settings. Results in bold indicate the best performance, and results with underlining represent the second-best performance.
Method
Backbone
Trainable Params(M)
Total Params(M)
1-shot
5-shot
Fold0
Fold1
Fold2
Fold3
Mean
Fold0
Fold1
Fold2
Fold3
Mean
BAM CVPR′22 [ 71 ]
ResNet50
4.9
51.7
43.4
50.6
47.5
43.4
46.2
49.3
54.2
51.6
49.6
51.2
HDMNet CVPR′23 [ 16 ]
4.2
50.9
43.8
55.3
51.6
49.4
50.0
50.6
61.6
55.7
56.0
56.0
AENet ECCV’24 [ 72 ]
4.4
51.2
45.4
57.1
52.6
50.0
51.3
52.7
62.6
56.8
56.1
57.1
OCNet ICCV′25 [ 73 ]
-
-
45.9
56.9
52.9
50.4
51.5
52.7
63.1
57.4
54.8
57.0
MSI ICCV’23 [ 45 ]
ResNet101
3.2
47.8
44.8
54.2
52.3
48.0
49.8
49.3
58.0
56.1
52.7
54.0
TABLE II: Performance comparisons on COCO-20 i under 1-shot and 5-shot settings. Our method consistently achieves the best performance, outperforming other approaches based on DINOv2-Base (DINOv2-B) as well as heavier architectures combining DINOv2-Large (DINOv2-L) and SAM-ViT-H (SAM-H).
Metric
Painter [ 76 ]
SegGPT [ 77 ]
PerSAM [ 25 ]
PerSAM-F [ 25 ]
SEGIC [ 26 ]
Ours
mIoU
61.7
85.6
71.2
75.6
86.8
89.0
TABLE III: Performance on FSS-1000 under one-shot setting.
Method
Train Params
Total Params
mIoU
Matcher
0
945.5M
52.7
GF-SAM
0
945.5M
58.7
FCP
2.9M
667.6M
52.5
SEGIC
5.2M
214.9M
53.6
VRP-SAM (ResNet50)
1.6M
666.4M
53.9
UINO-FSS-Comp*
0.07M
92.0M
57.2
TABLE IV: Additional Complexity ablation study on variants of UINO-FSS and prior methods on COCO-20 i under 1-shot setting. UINO-FSS-Comp(*) refers to our compact model without decoder training.
Method
Latency (ms)
Peak Memory (M)
mIoU
Matcher [ 1 ]
6218.0
6936.4
52.7
GF-SAM [ 2 ]
726.4
6910.2
58.7
FCP [ 75 ]
941.9
5975.1
52.5
VRP-SAM [ 3 ]
472.6
5942.3
53.9
UINO-FSS
93.4
5558.0
64.5
TABLE V: Comparison of inference efficiency across methods, including inference time and peak GPU memory usage during testing.
BA
PS
SVP
MHCM
CE
mIoU (%)
✓
✓
58.2
✓
✓
✓
59.6
✓
✓
62.0
✓
✓
✓
✓
62.3
✓
✓
✓
✓
✓
64.5
TABLE VI: Ablation study results on COCO-20 i under the one-shot setting. Here, we denote BA as the Bottleneck Adapter, PS the Prototype-based Similarity, SVP the Semantic-aware Visual Prompt, MHCM the Mamba-HyperCorrelation Module and CE the Contrastive Enhancement Module.
Dataset
Method
Fold0
Fold1
Fold2
Fold3
mIoU
COCO-20 i
UINO-FSS w/ CP4D Conv † [ 17 ]
61.9
65.9
64.9
63.1
64.0
UINO-FSS
62.2
66.0
65.5
64.2
64.5
PASCAL-5 i
UINO-FSS w/ CP4D Conv † [ 17 ]
79.6
82.9
77.0
79.3
79.7
UINO-FSS
80.7
83.2
77.8
80.5
80.6
TABLE VII: More ablation studies on HGMB under the one-shot setting, where † denotes the abbreviation for Center-pivot 4D Convolution. We replace HGMB with CP4D Conv, keeping all other modules unchanged. Our HGMB demonstrates significant improvements over CP4D Conv on both COCO-20 i and PASCAL-5 i .
Fig. 7: Qualitative results of UINO-FSS on COCO-20 i under the one-shot setting. From top to bottom in each row are the support image with its corresponding mask, the query image with ground-truth annotation, the output of UINO-FSS without CE and MHCM modules, the output of UINO-FSS without CE, and the output of the complete UINO-FSS model. Red circles indicate inaccurate segmentation.
Method
Fold0
Fold1
Fold2
Fold3
mIoU
UINO-FSS-Comp w/o distillation
50.5
55.2
53.5
52.1
52.8
UINO-FSS-Comp
55.5
63.8
59.4
59.5
59.6
TABLE VIII: More studies on cross-model distillation from coarse to fine.
Method
Fold0
Fold1
Fold2
Fold3
mIoU
UINO-FSS-Comp w/ DINOv2-Small
49.2
58.5
55.3
54.3
54.3
UINO-FSS-Comp w/ DINOv2-Base
55.5
63.8
59.4
59.5
59.6
UINO-FSS-Comp w/ DINOv2-Large
57.6
63.8
61.6
59.9
60.7
TABLE IX: Performance comparison of our compact model with different DINOv2 backbones (small/base/large) on COCO-20 i under 1-shot setting.
Few-Shot Semantic Segmentation (FSS) focuses on segmenting novel object categories from only a handful of annotated examples. Most existing approaches rely on extensive episodic training to learn transferable representations, which is both computationally demanding and sensitive to distribution shifts. In this work, we revisit FSS from the perspective of modern vision foundation models and explore the potential of Segment Anything Model 3 (SAM3) as a training-free solution. By repurposing its Promptable Concept Segmentation (PCS) capability, we adopt a simple spatial concatenation strategy that places support and query images into a shared canvas, allowing a fully frozen SAM3 to perform segmentation without any fine-tuning or architectural changes. Experiments on PASCAL-5i and COCO-20i show that this minimal design already achieves state-of-the-art performance, outperforming many heavily engineered methods. Beyond empirical gains, we uncover that negative prompts can be counterproductive in few-shot settings, where they often weaken target representations and lead to prediction collapse despite their intended role in suppressing distractors. These findings suggest that strong cross-image reasoning can emerge from simple spatial formulations, while also highlighting limitations in how current foundation models handle conflicting prompt signals. Code at: https://github.com/WongKinYiu/FSS-SAM3
Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao Wang
National Yang Ming Chiao Tung University · Institute of Information Science, Academia Sinica, Taiwan
Few-shot segmentation (FSS) has gained significant attention for its ability to generalize to novel classes with limited supervision, yet remains challenged by structural misalignment and cross-view inconsistency under large appearance or viewpoint variations. This paper tackles these challenges by introducing VINE (View-Informed NEtwork), a unified framework that jointly models structural consistency and foreground discrimination to refine class-specific prototypes. Specifically, VINE introduces a spatial-view graph on backbone features, where the spatial graph captures local geometric topology and the view graph connects features from different perspectives to propagate view-invariant structural semantics. To further alleviate foreground ambiguity, we derive a discriminative prior from the support-query feature discrepancy to capture category-specific contrast, which reweights SAM features by emphasizing salient regions and recalibrates backbone activations for improved structural focus. The foreground-enhanced SAM features and structurally enriched ResNet features are progressively integrated through masked cross-attention, yielding class-consistent prototypes used as adaptive prompts for the SAM decoder to generate accurate masks. Extensive experiments on multiple FSS benchmarks validate the effectiveness and robustness of VINE, particularly under challenging scenarios with viewpoint shifts and complex structures. The code is available at https://github.com/HongliLiu1/VINE-main.
Hongli Liu, Yu Wang, Shengjie Zhao
School of Computer Science and Technology, Tongji University
This paper investigates how to leverage the powerful capabilities of the Segment Anything Model 2 (SAM2) to tackle few-shot segmentation (FSS) tasks. One major challenge lies in the substantial appearance differences between the support and query images, which may diminish the effectiveness of the memory-matching process in SAM2. Through systematic analysis, we find that the memory generated based on the categorical semantic information is robust to this appearance divergence and is more compatible with SAM2. Based on these observations, we propose a new Memory-Enhanced SAM2 (MESAM) method, which first introduces a Semantic-Visual Memory Generation (SVMG) module to separately construct a query semantic memory, a query foreground visual memory, and a support adaptive visual memory. Then, in our designed Interactive Semantic Memory Refinement (ISMR) module, we further enrich the obtained semantic memory with fine-grained visual details, and suppress its misatcivated background responses. Extensive experiments on the PASCAL-5i, COCO-20i, and LVIS-92i datasets demonstrate that MESAM consistently outperforms existing FSS methods, achieving new state-of-the-art segmentation performance. Our source code is available at: https://github.com/tianyu-zou/MESAM.
Tianyu Zou, Shengwu Xiong, Rui Wu +2
Sanya Science and Education Innovation Park, Wuhan University of Technology Sanya, Hainan, China · School of Artificial Intelligence, Wuhan University of Technology Wuhan, Hubei, China · Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology Wuhan, Hubei, China +1