Unify the Views: View-Consistent Prototype Learning for Few-Shot Segmentation

Hongli Liu; Yu Wang; Shengjie Zhao

cs.CVMar 6, 2026

Unify the Views: View-Consistent Prototype Learning for Few-Shot Segmentation

Authors: Hongli Liu, Yu Wang, Shengjie Zhao

Organizations: School of Computer Science and Technology, Tongji University

Abstract

Few-shot segmentation (FSS) has gained significant attention for its ability to generalize to novel classes with limited supervision, yet remains challenged by structural misalignment and cross-view inconsistency under large appearance or viewpoint variations. This paper tackles these challenges by introducing VINE (View-Informed NEtwork), a unified framework that jointly models structural consistency and foreground discrimination to refine class-specific prototypes. Specifically, VINE introduces a spatial-view graph on backbone features, where the spatial graph captures local geometric topology and the view graph connects features from different perspectives to propagate view-invariant structural semantics. To further alleviate foreground ambiguity, we derive a discriminative prior from the support-query feature discrepancy to capture category-specific contrast, which reweights SAM features by emphasizing salient regions and recalibrates backbone activations for improved structural focus. The foreground-enhanced SAM features and structurally enriched ResNet features are progressively integrated through masked cross-attention, yielding class-consistent prototypes used as adaptive prompts for the SAM decoder to generate accurate masks. Extensive experiments on multiple FSS benchmarks validate the effectiveness and robustness of VINE, particularly under challenging scenarios with viewpoint shifts and complex structures. The code is available at https://github.com/HongliLiu1/VINE-main.

Figures & tables

Figure 1: Comparison between conventional prototype modeling and VINE. (a) View-induced challenges: large intra-class variation and inter-class similarity lead to prototype confusion. (b) Top: conventional mask-guided enhancement fails under viewpoint shifts, generating unstable prototypes. Bottom: VINE unifies spatial–view graph alignment and discriminative modulation to produce view-consistent, structurally reliable prototypes.

Figure 2: An overview of our proposed framework. We incorporate Spatial-View Graph Alignment (SVGA) to capture structural consistency across views, and Discriminative Foreground Modulation (DFM) to highlight foreground-relevant features, collaboratively improving the quality of support-query prototypes and guiding accurate mask prediction.

Method	IE	PASCAL-5 i										COCO-20 i
		1-shot					5-shot					1-shot					5-shot
		F-0	F-1	F-2	F-3	Mean	F-0	F-1	F-2	F-3	Mean	F-0	F-1	F-2	F-3	Mean	F-0	F-1	F-2	F-3	Mean
HDMNet [ 22 ]	VGG	64.8	71.4	67.7	56.4	65.1	68.1	73.1	71.8	64.0	69.3	40.7	50.6	48.2	44.0	45.9	47.0	56.5	54.1	51.9	52.4
VRP-SAM † [ 27 ]		69.9	73.9	67.6	62.3	68.4	72.8	75.0	67.5	63.2	69.6	39.4	52.0	50.6	47.5	47.4	43.6	57.7	54.7	51.8	51.9
HMNet [ 37 ]		66.7	74.5	68.9	59.0	67.3	70.5	76.0	72.2	65.7	71.1	44.2	51.8	51.9	48.4	49.1	48.8	58.0	57.9	53.2	54.5
FCP † [ 21 ]		71.9	75.0	69.8	65.5	70.5	73.5	76.4	71.6	65.8	71.8	44.5	54.3	53.5	47.7	50.0	49.0	58.3	55.2	52.3	53.7

Table 1: Comparison with state-of-the-art methods on PASCAL-5 i and COCO-20 i datasets under 1-shot and 5-shot settings. The mean mIoU (%) across 4 folds (F-0 to F-3) is reported. † denotes methods based on SAM.

Method	Train (A)	Test (B)	mIoU (%)	$\Delta$
Baseline (similar)	dog	person	43.69	–
Ours	dog	person	53.41	+9.72
Baseline (typical)	horse	person	44.28	–
Ours	horse	person	52.62	+8.34
Baseline (divergent)	dog	motorbike	17.76	–
Ours	dog	motorbike	36.28	+18.52

Table 2: Cross-class generalization on PASCAL-5 i (fold-2).

ID	Res	SAM	DFM	SVGA	mIoU (%)	$\Delta$
(a)	✓				71.8	–
(b)		✓			66.2	-5.6
(c)	✓	✓			72.5	+0.7
(d)	✓	✓	✓		73.3	+1.5
(e)	✓	✓		✓	73.8	+2.0
(f)	✓	✓	✓	✓	74.2	+2.4

Table 3: Ablation study of key components in VINE.

(1) Loss components
Configuration	mIoU (%)	$\Delta$
(a) Only $\mathcal{L}_{\text{predict}}$	73.4	–
(b) Only $\mathcal{L}_{\text{proto}}$	73.6	+0.2
(c) Both losses	74.2	+0.8
(2) SVGA graph configuration
None	73.3	–

Table 4: Ablation studies on key hyper-parameters. The analysis includes (1) the impact of loss composition, (2) the configuration of spatial and view graphs in SVGA, and (3) the sensitivity to loss-weight ratios

(\lambda_{\text{proto}},\lambda_{\text{predict}})

Panel of Figure 5 — Figure 5: Qualitative and feature-level comparison of segmentation results.

Few-Shot Semantic Segmentation

Few-shot semantic segmentation labels pixels in new image classes using only a small set of annotated examples.

See all 32 papers

Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentation
Sep 24, 2026Yijun Hu, Heng Fan, Libo ZhangFew-Shot Semantic Segmentation
UCAS
Few-Shot Semantic Segmentation Meets SAM3
Apr 7, 2026Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao WangFew-Shot Semantic Segmentation Vision Foundation Models
National Yang Ming Chiao Tung University · Institute of Information Science, Academia Sinica, Taiwan
Training-free Cross-domain Few-shot Segmentation via Robust Semantic Representation and Matching
Jun 23, 2026Sujun Sun, Mingwu Ren, Haofeng ZhangFew-Shot Learning Cross-Domain Transfer Learning
School of Computer Science and Engineering, Nanjing University of Science and Technology, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, China

Sep 24, 2026J/K move · Enter open · S save

Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentation

Yijun Hu, Heng Fan, Libo Zhang

UCAS

Few-Shot Semantic Segmentation

Open paper AskPDF arXiv

Few-shot segmentation (FSS) aims to segment unseen object categories with a few (e.g., one or five) labeled examples, enabling efficient adaptation to novel classes. Conventional models typically rely on appearance-based visual matching between support and query images for segmentation. While straightforward, these methods often struggle to handle significant appearance discrepancies and occlusions in the query image due to insufficient target knowledge. To mitigate this, we introduce a novel framework that mines target knowledge using the strong reasoning capacity of Multimodal Large Language Models (MLLMs) and employs it to enhance FSS. Specifically, building on SAM 2, our method, named MK-FSS, exploits two forms of complementary knowledge derived from a query image by an MLLM for FSS, including spatial knowledge, which provides a spatial prior indicating the potential target location, and semantic knowledge, which describes the target using text. The spatial knowledge is first encoded into a memory representation, and then resulting memory is integrated with the support-guided memory feature from query image through a carefully designed dual-memory debate-fusion (DMDF) module, yielding a more robust target memory feature. In parallel, the semantic knowledge is encoded into the textual feature, which is fused with multi-scale query features via a progressive cross-modal prompt generator (PCPG), producing a target-aware multimodal prompt for segmentation. Working together, the dual-memory feature and the multimodal prompt provide a comprehensive representation of the target, enabling more robust segmentation. In our extensive experiments, MK-FSS shows promising results and largely surpasses existing methods. Code will be released.

Abstract

Few-Shot Semantic Segmentation

Explore similar work

Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentation

Few-Shot Semantic Segmentation Meets SAM3

Training-free Cross-domain Few-shot Segmentation via Robust Semantic Representation and Matching