From Visual Matching to Textual Conditioning: Cross-Domain Few-Shot Segmentation via Reference Inversion

Jiahao Nie; Yun Xing; Wenbin An; Qixin Zhang; Yap-Peng Tan; Alex C. Kot; Shijian Lu

cs.CVNov 2, 2026

From Visual Matching to Textual Conditioning: Cross-Domain Few-Shot Segmentation via Reference Inversion

Authors: Jiahao Nie, Yun Xing, Wenbin An, Qixin Zhang, Yap-Peng Tan, Alex C. Kot, Shijian Lu

Organizations: Nanyang Technological University · Xi’an Jiaotong University · VinUniversity · Shenzhen MSU-BIT University

Abstract

Cross-Domain Few-Shot Segmentation (CD-FSS) aims to segment objects from unseen domains using only a few annotated references. Recent Segment Anything Model (SAM)-based methods have advanced CD-FSS, yet their reference-target visual matching is highly sensitive to target appearance variations. SAM3 introduces text-conditioned segmentation, enabling natural-language descriptions to guide segmentation across target images, but directly using the category name as the condition still yields unstable performance under severe domain shifts. We propose Reference Inversion for SAM3 (RI-SAM3) that leverages SAM3's large-scale vision-language pre-training to learn suitable textual tokens for each target category. RI-SAM3 comprises three core designs: (i) Bi-level Reference Inversion that learns complementary tokens before and after the frozen text encoder: the former remain anchored in SAM3's text space while the latter represent the category more flexibly; (ii) Layer-adaptive Textual Conditioning combines these tokens differently as object queries evolve across decoder layers; and (iii) Presence-aware Confidence Supervision that jointly supervises query confidence and target presence for reliable predictions. With SAM3 frozen, RI-SAM3 optimizes around 12K parameters only per category and it efficiently segments each target in a single forward pass with reusable textual conditioning without per-sample reference-target visual matching. Extensive experiments over multiple widely adopted benchmarks show that RI-SAM3 achieves superior performance consistently in CD-FSS tasks, together with strong robustness, efficiency, and generalization.

Figures & tables

Figure 1: Motivation for Reference Inversion for SAM3 (RI-SAM3). Left : Using the same reference image, GF-SAM ( Zhang et al. 2024a ) exhibits substantial IoU variations across different target images, while manually specified text prompts produce low and unstable results under severe domain shifts. Our proposed RI-SAM3 instead inverts annotated references into a reusable textual condition for inference targets. Right : RI-SAM3 achieves higher mIoU (

\uparrow

) and a lower standard deviation of per-image IoU (

\downarrow

) than GF-SAM and the category-name prompt baseline, demonstrating improved accuracy and stability.

Figure 2: Overview of our proposed RI-SAM3 framework. Bi-level Reference Inversion (BRI) learns reference word tokens $z_{w}^{*}$ and reference concept tokens $z_{c}^{*}$ from annotated references, collectively termed inverted reference tokens. Layer-adaptive Textual Conditioning (LTC) combines them with layer-specific fusion coefficients $\alpha_{*}$ to produce the bi-level reference tokens $T_{c}^{*}$ for conditioning frozen SAM3. Presence-aware Confidence Supervision (PCS) jointly supervises query-level classification and global presence scores. Once learned, $T_{c}^{*}$ is reused for target segmentation without references or visual matching.

Conventional ResNet-/ViT-based Methods
Method	Venue	ISIC		Chest X-Ray		DeepGlobe		SUIM
Method	Venue	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
ISA	ICCV 2025	37.2	56.1	83.4	86.3	44.3	52.7	44.1	–
DR-Adapter	CVPR 2024	40.8	48.9	82.4	82.3	41.3	50.1	–	–
PATNet	ECCV 2022	41.2	53.6	66.6	70.2	37.9	43.0	32.1	40.2
APM	NeurIPS 2024	41.7	51.2	78.3	82.8	40.9	44.9	–	–

Table 1: Quantitative comparison between the proposed RI-SAM3 and existing methods on four CD-FSS benchmarks in mIoU (%). Results are taken from the original papers whenever available. Methods marked by ∗ are averaged over five random seeds. The best result within each method group is highlighted in bold .

Incorporated design	ISIC	Chest X-Ray	DeepGlobe	SUIM
SAM3 $+$ category-name	10.2	24.4	21.5	16.1
SAM3 $+$ BRI	54.9	89.3	23.8	54.4
$+$ LTC	59.8	89.5	43.9	58.8
$+$ PCS	64.1	90.5	51.8	63.2

Table 2: Ablation studies of proposed components of RI-SAM3 under the 1-shot setting (mIoU (%)). All variants use the same pre-trained SAM3 with frozen parameters.

Figure 3: Qualitative segmentation results (red mask) of RI-SAM3 on samples from four datasets.

Method	ISIC	Chest X-Ray	DeepGlobe	SUIM
PerSAM-F	$20.5\pm 6.3$	$38.7\pm 7.7$	$46.5\pm 10.5$	$10.3\pm 5.1$
VLP-SAM	$34.8\pm 5.0$	$54.7\pm 14.5$	$48.1\pm 16.1$	$16.9\pm 6.3$
RI-SAM3	$\textbf{64.1}\pm\textbf{4.8}$	$\textbf{90.5}\pm\textbf{0.6}$	$\textbf{51.8}\pm\textbf{2.9}$	$\textbf{63.2}\pm\textbf{3.4}$

Table 3: Stability comparison over five random reference selections, reported as mean mIoU (%)

\pm

standard deviation.

ResNet-based Methods
Method	Runtime (min)
Method	Preparation	Inference	Total
PATNet	538	5	543
IFA	541	5	545

Table 4: Efficiency comparison under the 1-shot CD-FSS setting. Preparation refers to the optimization performed before target inference, including source training, target adaptation, or concept acquisition, depending on the method.

Method	In-Domain FSS			One-Shot Part Seg.
Method	Pasc.-5i	COCO-20i	FSS.	PASC.-Part	PACO-Part
PerSAM	43.1	23.0	71.2	32.5	22.5
PerSAM-F	48.5	23.5	75.6	32.9	22.7
Matcher	68.1	52.7	87.0	42.9	34.7
VRP-SAM	71.9	53.9	–	–	–
GF-SAM	72.1	58.7	88.0	44.5	36.3

Table 5: Generalization under the 1-shot setting in mIoU (%). The best results are highlighted in bold .

Few-Shot Semantic Segmentation

Few-shot semantic segmentation labels pixels in new image classes using only a small set of annotated examples.

See all 37 papers

Oct 19, 2026

Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation

Few-shot segmentation (FSS) aims to segment target objects with limited annotated examples. Recent studies have begun to explore large foundation models to improve generalization for this task. In particular, SAM 3 provides strong visual representations and supports text-guided segmentation. However, its performance remains limited in specialized domains, where semantic prompts and visual representations are often poorly aligned. To address this challenge, we propose ViTA-SAM, a framework built upon SAM 3 that integrates visual memory and textual features through cross-modal alignment. Specifically, we construct a visual memory module that encodes support features together with their corresponding masks, providing support-aware visual guidance for query prediction. In parallel, we leverage the text encoder of SAM 3 to extract semantic features as complementary priors. To bridge the gap between semantic and visual representations, we introduce a cross-modal alignment module that conditions textual features on memory-enhanced visual representations, yielding more consistent and visually grounded predictions. In addition, we propose a pseudo-label refinement strategy that updates the memory with predicted query masks, enabling query-adaptive representation learning. Extensive experiments on multiple benchmarks show that ViTA-SAM achieves the best overall performance among the compared methods, outperforming the strongest competing method by over 5% in average mIoU.

Tianqi Wei, Zhi Chen, Xin Yu +2

The University of Queensland, Brisbane, Australia · University of Southern Queensland, Toowoomba, Australia · Adelaide University, Adelaide, Australia

Few-Shot Semantic Segmentation Text-Guided Image Segmentation Vision-Language Alignment

Ask PDF

Apr 7, 2026

Few-Shot Semantic Segmentation Meets SAM3

Few-Shot Semantic Segmentation (FSS) focuses on segmenting novel object categories from only a handful of annotated examples. Most existing approaches rely on extensive episodic training to learn transferable representations, which is both computationally demanding and sensitive to distribution shifts. In this work, we revisit FSS from the perspective of modern vision foundation models and explore the potential of Segment Anything Model 3 (SAM3) as a training-free solution. By repurposing its Promptable Concept Segmentation (PCS) capability, we adopt a simple spatial concatenation strategy that places support and query images into a shared canvas, allowing a fully frozen SAM3 to perform segmentation without any fine-tuning or architectural changes. Experiments on PASCAL-

5^i

and COCO-

20^i

show that this minimal design already achieves state-of-the-art performance, outperforming many heavily engineered methods. Beyond empirical gains, we uncover that negative prompts can be counterproductive in few-shot settings, where they often weaken target representations and lead to prediction collapse despite their intended role in suppressing distractors. These findings suggest that strong cross-image reasoning can emerge from simple spatial formulations, while also highlighting limitations in how current foundation models handle conflicting prompt signals. Code at: https://github.com/WongKinYiu/FSS-SAM3

Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao Wang

National Yang Ming Chiao Tung University · Institute of Information Science, Academia Sinica, Taiwan

Few-Shot Semantic Segmentation Vision Foundation Models Promptable Image Segmentation

Ask PDF

Oct 10, 2026

TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision

The recent Segment Anything Model 3 (SAM3) extends powerful promptable segmentation from spatial cues to concepts, raising the prospect of a single foundation model for universal image segmentation. However, this promise often fails to translate into reliable performance on downstream segmentation tasks. In this work, we empirically identify concept recognition, rather than mask quality, as the primary bottleneck: SAM3 accurately delineates the objects it recognizes yet struggles to robustly determine which concepts are present. Driven by this, we present Text Anchor re-Grounding (\TAG{}), a simple yet effective framework that rectifies the concept recognition of SAM3 on the target domain without dense annotations. Specifically, \TAG{} introduces an Anchor Residual Pathway comprising image-conditioned, low-rank adapters to re-ground text anchors in the target distribution and modulate subsequent text-conditioned interactions. Guided solely by image-level category labels, \TAG{} jointly optimizes binary cross-entropy with Soft Boundary Ranking objectives to calibrate presence confidence and distinguish present concepts from confusable negatives. Building upon this, we devise Confidence-Weighted Distillation to couple rich semantic context of a multimodal large language model with fine-grained spatial priors of SAM3, thereby achieving effective calibration in the absence of manual annotations. Extensive experiments demonstrate that \TAG{} unlocks SAM3's segmentation potential with only 3.1M trainable parameters, achieving impressive performance across diverse scenarios, and notably, even surpassing leading mask-supervised methods on several metrics. Our code will be publicly available at GitHub \faGithub.

Hao Zhu, Heyu Sun, Shuo Jin +2

Promptable Image Segmentation Open-Vocabulary Semantic Segmentation Weakly Supervised Semantic Segmentation

Ask PDF

Abstract

Few-Shot Semantic Segmentation

Explore similar work

Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation

Few-Shot Semantic Segmentation Meets SAM3

TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision