From Visual Matching to Textual Conditioning: Cross-Domain Few-Shot Segmentation via Reference Inversion

Jiahao Nie; Yun Xing; Wenbin An; Qixin Zhang; Yap-Peng Tan; Alex C. Kot; Shijian Lu

cs.CVNov 2, 2026

From Visual Matching to Textual Conditioning: Cross-Domain Few-Shot Segmentation via Reference Inversion

Authors: Jiahao Nie, Yun Xing, Wenbin An, Qixin Zhang, Yap-Peng Tan, Alex C. Kot, Shijian Lu

Organizations: Nanyang Technological University · Xi’an Jiaotong University · VinUniversity · Shenzhen MSU-BIT University

Abstract

Cross-Domain Few-Shot Segmentation (CD-FSS) aims to segment objects from unseen domains using only a few annotated references. Recent Segment Anything Model (SAM)-based methods have advanced CD-FSS, yet their reference-target visual matching is highly sensitive to target appearance variations. SAM3 introduces text-conditioned segmentation, enabling natural-language descriptions to guide segmentation across target images, but directly using the category name as the condition still yields unstable performance under severe domain shifts. We propose Reference Inversion for SAM3 (RI-SAM3) that leverages SAM3's large-scale vision-language pre-training to learn suitable textual tokens for each target category. RI-SAM3 comprises three core designs: (i) Bi-level Reference Inversion that learns complementary tokens before and after the frozen text encoder: the former remain anchored in SAM3's text space while the latter represent the category more flexibly; (ii) Layer-adaptive Textual Conditioning combines these tokens differently as object queries evolve across decoder layers; and (iii) Presence-aware Confidence Supervision that jointly supervises query confidence and target presence for reliable predictions. With SAM3 frozen, RI-SAM3 optimizes around 12K parameters only per category and it efficiently segments each target in a single forward pass with reusable textual conditioning without per-sample reference-target visual matching. Extensive experiments over multiple widely adopted benchmarks show that RI-SAM3 achieves superior performance consistently in CD-FSS tasks, together with strong robustness, efficiency, and generalization.

Figures & tables

Figure 1: Motivation for Reference Inversion for SAM3 (RI-SAM3). Left : Using the same reference image, GF-SAM ( Zhang et al. 2024a ) exhibits substantial IoU variations across different target images, while manually specified text prompts produce low and unstable results under severe domain shifts. Our proposed RI-SAM3 instead inverts annotated references into a reusable textual condition for inference targets. Right : RI-SAM3 achieves higher mIoU (

\uparrow

) and a lower standard deviation of per-image IoU (

\downarrow

) than GF-SAM and the category-name prompt baseline, demonstrating improved accuracy and stability.

Figure 2: Overview of our proposed RI-SAM3 framework. Bi-level Reference Inversion (BRI) learns reference word tokens $z_{w}^{*}$ and reference concept tokens $z_{c}^{*}$ from annotated references, collectively termed inverted reference tokens. Layer-adaptive Textual Conditioning (LTC) combines them with layer-specific fusion coefficients $\alpha_{*}$ to produce the bi-level reference tokens $T_{c}^{*}$ for conditioning frozen SAM3. Presence-aware Confidence Supervision (PCS) jointly supervises query-level classification and global presence scores. Once learned, $T_{c}^{*}$ is reused for target segmentation without references or visual matching.

Conventional ResNet-/ViT-based Methods
Method	Venue	ISIC		Chest X-Ray		DeepGlobe		SUIM
Method	Venue	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
ISA	ICCV 2025	37.2	56.1	83.4	86.3	44.3	52.7	44.1	–
DR-Adapter	CVPR 2024	40.8	48.9	82.4	82.3	41.3	50.1	–	–
PATNet	ECCV 2022	41.2	53.6	66.6	70.2	37.9	43.0	32.1	40.2
APM	NeurIPS 2024	41.7	51.2	78.3	82.8	40.9	44.9	–	–

Table 1: Quantitative comparison between the proposed RI-SAM3 and existing methods on four CD-FSS benchmarks in mIoU (%). Results are taken from the original papers whenever available. Methods marked by ∗ are averaged over five random seeds. The best result within each method group is highlighted in bold .

Incorporated design	ISIC	Chest X-Ray	DeepGlobe	SUIM
SAM3 $+$ category-name	10.2	24.4	21.5	16.1
SAM3 $+$ BRI	54.9	89.3	23.8	54.4
$+$ LTC	59.8	89.5	43.9	58.8
$+$ PCS	64.1	90.5	51.8	63.2

Table 2: Ablation studies of proposed components of RI-SAM3 under the 1-shot setting (mIoU (%)). All variants use the same pre-trained SAM3 with frozen parameters.

Figure 3: Qualitative segmentation results (red mask) of RI-SAM3 on samples from four datasets.

Method	ISIC	Chest X-Ray	DeepGlobe	SUIM
PerSAM-F	$20.5\pm 6.3$	$38.7\pm 7.7$	$46.5\pm 10.5$	$10.3\pm 5.1$
VLP-SAM	$34.8\pm 5.0$	$54.7\pm 14.5$	$48.1\pm 16.1$	$16.9\pm 6.3$
RI-SAM3	$\textbf{64.1}\pm\textbf{4.8}$	$\textbf{90.5}\pm\textbf{0.6}$	$\textbf{51.8}\pm\textbf{2.9}$	$\textbf{63.2}\pm\textbf{3.4}$

Table 3: Stability comparison over five random reference selections, reported as mean mIoU (%)

\pm

standard deviation.

ResNet-based Methods
Method	Runtime (min)
Method	Preparation	Inference	Total
PATNet	538	5	543
IFA	541	5	545

Table 4: Efficiency comparison under the 1-shot CD-FSS setting. Preparation refers to the optimization performed before target inference, including source training, target adaptation, or concept acquisition, depending on the method.

Method	In-Domain FSS			One-Shot Part Seg.
Method	Pasc.-5i	COCO-20i	FSS.	PASC.-Part	PACO-Part
PerSAM	43.1	23.0	71.2	32.5	22.5
PerSAM-F	48.5	23.5	75.6	32.9	22.7
Matcher	68.1	52.7	87.0	42.9	34.7
VRP-SAM	71.9	53.9	–	–	–
GF-SAM	72.1	58.7	88.0	44.5	36.3

Table 5: Generalization under the 1-shot setting in mIoU (%). The best results are highlighted in bold .

Few-Shot Semantic Segmentation

Few-shot semantic segmentation labels pixels in new image classes using only a small set of annotated examples.

See all 37 papers

Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation
Oct 19, 2026Tianqi Wei, Zhi Chen, Xin Yu +2Few-Shot Semantic Segmentation Text-Guided Image Segmentation
The University of Queensland, Brisbane, Australia · University of Southern Queensland, Toowoomba, Australia · Adelaide University, Adelaide, Australia
Few-Shot Semantic Segmentation Meets SAM3
Apr 7, 2026Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao WangFew-Shot Semantic Segmentation Vision Foundation Models
National Yang Ming Chiao Tung University · Institute of Information Science, Academia Sinica, Taiwan
TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision
Oct 10, 2026Hao Zhu, Heyu Sun, Shuo Jin +2Promptable Image Segmentation Open-Vocabulary Semantic Segmentation

Oct 19, 2026J/K move · Enter open · S save

Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation

Tianqi Wei, Zhi Chen, Xin Yu, Scott Chapman+1

The University of Queensland, Brisbane, Australia · University of Southern Queensland, Toowoomba, Australia · Adelaide University, Adelaide, Australia

Few-Shot Semantic Segmentation Text-Guided Image Segmentation Vision-Language Alignment

Open paper AskPDF arXiv

Few-shot segmentation (FSS) aims to segment target objects with limited annotated examples. Recent studies have begun to explore large foundation models to improve generalization for this task. In particular, SAM 3 provides strong visual representations and supports text-guided segmentation. However, its performance remains limited in specialized domains, where semantic prompts and visual representations are often poorly aligned. To address this challenge, we propose ViTA-SAM, a framework built upon SAM 3 that integrates visual memory and textual features through cross-modal alignment. Specifically, we construct a visual memory module that encodes support features together with their corresponding masks, providing support-aware visual guidance for query prediction. In parallel, we leverage the text encoder of SAM 3 to extract semantic features as complementary priors. To bridge the gap between semantic and visual representations, we introduce a cross-modal alignment module that conditions textual features on memory-enhanced visual representations, yielding more consistent and visually grounded predictions. In addition, we propose a pseudo-label refinement strategy that updates the memory with predicted query masks, enabling query-adaptive representation learning. Extensive experiments on multiple benchmarks show that ViTA-SAM achieves the best overall performance among the compared methods, outperforming the strongest competing method by over 5% in average mIoU.

Abstract

Few-Shot Semantic Segmentation

Explore similar work

Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation

Few-Shot Semantic Segmentation Meets SAM3

TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision