CASE: Concept-Aware Segmentation from Positive and Negative Visual Exemplars

Yi-Jen Tsai; Yen-Yu Lin; Chien-Yao Wang

cs.CVOct 29, 2026

CASE: Concept-Aware Segmentation from Positive and Negative Visual Exemplars

Authors: Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao Wang

Organizations: National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan

Abstract

Few-shot semantic segmentation typically identifies target objects based on positive support examples of a single semantic class. In practice, however, positive cues alone are often underspecified: for instance, "red cars" may be confused with cars of all colors, "vehicles" may include cars that should be excluded, and "material-defined objects" may span multiple categories. Fulfilling precise requests requires positive exemplars to provide inclusion evidence and negative exemplars to act as intent-defining constraints for exclusion and disambiguation. We formulate this setting as Specified Visual Prompt Segmentation (SVPS), where the target concept is specified by both positive and negative visual prompts. In this work, visual exemplars refer to support-image regions specified by bounding-box prompts. To study SVPS, we introduce CASE (Concept-Aware Segmentation from Positive and Negative Visual Exemplars), a benchmark built on COCO and PACO. CASE encompasses six category- and attribute-level task families: exclusion, isolation, union, attribute exclusion, attribute isolation, and attribute union. We evaluate SAM 3 under this setting and find that, despite its powerful positive-only few-shot segmentation capability, frozen SAM 3 does not consistently treat negative visual prompts as exclusion evidence. To address this, we present Concept Prompt Adaptation (CPA), a conservative LoRA-based adaptation baseline that improves positive-negative prompt interpretation on CASE while remaining competitive under positive-only few-shot segmentation. Code is available at https://github.com/WongKinYiu/FSS-CASE.

Figures & tables

Figure 1: Motivation and overview of CASE. Conventional negative prompts are commonly used as local refinement signals for an already specified target. In contrast, Specified Visual Prompt Segmentation (SVPS) treats positive and negative visual exemplars as intent-defining cues. The same query image can therefore induce different target under different prompts. CASE evaluates this setting through six task families, requiring a unified model to infer target intent from positive-negative exemplar.

Task	Target Concept	Positive Exemplars	Negative Exemplars	Challenge
SuperEx	Supercategory excluding one class, e.g., vehicles except cars.	Included classes from the supercategory, e.g., buses and trains.	The excluded class, e.g., cars.	Generalize to a broad group while enforcing explicit exclusion.
FineIso	One fine-grained class within a supercategory, e.g., dogs.	Target class instances, e.g., dogs.	Related distractor classes, e.g., cats or sheep.	Isolate a target class from visually close distractors.
Union	Classes sharing a common function, e.g., furniture for resting.	Functionally related classes, e.g., beds and sofas.	Related classes with different functions, e.g., chairs or tables.	Infer functional commonality across distinct objects.
AttrEx	A category excluding one attribute, e.g., non-red cars.	Instances without the excluded attribute, e.g., black or white cars.	Instances with the excluded attribute, e.g., red cars.	Apply visual negation by fine-grained attribute differences.
AttrIso	A category with one specified attribute, e.g., red cars.	Target attribute instances, e.g., red cars.	Same category with other attributes, e.g., blue or white cars.	Separate attribute- specific targets from general object identity.
AttrUnion	An attribute shared across categories, e.g., wooden objects.	Diverse classes with the attribute, e.g., wooden chairs or bowls.	Similar classes without the attribute, e.g., plastic chairs or bowls.	Group objects by attribute while avoiding category or shape bias.

Table 1: Task taxonomy of the CASE benchmark. Each task specifies a target concept through positive and negative visual exemplars.

Figure 2: Overview of Concept Prompt Adaptation (CPA). A $K$ -shot support set $\mathcal{S}$ and query image $I^{q}$ are composed into a shared canvas $I^{c}$ , with positive and negative support boxes used as visual prompts. CPA augments the original SAM 3 detector with two trainable LoRA modules while keeping the original SAM 3 weights frozen. The final query prediction is obtained from the cropped query outputs $Z^{q}$ and $s^{q}$ . The inset summarizes the two prompting conditions used during training; their formal objectives are defined in Sec. 4.5 .

1-shot
Method	Prompt	SuperEx	FineIso	Union	AttrEx	AttrIso	AttrUnion	Mean
GF-SAM [ 34 ]	Pos Only	24.87	35.61	33.25	16.74	17.44	10.26	23.03
FSSAM [ 32 ]	Pos Only	28.55	38.97	35.89	17.28	15.87	14.83	25.23
SANSA [ 5 ]	Pos Only	26.96	53.65	29.83	15.65	22.48	17.07	27.61
FSS-SAM3 [ 27 ]	Pos Only	28.80	56.23	43.43	20.05	19.83	15.32	30.61
FSS-SAM3 [ 27 ]	Pos+Neg	26.22	20.43	30.52	17.06	6.33	11.17	18.62

Table 2: CASE benchmark results: task-wise and average mIoU across six tasks.

Method	Prompt	1-shot			5-shot
Method	Prompt	Category	Attribute	Mean	Category	Attribute	Mean
GF-SAM [ 34 ]	Pos Only	31.24	14.81	23.03	43.47	21.18	32.32
FSSAM [ 32 ]	Pos Only	34.47	15.99	25.23	45.58	18.75	32.17
SANSA [ 5 ]	Pos Only	36.81	18.40	27.61	44.27	22.52	33.40
FSS-SAM3 [ 27 ]	Pos Only	42.82	18.40	30.61	51.72	24.48	38.10
FSS-SAM3 [ 27 ]	Pos+Neg	25.72	11.52	18.62	46.03	22.26	34.14

Table 3: Aggregated CASE results by category-level and attribute-level concepts.

Method	Type	1-shot					5-shot
Method	Type	$20^{0}$	$20^{1}$	$20^{2}$	$20^{3}$	Mean	$20^{0}$	$20^{1}$	$20^{2}$	$20^{3}$	Mean
PFENet [ 26 ]	Classical FSS	36.5	38.6	34.5	33.8	35.8	36.5	43.3	37.8	38.4	39.0
HSNet [ 17 ]	Classical FSS	36.3	43.1	38.7	38.7	39.2	43.3	51.3	48.2	45.0	46.9
BAM [ 12 ]	Classical FSS	43.4	50.6	47.5	43.4	46.2	49.3	54.2	51.6	49.6	51.2
HDMNet [ 20 ]	Classical FSS	43.8	55.3	51.6	49.4	50.0	50.6	61.6	55.7	56.0	56.0
Matcher [ 14 ]	Foundation-based	52.7	53.5	52.6	52.1	52.7	60.1	62.7	60.9	59.2	60.7

Table 4: Comparison with state-of-the-art FSS methods on COCO-

20^{i}

. CPA is evaluated under the standard positive-only protocol.

Method	CASE			COCO- $20^{i}$
Method	Category	Attribute	Mean	1-shot	5-shot
FSS-SAM3 [ 27 ]	25.7	11.5	18.6	66.6	71.8
CPA w/o Preservation	50.6	25.2	37.9	50.8	66.8
CPA	48.6	23.0	35.8	61.6	74.8

Table 5: Effect of positive-only preservation. CASE is evaluated with positive-negative prompts, while COCO-

20^{i}

is evaluated under the standard FSS protocol.

Group	Method	0 Neg	1 Neg	2 Neg	All Neg
Category	FSS-SAM3 [ 27 ]	37.28	29.82	21.86	12.53
Category	CPA	49.29	59.66	60.27	59.19
Attribute	FSS-SAM3 [ 27 ]	10.04	8.68	5.40	2.74
Attribute	CPA	18.22	24.12	25.49	22.75
Overall	FSS-SAM3 [ 27 ]	23.66	19.25	13.63	7.63
Overall	CPA	33.76	41.89	42.88	40.97

Table 6: Negative prompt sensitivity on CASE-NegStress. We report mIoU (%). The full per-task results are provided in the supplementary material.

Figure 3: Qualitative comparison on CASE. Green/red boxes in the support image indicate positive/negative visual prompts. Yellow contours denote ground-truth query masks, and blue overlays denote model predictions. We compare SAM 3 (FSS-SAM3) and CPA under positive-only and positive-negative prompting.

Task family	Source annotation	Split usage
SuperEx	COCO instance segmentation	train2017 / val2017
FineIso	COCO instance segmentation	train2017 / val2017
Union	COCO instance segmentation	train2017 / val2017
AttrIso	PACO-LVIS object attributes	train / val
AttrUnion	PACO-LVIS object attributes	train / val
AttrEx	PACO-LVIS object attributes	train / val

Table S1 : Data sources used for each CASE task family. Category-level tasks are constructed from COCO, while attribute-level tasks are constructed from PACO-LVIS.

Task	Source	#Train Ep.	#Eval Ep.	#Concepts	Avg. Pos / Neg
SuperEx	COCO	156,000	9,004	78	6.71 / 2.31
FineIso	COCO	140,727	8,662	79	2.43 / 6.17
Union	COCO	50,000	1,806	25	5.04 / 3.96
AttrEx	PACO	12,909	1,528	109	2.09 / 1.04
AttrIso	PACO	66,942	1,100	94	1.04 / 2.09
AttrUnion	PACO	34,812	1,116	21	2.15 / 1.64

Table S2 : Detailed statistics of the CASE benchmark . We report the number of candidate training episodes, fixed evaluation episodes, concept definitions, and average prompts per episode for each task family. The overall average prompt counts are weighted by the number of candidate training episodes.

Parameter	Value
Minimum target foreground area	500 pixels
Training support candidate pool size	300
Maximum training support pool size	100
Maximum target images per concept	2000
Training episode seed	42
Evaluation support sampling seed	42

Table S3 : General construction parameters used in CASE. The overall average prompt counts reported in the main paper are weighted by the number of candidate training episodes.

Task	Intent	Target mask	Pos. prompts	Neg. prompts
SuperEx	All objects in a COCO supercategory except one excluded category.	Union of all instances in the supercategory minus instances of the excluded category.	Instances from non-excluded categories in the same supercategory.	Instances from the excluded category.
FineIso	Only one fine-grained category within a COCO supercategory.	Instances of the target category.	Instances of the target category.	Instances from other categories in the same supercategory.
Union	A fixed high-level category group, such as public transportation or rest furniture.	Union of all instances belonging to the positive categories of the group.	Instances from at least two positive categories when possible.	Instances from fixed distractor categories associated with the group.
AttrIso	Objects in a fixed object group with a specified color or material attribute.	Instances in the target object group that possess the target attribute.	Same-group instances with the target attribute.	Same-group instances with different attributes.
AttrUnion	All object instances with a specified color or material attribute, regardless of category.	All valid object-level instances that possess the target attribute.	Instances from multiple categories sharing the target attribute.	Instances with different attributes from the same attribute family.
AttrEx	Objects in a semantic object group excluding a specified color or material attribute.	Instances in the semantic group minus instances with the excluded attribute.	Instances with non-excluded attributes from the same attribute family.	Instances with the excluded attribute.

Table S4 : Task-specific construction rules in CASE. Pos. and Neg. indicate the positive and negative support prompts, respectively.

Figure S1 : Example SuperEx episode. The intent is to segment all vehicles except buses.

Figure S2 : Example FineIso episode. The intent is to isolate trucks from other vehicles.

Figure S3 : Example Union episode. The intent is to segment electronic displays across related object categories (electronic devices).

Intent	Positive categories	Negative categories
Public Transportation	bus, train	car, motorcycle
Human Powered Mobility	bicycle, skateboard	car, motorcycle
Rest Furniture	bed, couch	chair, dining table
Domestic Pets	dog, cat	cow, horse, zebra
Drink Containers	cup, wine glass, bottle	fork, knife
Electronic Displays	tv, laptop, cell phone	remote, keyboard, mouse, microwave

Table S5 : Representative examples of manually defined Union concepts.

Figure S4 : Example AttrIso episode. The intent is to segment glass containers while excluding containers with other attributes.

Figure S5 : Example AttrUnion episode. The intent is to segment all ceramic objects.

Figure S6 : Example AttrEx episode. The intent is to segment non-rattan containers while excluding rattan containers.

Figure S7 : Shared-canvas layouts used by CPA. Left: the 1-shot setting uses a vertical layout, where the support image occupies the top half of the $1008\times 1008$ canvas and the query image occupies the bottom half. Right: the 5-shot setting uses an L-shaped layout, where three support images are placed in the top row, two support images are placed in the left column, and the query image occupies the bottom-right region.

Component	Setting
Base model	SAM 3
Frozen model notation	$f$
Adapted model notation	$f_{\theta}$
Input canvas size	$1008\times 1008$
1-shot layout	Vertical support-query split
5-shot layout	L-shaped support-query canvas

Table S6 : CPA training and optimization settings.

Shot	Task	Split 0	Split 1	Split 2	Split 3	Avg.
1-shot	FineIso	40.75	37.74	38.25	39.15	38.97
	SuperEx	27.88	28.80	29.19	28.35	28.55
	Union	36.89	34.92	35.96	35.80	35.89
	AttrIso	18.69	13.90	15.04	15.84	15.87
	AttrUnion	15.04	15.25	13.90	15.13	14.83
	AttrEx	17.25	17.16	17.31	17.40	17.28

Table S7 : Split-wise FSSAM results on CASE. FSSAM is evaluated using the publicly released split-specific checkpoints. The main paper reports the average over four splits.

1-shot
Method	Prompt	Category	Attribute	Exclusion	Isolation	Union	Mean
GF-SAM [ 34 ]	Pos Only	31.24	14.81	20.80	26.52	21.75	23.03
FSSAM [ 32 ]	Pos Only	34.47	15.99	22.92	27.42	25.36	25.23
SANSA [ 5 ]	Pos Only	36.81	18.40	21.30	38.07	23.45	27.61
FSS-SAM3 [ 27 ]	Pos Only	42.82	18.40	24.43	38.03	29.38	30.61
FSS-SAM3 [ 27 ]	Pos+Neg	25.72	11.52	21.64	13.38	20.85	18.62

Table S8 : Full aggregated CASE results across concept dimensions. Category and Attribute correspond to the source concept type, while Exclusion, Isolation, and Union correspond to the compositional operation required by the task.

Model	Prompt	mIoU	Precision	Recall	F1
FSS-SAM3	Pos	30.61	64.10	36.21	46.28
CPA (Ours)	Pos	28.48	35.12	58.02	43.75
FSS-SAM3	Pos+Neg	18.62	68.29	20.66	31.72
CPA (Ours)	Pos+Neg	35.83	63.32	55.72	59.28

Table S9 : 1-shot CASE precision–recall analysis. All methods use the same evaluation episodes and

\tau=0.5

Method	CASE FT	Prompt	1-shot	5-shot
SANSA	No	Pos	27.61	33.40
SANSA	Yes	Pos	30.01	36.27
CPA (Ours)	Yes	Pos+Neg	35.83	42.15

Table S10 : In-domain training control on CASE (mIoU).

Prompt Type	1-shot	5-shot
Text only	5.79	5.79
Text + Pos Visual	14.26	26.03
Text + Pos+Neg Visual	13.58	28.05

Table S11 : SAM 3 text controls on SuperEx (mIoU). The two blocks are separate controls and are not row-wise paired.

Criterion	Mean	Within $\pm 1$	Linear $\kappa$
Intent quality	4.22	84.7%	0.453
Positive consistency	4.38	85.3%	0.490
Negative consistency	4.54	86.0%	0.417
Target consistency	4.10	86.7%	0.570

Table S12 : Human validation of CASE. Agreement is computed over 300 paired episode ratings.

Figure S8 : Additional 1-shot qualitative comparisons on category-level CASE tasks. The figure includes two examples from each category-level task family: SuperEx, FineIso, and Union. In the support-prompt column, the upper region shows the 1-shot support image with positive and negative box prompts, while the lower gray region indicates that the query image is omitted from the prompt visualization. Frozen SAM 3 with positive-only prompts often follows visual similarity without resolving the intended exclusion, isolation, or union constraint. Directly adding negative prompts to frozen SAM 3 is unreliable, either having little effect or suppressing valid target regions. CPA with positive-only prompts produces cleaner positive-prompt responses, while CPA with positive-negative prompts better matches the concept jointly specified by positive and negative exemplars.

Figure S9 : Additional 1-shot qualitative comparisons on attribute-level CASE tasks. The figure includes two examples from each attribute-level task family: AttrEx, AttrIso, and AttrUnion. These examples highlight the difficulty of specifying targets by color or material cues rather than object category alone. Frozen SAM 3 tends to rely on category-level or appearance-level similarity and may fail to respect the intended attribute constraint. CPA improves the use of positive and negative attribute exemplars, although attribute-level concepts remain more challenging than category-level concepts due to subtle visual cues and context-dependent material appearance.

Figure S10 : Additional 5-shot qualitative examples on CASE. The figure includes one example from each of the six task families. The support-prompt column follows the L-shaped 5-shot shared-canvas layout, where the five support images occupy the top row and left column, and the lower-right gray region indicates that the query image is omitted from the prompt visualization. Multiple support exemplars provide more evidence about intra-concept variation, but the target concept can remain ambiguous without properly using negative exemplars. CPA uses the expanded support set more effectively under positive-negative prompting.

Figure S11 : Representative failure cases. Yellow contours denote ground-truth masks and blue overlays denote model predictions. Red circles highlight erroneous regions. Top: in attribute-isolation cases, the adapted model may follow object-category priors and expand from the specified attribute target to visually related containers. Middle: attribute annotations can be visually ambiguous; for example, the distinction between brown and non-brown containers may depend on lighting, texture, and annotation convention. Bottom: negative exemplars do not always fully suppress distractors, leading to under-exclusion when excluded categories remain visually salient in the query image.

Failure mode	Description	Typical cause
Attribute ambiguity	The model fails to distinguish visually subtle colors or materials, such as glossy plastic versus glass or dark brown versus black.	PACO attributes can be visually subtle, lighting-dependent, or partially visible.
Over-exclusion	Negative exemplars suppress valid foreground regions together with distractors.	The negative prompt may share shape, texture, or context with the target concept.
Under-exclusion	Distractor instances matching the negative exemplars remain in the prediction.	The model relies on positive similarity and does not fully use negative evidence.
Category-attribute entanglement	The model follows object category more strongly than the specified attribute.	Foundation segmentation models retain object-centric priors from pretraining.
Small or occluded objects	The model misses small, partially visible, or heavily occluded target instances.	Shared-canvas resizing reduces fine details, and support-query matching becomes less reliable.
Ambiguous grouping	The model inconsistently handles broad union concepts or semantic object groups.	Some manually defined groups reflect functional relations that are not always visually apparent.

Table S13 : Summary of common failure modes in CASE.

Promptable Image Segmentation

Promptable image segmentation uses points, boxes, text, or masks to specify objects and generate pixel-level regions.

See all 154 papers

Few-Shot Semantic Segmentation Meets SAM3
Apr 7, 2026Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao WangFew-Shot Semantic Segmentation Vision Foundation Models
National Yang Ming Chiao Tung University · Institute of Information Science, Academia Sinica, Taiwan
Few-Shot Concept Prompt Learning for Segmentation Foundation Models via Visual Grounding
Aug 3, 2026Rahul Venkataramani, Rachana SathishMedical Imaging Foundation Models Promptable Image Segmentation
Advanced Technology Group, GE HealthCare
Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentation
Sep 24, 2026Yijun Hu, Heng Fan, Libo ZhangFew-Shot Semantic Segmentation
UCAS

Apr 7, 2026J/K move · Enter open · S save

Few-Shot Semantic Segmentation Meets SAM3

Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao Wang

National Yang Ming Chiao Tung University · Institute of Information Science, Academia Sinica, Taiwan

Few-Shot Semantic Segmentation Vision Foundation Models Promptable Image Segmentation

Open paper AskPDF arXiv

Few-Shot Semantic Segmentation (FSS) focuses on segmenting novel object categories from only a handful of annotated examples. Most existing approaches rely on extensive episodic training to learn transferable representations, which is both computationally demanding and sensitive to distribution shifts. In this work, we revisit FSS from the perspective of modern vision foundation models and explore the potential of Segment Anything Model 3 (SAM3) as a training-free solution. By repurposing its Promptable Concept Segmentation (PCS) capability, we adopt a simple spatial concatenation strategy that places support and query images into a shared canvas, allowing a fully frozen SAM3 to perform segmentation without any fine-tuning or architectural changes. Experiments on PASCAL-

5^i

and COCO-

20^i

show that this minimal design already achieves state-of-the-art performance, outperforming many heavily engineered methods. Beyond empirical gains, we uncover that negative prompts can be counterproductive in few-shot settings, where they often weaken target representations and lead to prediction collapse despite their intended role in suppressing distractors. These findings suggest that strong cross-image reasoning can emerge from simple spatial formulations, while also highlighting limitations in how current foundation models handle conflicting prompt signals. Code at: https://github.com/WongKinYiu/FSS-SAM3

Abstract

Promptable Image Segmentation

Explore similar work

Few-Shot Semantic Segmentation Meets SAM3

Few-Shot Concept Prompt Learning for Segmentation Foundation Models via Visual Grounding

Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentation