TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision

Hao Zhu; Heyu Sun; Shuo Jin; Jiayu Xiao; Feng Dai

cs.CVOct 10, 2026

TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision

Authors: Hao Zhu, Heyu Sun, Shuo Jin, Jiayu Xiao, Feng Dai

Abstract

The recent Segment Anything Model 3 (SAM3) extends powerful promptable segmentation from spatial cues to concepts, raising the prospect of a single foundation model for universal image segmentation. However, this promise often fails to translate into reliable performance on downstream segmentation tasks. In this work, we empirically identify concept recognition, rather than mask quality, as the primary bottleneck: SAM3 accurately delineates the objects it recognizes yet struggles to robustly determine which concepts are present. Driven by this, we present Text Anchor re-Grounding (\TAG{}), a simple yet effective framework that rectifies the concept recognition of SAM3 on the target domain without dense annotations. Specifically, \TAG{} introduces an Anchor Residual Pathway comprising image-conditioned, low-rank adapters to re-ground text anchors in the target distribution and modulate subsequent text-conditioned interactions. Guided solely by image-level category labels, \TAG{} jointly optimizes binary cross-entropy with Soft Boundary Ranking objectives to calibrate presence confidence and distinguish present concepts from confusable negatives. Building upon this, we devise Confidence-Weighted Distillation to couple rich semantic context of a multimodal large language model with fine-grained spatial priors of SAM3, thereby achieving effective calibration in the absence of manual annotations. Extensive experiments demonstrate that \TAG{} unlocks SAM3's segmentation potential with only 3.1M trainable parameters, achieving impressive performance across diverse scenarios, and notably, even surpassing leading mask-supervised methods on several metrics. Our code will be publicly available at GitHub \faGithub.

Figures & tables

Figure 1: TAG calibrates SAM3 with image-level supervision. TAG learns an ARP with only 3.1M parameters using image-level presence objectives while keeping SAM3 fully frozen. The examples illustrate how TAG corrects the inter-class confusion and under-activation of SAM3. Right panel reports the gains achieved with image-level ground-truth labels across different scenarios.

Figure 2: Recognition errors in frozen SAM3. (a) SAM3 exhibits strong foreground perception, yet its concept recognition often fails. (b) Oracle interventions: Frozen SAM3 gives original predictions. GT mask regions pairs GT mask regions with SAM3’s original category scores. Positive gates keeps raw SAM3 masks, setting only GT categories gate to $1$ for instance filtering. GT categories pairs SAM3 masks with GT categories. Gates & Categories combines both semantic corrections.

Figure 3: Overview of our proposed TAG . Frozen SAM3 encodes the image and class prompts, while trainable adapters refine text features, cross-modal attention, and presence prediction using image-level supervision. Here, and denote the object queries and presence token, respectively.


				ADE20K			Cityscapes			iSAID
Method	Sup.	Backbone	#Par.	PQ	AP	mIoU	PQ	AP	mIoU	PQ	AP	mIoU
Fully supervised universal segmentation
Mask2Former [CVPR22]	$\mathcal{M}$	Swin-L	216M	48.1	34.9	56.1	66.6	43.7	83.3	58.1	39.5	68.5
MaskDINO [CVPR23]	$\mathcal{M}$	R50	52M	–	–	48.7	–	–	80.0	57.4	36.8	64.6
OneFormer [CVPR23]	$\mathcal{M}$	DiNAT-L	223M	50.5	36.0	58.3	67.6	45.6	83.1	51.3	31.7	62.9

Table 1: Universal segmentation comparison on three validation benchmarks. Sup.:

\mathcal{M}

target masks,

\widetilde{\mathcal{M}}

source masks,

\mathcal{I}

image labels,

\mathcal{P}

points,

\mathcal{V}

MLLM labels; #Par.: trainable parameters.

\uparrow

: gains over frozen SAM3. See Appendix B for more details.

Figure 4: Transferability of TAG . (a) Text residuals trained on ADE20K and generalized to shared VOC21 classes. (b) Class-agnostic adapters trained on ADE-150 and transferred to new classes.

Table 6

Figure 5: Qualitative comparison results across natural images, urban scenes and aerial imagery.

Figure 6: Adaptation strategies on ADE20K. PQ comparison of TAG-I , LoRA, and residual variants.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

		Direct transfer			After SIOR adaptation
Method	Backbone	PQ	AP	mIoU	PQ	AP	mIoU
X-Decoder ( Zou et al., 2023a )	DaViT-L	1.32	0.33	4.78	28.09	17.61	37.10
HIPIE ( Wang et al., 2023b )	ViT-H	2.28	0.57	5.53	28.56	13.17	9.53
APE ( Shen et al., 2024 )	EVA-02 ViT-L	6.05	3.09	7.62	32.86	23.54	37.36
OpenWorldSAM ( Xiao et al., 2025 )	SAM2-L + BEiT-3	2.22	0.04	3.56	21.6	10.5	22.3
FrozenSeg ( Chen et al., 2024b )	ConvNeXt-L + SAM	3.26	2.97	8.08	33.54	19.18	32.53

Appendix

Table 4: Open-vocabulary transfer to iSAID before and after SIOR adaptation. Direct transfer uses models pretrained on natural-image data; adaptation uses SIOR masks, without iSAID training masks. Both settings evaluate the full iSAID vocabulary on 11,644 validation tiles; per-image GT class lists are never supplied. The adapted scores correspond to the iSAID open-vocabulary entries in Table 1 . AP denotes mask AP. Bold marks the best score within each setting.

Setting	Shared value
Learning rate	$5{\times}10^{-3}$
Training epochs	10
Presence weight $\lambda_{\mathrm{pres}}$	4
Ranking weight $\lambda_{\mathrm{rank}}$	0.5
Hard negatives $m$	3
Adapter rank $r$	16

Appendix

Table 5: Shared training hyperparameters across datasets. The same configuration is used without dataset-specific tuning. The loss weights follow Eq. 8 , with CWD replacing BCE for TAG-V . Epochs denote the scheduled training budget.

Module	Residual input	Conditioning feature	Zero-initialized component
Text low-rank branch	$\mathbf{F}_{t}^{k}$	$\bar{\mathbf{e}}_{k}$	output projection $\mathbf{B}$
Text image branch	$\mathbf{F}_{t}^{k},\mathbf{F}_{v}$	image features and text affinity	$\operatorname{Proj}_{\mathrm{out}}$ weight/bias
Presence token	$\bar{\mathbf{e}}^{\prime}_{k}$	$\bar{\mathbf{e}}^{\prime}_{k}$	final layer of $\mathrm{MLP}_{\mathrm{p}}$
Fusion text K/V	$\mathbf{F}_{t}^{\prime k}(I)$	$\bar{\mathbf{e}}^{\prime}_{k}$	both low-rank output projections
Decoder image K/V	$\mathbf{F}_{v}^{k}$	$\bar{\mathbf{e}}^{\prime}_{k}$	both low-rank output projections

Appendix

Table 6: ARP inputs and zero initialization. Each residual starts at zero through its output projection. K/V updates are applied before splitting features into attention heads.

Figure 7: Presence correction on ADE20K. PQ of score calibration, presence classifiers, and TAG-I ; the axis starts at 25.

Method	ADE	City	iSAID	Avg.	$\Delta$
w/o text low-rank	43.51	59.73	41.39	48.21	$-3.05$
w/o image conditioning	44.19	61.02	42.60	49.27	$-1.99$
TAG-I	46.86	62.67	44.24	51.26	–

Appendix

Table 7: Complementary text corrections. Branch ablations of TAG-I ; PQ (%).

Figure 8: MLLM supervision on ADE20K. Naive BCE versus CWD; PQ (%). Green arrows show CWD gains; spans are the ranges across MLLMs.

Figure 9: Sensitivity to objective weights. Rows vary (a) the presence weight and (b) the ranking weight while fixing all other settings to Table 5 . Columns show ADE20K, Cityscapes, and iSAID on separate absolute-PQ axes, each spanning 3.5 points. Axis limits are fixed for each dataset across all sweeps. Colored subtitles report the common setting and its PQ; bold x-axis ticks identify that setting.

Figure 10: Sensitivity to hard-negative mining and adapter capacity. Rows show one-factor sweeps of (c) hard-negative count

m

, (d) adapter rank

r

, and (e) shared presence directions

G

. The common defaults are

m=3

r=16

, and

G=16

, with

\lambda_{\mathrm{pres}}=4

and

\lambda_{\mathrm{rank}}=0.5

fixed. Dataset columns, colors, default annotations, and the 3.5-point axis spans follow Figure 9 .

VOC 2012 (1,449 images)
Method	mIoU	PQ	AP	AP 50
Frozen SAM3	83.49	81.38	57.34	80.92
TAG-I (Ours)	89.46	85.94	59.58	83.61
COCO-Object 2017 (5,000 images)
Frozen SAM3	63.49	57.62	43.85	68.26
TAG-I (Ours)	67.58	61.22	46.56	71.03

Appendix

Table 8: SAM3 and TAG-I results on VOC and COCO-Object. AP denotes mask average precision over IoU thresholds

0.50{:}0.05{:}0.95

; AP 50 uses IoU

0.50

. VOC PQ uses 20 thing classes plus background. COCO PQ uses 81 classes , merging all stuff into background; Table 10 reports the 133-class evaluation. Bold marks the higher score within each protocol.

Method	Prior	Backbone	VOC 2012 val	COCO 2014 val	COCO 2017 val
WeCLIP ( Zhang et al., 2024 )	CLIP	ViT-B/16	76.4	47.1	–
S2C ( Kweon & Yoon, 2024 )	SAM	WRN-38	78.2	49.8	–
Yang et al. ( Yang & Gong, 2024 )	CLIP + SAM	R101	77.3	48.6	–
Yang et al. ( Yang & Gong, 2024 )	CLIP + SAM	Swin-L	82.6	55.4	–
Frozen SAM3	SAM3	SAM3	83.49	–	63.49
TAG-I (Ours)	SAM3	SAM3	89.46	–	67.58

Appendix

Table 9: Image-level semantic segmentation comparisons (mIoU). Prior denotes the foundation model used by the pipeline; backbone denotes the segmentation backbone. All adapted methods use image-level target labels. The COCO-Object columns contain 80 foreground classes plus background; COCO 2014 and 2017 are reported separately by evaluation split. VOC training-set sizes are specified in the text. Bold marks the highest score in each column.

Image-level panoptic methods
				Panoptic PQ		Instance precision
Method	Sup.	Prior	Backbone	VOC	COCO-133	VOC AP 50	COCO AP
JTSM ( Shen et al., 2021 )	$\mathcal{I}$	–	R18-WS	39.0	5.3	44.2	6.1
DHR+TRACE ( Jo et al., 2026 )	$\mathcal{I}$	SD	R50	56.9	32.8	–	–
DHR+TRACE ( Jo et al., 2026 )	$\mathcal{I}$	SD	Swin-L	69.8	43.1	–	–
Point-supervised panoptic methods

Appendix

Table 10: Weakly supervised panoptic and instance segmentation. PQ follows VOC-panoptic (20 things plus background) and COCO-panoptic 2017 (133 classes). Instance columns report VOC AP 50 and COCO 2017 mask AP@[.5:.95]. Sup. follows Table 1 ; prior denotes the foundation model used. A dash means not reported. Our PQ results use complete validation sets and the same inference procedure for frozen and adapted SAM3. Published methods retain their original training and evaluation settings. Bold marks the highest score in each metric column.


Method	Target labels	ADE20K	Cityscapes	iSAID	PC59
ConceptBank	GT masks	43.3	75.1	35.4	63.0
CoCo-SAM3	None	38.3	70.7	–	61.2
ActiveSAM	None	40.0	69.7	$28.3^{\ast}$	60.1
ProC-SAM3	None	–	–	29.5	–
TAG-V (Ours)	MLLM labels	47.0 $\uparrow$ 3.7	78.1 $\uparrow$ 3.0	57.4 $\uparrow$ 22.0	65.2 $\uparrow$ 2.2

Appendix

Table 11: Comparison with SAM3 calibration methods (mIoU, %). PC59: Pascal Context-59. Target labels specify the supervision used by each method; ConceptBank uses training-set GT masks. Published results retain their original inference protocols. ∗ : ActiveSAM reproduced by Song et al. (2026) ; –: not reported. Bold and underline mark the best and second-best scores in each column.

\uparrow

gives each TAG variant’s gain in mIoU points over the best non- TAG result in that column.

Dataset	Class name	Nearest alternative phrase
Cityscapes	train †	tram
	rider †	rider in the foreground
ADE20K	table †	table furniture
	cushion †	pillow
iSAID	baseball diamond	baseball diamond next to other objects
	ground track field	ground track field in bright daylight

Appendix

Table 12: Nearest alternative phrases. Exact names excluded;

\dagger

: original name is top-1.

Figure 11: Segmentation with names, retrieved phrases, and TAG-I . (a) Mean target IoU; whiskers are 95% intervals from 2,000 image-bootstrap resamples. (b) Cushion IoU when exact ties are reassigned to the existing pillow class (hatched), with model scores fixed.

Dataset	Images / tiles	GT objects	Failed objects
ADE20K	2,000	40,740	16,281
Cityscapes	500	46,490	19,225
iSAID	11,644	229,354	130,650

Appendix

Table 13: Evaluation populations for the object-level spatial statistics. iSAID observations are validation tiles. Failed objects have correct-class pixel recall below

0.5

Figure 12: Object-level structure of frozen SAM3’s recognition errors. Failed objects are GT foreground connected components with correct-class pixel recall below 0.5. (a) Mean fraction of mislabeled pixels within each failed object. (b) Fraction of each failed object covered by its most frequent predicted label, including background where applicable. Boxes and whiskers denote the interquartile range and 5th–95th percentiles; dashed lines mark medians, and the iSAID box collapses at 100%. Numerical labels give means in (a) and medians in (b). Each failed object contributes equally to both summaries.

Figure 13: Spatial quality of recognized objects. Median boundary F1 at

5

px tolerance and mask IoU (%).

Transferred components	Mapped 150	Additional 697
None (frozen SAM3)	30.52	11.39
Text correction only	45.18	12.68
Downstream correction only	30.88	12.28
Text + downstream correction	45.45	13.32
Change from frozen SAM3	$+14.93$	$+1.93$

Appendix

Table 14: ADE150-to-ADE847 vocabulary expansion (mIoU, %). All conditions evaluate the complete validation set with 847 candidate labels. Columns partition the labels by correspondence to the ADE150 training vocabulary. Bold marks the highest score in each column. The last row gives the combined correction’s change from frozen SAM3 in mIoU points.

Figure 14: Additional panoptic results on Cityscapes. Columns show the image, ground truth, frozen SAM3, TAG-V , and TAG-I . TAG improves person/rider and bus/train discrimination and recovers vegetation regions missed by frozen SAM3. All columns use the same crop and category palette.

Figure 15: Additional panoptic results on ADE20K. TAG improves recognition of cushions and furniture categories and recovers regions missed by frozen SAM3, including cradles and kitchen islands. The indoor and outdoor scenes illustrate gains in category consistency and region coverage.

Figure 16: Additional panoptic results on iSAID. TAG improves court/field discrimination, recovers storage tanks and athletics facilities, and reduces foreground leakage around piers. The last two rows show predictions for crowded boats and small vehicles.

Promptable Image Segmentation

Promptable image segmentation uses points, boxes, text, or masks to specify objects and generate pixel-level regions.

See all 149 papers

Aug 6, 2026

Universal Concept Disruption for SAM3 Image Segmentation

SAM3 extends promptable segmentation from geometry-driven mask prediction to open-vocabulary concept segmentation, where a text-conditioned grounding model decides whether a concept is present and segments all matching instances. While this presence-gated design improves concept-level prediction, its adversarial robustness remains unexplored. In this paper, we introduce Universal Concept Disruption (UCD), the first universal cross-concept adversarial attack tailored to SAM3 image segmentation. UCD learns a single bounded image perturbation from (image, noun-phrase) pairs and attacks SAM3 as an integrated concept-grounding system. It jointly disrupts the text-conditioned input path, maximizes divergence in prompt-shared visual features, suppresses the final presence-gated concept scores, and corrupts the spatial validity of retained masks through area collapse and clean-mask Dice disruption. Across SACo-Gold, LVIS, RefCOCO, PhraseCut, and OpenImages datasets, UCD consistently outperforms all baselines under a matched evaluation protocol, reducing average mask AP from 59.43 to 18.73 and average cgF1 from 50.32 to 20.49. The learned perturbation also transfers to SAM3.1 and to SAM3 video inference without re-optimization, while prompt ensembling, lightweight head fine-tuning, and temporal filtering provide limited recovery.

Hao Wang, Yuxuan Zhang, Wei Yang

School of Computer Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China +1

Promptable Image Segmentation Adversarial Robustness Adversarial Perturbations Open-Vocabulary Semantic Segmentation

Ask PDF

Apr 20, 2026

Is SAM3 ready for pathology segmentation?

Is Segment Anything Model 3 (SAM3) capable in segmenting Any Pathology Images? Digital pathology segmentation spans tissue-level and nuclei-level scales, where traditional methods often suffer from high annotation costs and poor generalization. SAM3 introduces Promptable Concept Segmentation, offering a potential automated interface via text prompts. With this work, we propose a systematic evaluation protocol to explore the capability space of SAM3 in a structured manner. Specifically, we evaluate SAM3 under different supervision settings including zero-shot, few-shot, and supervised with varying prompting strategies. Our extensive evaluation on pathological datasets including NuInsSeg, PanNuke and GlaS, reveals that: (1) text-only prompts poorly activate nuclear concepts; (2) performance is highly sensitive to visual prompt types and budgets; (3) few-shot learning offers gains, but SAM3 lacks robustness against visual prompt noise; and (4) a significant gap persists between prompt-based usage and task-trained adapter-based reference. Our study delineates SAM3's boundaries in pathology image segmentation and provides practical guidance on the necessity of pathology domain adaptation.

Qiuyu Kong, Shakiba Sharifi, Yiming Wang +2

Sapienza University of Rome · University of Verona · Fondazione Bruno Kessler

Computational Pathology Image Segmentation Histopathology Image Segmentation Promptable Image Segmentation Medical Image Segmentation Few-Shot Medical Image Segmentation

Ask PDF

Nov 17, 2025

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

The Segment Anything Model (SAM) family has become a widely adopted vision foundation model, but its ability to control segmentation granularity remains limited. Users often need to refine results manually - by adding more prompts or selecting from pre-generated masks - to achieve the desired level of detail. This process can be ambiguous, as the same prompt may correspond to several plausible masks, and collecting dense annotations across all granularities is prohibitively expensive, making supervised solutions infeasible. To address this limitation, we introduce UnSAMv2, which can segment anything at any granularity without using any human annotations. UnSAMv2 extends the divide-and-conquer strategy of UnSAM by discovering abundant mask-granularity pairs and introducing a novel granularity control embedding that enables precise, continuous control over segmentation scale. Remarkably, with only

6

K unlabeled images and

0.02\%

additional parameters, UnSAMv2 substantially enhances SAM-2, enabling segmentation of anything at any granularity across interactive, whole-image, and video segmentation tasks. Furthermore, we introduce a language-guided segmentation pipeline that treats granularity as a predictable variable by grounding it in text semantics via an external vision-language model, improving upon the previous state of the art in 1-IoU (40.4

\rightarrow

47.3). Across 11+ benchmarks, UnSAMv2 achieves consistent gains over SAM/SAM-2 baselines, improving

\text{NoC}_{90}

(5.69

\rightarrow

4.75), 1-IoU (58.0

\rightarrow

73.1), and

\text{AR}_{1000}

(49.6

\rightarrow

68.3) by a large margin. Our work demonstrates that a small amount of unlabeled data can deliver large, practical improvements to strong supervised baselines. Our code, dataset, and demo are available at https://github.com/yujunwei04/UnSAMv2.

Junwei Yu, Trevor Darrell, XuDong Wang

UC Berkeley

Image Segmentation Promptable Image Segmentation Unsupervised Image Segmentation Video Object Segmentation

Ask PDF

Abstract

Promptable Image Segmentation

Explore similar work

Universal Concept Disruption for SAM3 Image Segmentation

Is SAM3 ready for pathology segmentation?

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity