PTQ4SAM3: Semantics-Preserving Post-Training Quantization for Segment Anything Model 3
Authors: Xi Xiao, Yunbei Zhang, Lin Zhao, Janet Wang, Xinru Jiang, Yanshu Li, Chenrui Ma, Pan Wang, Tianyang Wang, Xiao Wang
Organizations: University of Alabama at Birmingham, USA · Tulane University, USA · Northeastern University, USA · University of Texas at Austin, USA · University of Virginia, USA · University of Pittsburgh, USA · Oak Ridge National Laboratory, USA
Segment Anything Model 3 (SAM 3) can segment objects from text prompts and track them across video, but its memory and computation costs make deployment difficult. Post-training quantization (PTQ) reduces these costs by using fewer bits. However, a quantized model can select the wrong object or lose track of its target across frames. We call this prompt-alignment drift. Controlled tests show that perturbing low-variance feature directions strongly affects target selection. We propose Semantics-Preserving Quantization (SPQ), which gives more calibration weight to concept-sensitive parameters, preserves residual features, and corrects the mean error in video memory. Fixed text concepts guide calibration without task annotations. All three components operate offline and add no inference operators. With a W4A8 image encoder, SPQ improves RefCOCO [email protected] from 42.1 to 72.5 over PTQ4SAM while retaining 54.8 COCO mask AP. Experiments on LVIS, RefCOCO variants, and SA-V show consistent gains. A CLIP study also shows improved image-text retrieval.
Figures & tables
Figure 2: Overview of SPQ. Designs A and B calibrate image blocks with a fixed concept codebook and residual-basis feature matching. Design C estimates the mean memory error from paired video rollouts and folds the correction into an existing affine map. SPQ performs these steps offline and preserves the exported graph.
Bit-width
Method
COCO Val
LVIS v1.0 Val
AP ↑
AP 50 ↑
AP 75 ↑
AP ↑
AP r ↑
AP c ↑
AP f ↑
FP16
Full Precision (Baseline)
55.7
76.2
61.4
48.5
41.2
47.5
52.6
W8A8
MinMax
53.8
74.5
59.3
45.5
36.2
45.0
50.5
PTQ4ViT [ 55 ]
54.6 (+0.8)
75.4 (+0.9)
60.2 (+0.9)
46.8 (+1.3)
38.1 (+1.9)
46.1 (+1.1)
51.5
PTQ4SAM [ 30 ]
55.2 (+1.4)
75.8 (+1.3)
60.9 (+1.6)
47.9 (+2.4)
39.6 (+3.4)
46.9 (+1.9)
52.0
SPQ (Ours)
55.6 (+1.8)
76.1 (+1.6)
61.2 (+1.9)
48.4 (+2.9)
40.5 (+4.3)
47.3 (+2.3)
52.4
Table 1: Structural Integrity on COCO and LVIS. We report mask Average Precision (AP). The FP16 entries follow the official SAM 3 reported metrics. PTQ4SAM and SPQ both preserve much of the structural segmentation accuracy under low-bit quantization.
Table 2: Zero-shot referring segmentation. Reconstruction PTQ loses substantially more prompt-grounding accuracy than structural AP.
Figure 3: Qualitative comparison under W4A8. Top: instance-segmentation and text-prompted segmentation cases, with each text prompt shown in its input image. Bottom: additional prompt-labeled cases, including “orange” and “zebra,” that isolate prompt-conditioned target selection. SPQ selects the intended objects and returns more coherent masks than PTQ4SAM in the shown examples.
Figure 4: Temporal degradation and prompt-alignment diagnostics. (a) SA-V long-horizon tracking mIoU: PTQ4SAM under W4A8 decays over time, while SPQ remains close to FP16. (b) Angle between the estimated FP16 and quantized residual bases across transformer blocks: standard PTQ shows a larger angle, while SPQ shows a smaller one. (c) Vision-language alignment across blocks: PTQ4SAM drops with depth, while SPQ better preserves alignment. Shaded regions denote sample variability.
Bit-width
Method
Progressive Frame mIoU
Video J&F ↑
Consistency Score ( Tcs ) ↑
Frame 1 ↑
Frame 50 ↑
Frame 100 ↑
FP16
Full Precision (Baseline)
85.0
83.5
82.2
83.5
96.7%
W8A8
MinMax
80.1
72.5
63.8
71.5
79.6%
PTQ4ViT [ 55 ]
82.2 (+2.1)
76.5 (+4.0)
69.8 (+6.0)
76.2 (+4.7)
84.9%
PTQ4SAM [ 30 ]
83.6 (+3.5)
78.8 (+6.3)
72.5 (+8.7)
78.5 (+7.0)
86.7%
SPQ (Ours)
84.2 (+4.1)
83.1 (+10.6)
82.2 (+18.4)
83.2 (+11.7)
97.6%
Table 3: Temporal stability on SA-V. Progressive mIoU, J&F, and consistency under the shared 32-clip calibration protocol.
Table S1 : Sensitivity Analysis of α and λ on RefCOCO. The combination α=0.1 and λ=1.0 achieves the highest grounding accuracy among these settings.
Basis
k/D
AP ↑
Ref ↑
PCA residual
0.25
54.4
71.8
PCA residual
0.50
54.8
72.5
PCA residual
0.75
54.6
71.9
Text-gradient, equal-rank
0.50
54.7
72.8
Appendix
Table S2 : Rank and basis sensitivity under W4A8. AP is COCO mask AP and Ref is RefCOCO [email protected]. The text-gradient basis stacks and orthogonalizes held-out gradients of normalized visual–text concept scores and retains the same rank as the default PCA basis.
Intervention
Δ AP
Δ Ref
Ures , perturb
−1.3
−22.4
Udom , perturb
−8.7
−4.6
Random, rank-matched
−3.0
−7.2
Remove Ures
−2.1
−26.8
Remove Udom
−10.2
−5.1
Appendix
Table S3 : Matched functional interventions. Each Δ is relative to the same unperturbed quantized checkpoint. Perturbations use matched rank and norm; Ref denotes RefCOCO [email protected].
Table S6 : Image-calibration size under W4A8. “A/B images” is the only quantity varied; Design C always uses the same 32 video clips.
Method
Bit-width
Flickr30k (Zero-shot)
MSCOCO (Zero-shot)
I2T R@1 ↑
I2T R@5 ↑
T2I R@1 ↑
T2I R@5 ↑
I2T R@1 ↑
I2T R@5 ↑
T2I R@1 ↑
T2I R@5 ↑
FP16
Baseline
81.2
96.5
62.4
85.1
58.2
82.4
38.5
64.2
MinMax
W4A8
42.5
68.2
31.2
55.4
28.5
54.1
18.2
35.6
PTQ4ViT
W4A8
58.4 (+15.9)
81.5 (+13.3)
44.8 (+13.6)
68.2 (+12.8)
41.2 (+12.7)
65.5 (+11.4)
26.4 (+8.2)
45.1 (+9.5)
SPQ (Ours)
W4A8
78.5 (+36.0)
94.2 (+26.0)
59.8 (+28.6)
82.4 (+27.0)
55.8 (+27.3)
79.5 (+25.4)
36.7 (+18.5)
61.8 (+26.2)
Appendix
Table S7 : Generalization to CLIP (ViT-B/16) on Flickr30k and MSCOCO. We evaluate zero-shot image–text retrieval under W4A8. SPQ recovers retrieval accuracy on both datasets, extending alignment-aware calibration to a separate vision–language model.
Method
Bit-width
Model Size
Peak Memory
Latency
Throughput
(Weight / Act. / Memory)
(GB) ↓
(GB) ↓
(ms / frame) ↓
(FPS) ↑
Full Precision
FP16 / FP16 / FP16
1.35
14.8
42.5
23.5
PTQ4SAM
INT8 / INT8 / INT8
0.68 ( 2.0× )
8.5
25.4
39.3 ( 1.6× )
SPQ (Ours)
INT8 / INT8 / INT8
0.68 ( 2.0× )
8.5
25.4
39.3 ( 1.6× )
SPQ (Ours)
INT4 / INT8 / INT8
0.38 ( 3.5× )
7.9
18.2
54.9 ( 2.3× )
Appendix
Table S8 : Hardware Efficiency Analysis on an NVIDIA RTX 3090 GPU. Measurements use 1024 × 1024 video frames. SPQ achieves up to 3.5× compression and nearly 2.3× speedup over FP16. At the same bit-width, SPQ preserves the inference cost of standard PTQ.
Segment Anything Models (SAMs) are extensively used in computer vision for universal image segmentation, but deploying them on resource-constrained devices is challenging due to their high computational and memory demands. Post-Training Quantization (PTQ) is a widely used technique for model compression and acceleration. However, existing PTQ methods fail to consider the cross-attention architecture in the SAM decoder. This degradation primarily stems from the unique challenges posed by SAMs: (1) Attention dissipation, where the attention information in the decoder, which is crucial for representing segmentation masks, collapses into a diffuse and non-semantic form under low-bit quantization; and (2) Reconstruction oscillation, where bidirectional coupling within the two-way transformer introduces cross-branch error interference and destabilizes convergence. To tackle these issues, we propose CAR-SAM, a unified quantization framework tailored for SAMs. Firstly, to mitigate attention dissipation, we introduce MatMul-Aware Compensation (MAC) mechanism that transfers activation-induced quantization errors from MatMul to preceding linear weights. Secondly, to mitigate oscillation in decoder optimization, we develop a Joint Cross-Attention Reconstruction (JCAR) strategy that jointly reconstructs coupled attention branches, suppressing oscillatory behavior and promoting stable convergence. Extensive experiments show that CAR-SAM robustly quantizes SAM models down to 4-bit precision, surpassing existing methods by 14.6% and 6.6% mAP on SAM-B and SAM-L respectively.
Houji Wen, Jiangyong Yu, Jun Li +1
1Nanjing University of Science and Technology · 2Houmo AI
The recent Segment Anything Model 3 (SAM3) extends powerful promptable segmentation from spatial cues to concepts, raising the prospect of a single foundation model for universal image segmentation. However, this promise often fails to translate into reliable performance on downstream segmentation tasks. In this work, we empirically identify concept recognition, rather than mask quality, as the primary bottleneck: SAM3 accurately delineates the objects it recognizes yet struggles to robustly determine which concepts are present. Driven by this, we present Text Anchor re-Grounding (\TAG{}), a simple yet effective framework that rectifies the concept recognition of SAM3 on the target domain without dense annotations. Specifically, \TAG{} introduces an Anchor Residual Pathway comprising image-conditioned, low-rank adapters to re-ground text anchors in the target distribution and modulate subsequent text-conditioned interactions. Guided solely by image-level category labels, \TAG{} jointly optimizes binary cross-entropy with Soft Boundary Ranking objectives to calibrate presence confidence and distinguish present concepts from confusable negatives. Building upon this, we devise Confidence-Weighted Distillation to couple rich semantic context of a multimodal large language model with fine-grained spatial priors of SAM3, thereby achieving effective calibration in the absence of manual annotations. Extensive experiments demonstrate that \TAG{} unlocks SAM3's segmentation potential with only 3.1M trainable parameters, achieving impressive performance across diverse scenarios, and notably, even surpassing leading mask-supervised methods on several metrics. Our code will be publicly available at GitHub \faGithub.
Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted segmentation, but applying it directly to open-vocabulary semantic segmentation (OVSS) is inefficient: full-resolution decoding is typically run over the entire dataset vocabulary, whereas each image contains only a small active subset of classes. We introduce ActiveSAM, a training-free inference framework that turns SAM 3 into an active-vocabulary segmenter. ActiveSAM first canonicalizes and expands class prompts, then uses evidence-proportional grounding to estimate an image-conditioned active set from a low-resolution presence preview. Only retained prompts receive full-resolution mask prediction, using bucketed prompt multiplexing with the frozen SAM 3 decoder. The preview stage uses only class-presence evidence and skips unnecessary segmentation-head computation. To resolve overlapping concept responses, exclusive concept decoding compares each pixel's joint score vector with class signatures estimated once per vocabulary from unlabeled images. ActiveSAM requires no weight updates, no oracle class-presence labels and no per-dataset hyperparameter tuning. Across eight OVSS benchmarks, ActiveSAM improves the speed-accuracy tradeoff of training-free open-vocabulary semantic segmentation, outperforming the current state-of-the-art SegEarth-OV3 by +2.1 mIoU on average while running much faster, with 7.3-12.2x speedups on large-vocabulary datasets. ActiveSAM also achieves the highest accuracy under image corruptions that simulate real-world distribution shift, making it well-suited for deployment in noisy-input domains such as autonomous driving and embodied AI. Code is available at https://github.com/VILA-Lab/ActiveSAM
Tran Dinh Tien, Zhiqiang Shen
VILA Lab, Mohamed bin Zayed University of Artificial Intelligence