PTQ4SAM3: Semantics-Preserving Post-Training Quantization for Segment Anything Model 3
Authors: Xi Xiao, Yunbei Zhang, Lin Zhao, Janet Wang, Xinru Jiang, Yanshu Li, Chenrui Ma, Pan Wang, Tianyang Wang, Xiao Wang
Organizations: University of Alabama at Birmingham, USA · Tulane University, USA · Northeastern University, USA · University of Texas at Austin, USA · University of Virginia, USA · University of Pittsburgh, USA · Oak Ridge National Laboratory, USA
Segment Anything Model 3 (SAM 3) can segment objects from text prompts and track them across video, but its memory and computation costs make deployment difficult. Post-training quantization (PTQ) reduces these costs by using fewer bits. However, a quantized model can select the wrong object or lose track of its target across frames. We call this prompt-alignment drift. Controlled tests show that perturbing low-variance feature directions strongly affects target selection. We propose Semantics-Preserving Quantization (SPQ), which gives more calibration weight to concept-sensitive parameters, preserves residual features, and corrects the mean error in video memory. Fixed text concepts guide calibration without task annotations. All three components operate offline and add no inference operators. With a W4A8 image encoder, SPQ improves RefCOCO [email protected] from 42.1 to 72.5 over PTQ4SAM while retaining 54.8 COCO mask AP. Experiments on LVIS, RefCOCO variants, and SA-V show consistent gains. A CLIP study also shows improved image-text retrieval.
Figures & tables
Figure 2: Overview of SPQ. Designs A and B calibrate image blocks with a fixed concept codebook and residual-basis feature matching. Design C estimates the mean memory error from paired video rollouts and folds the correction into an existing affine map. SPQ performs these steps offline and preserves the exported graph.
Bit-width
Method
COCO Val
LVIS v1.0 Val
AP ↑
AP 50 ↑
AP 75 ↑
AP ↑
AP r ↑
AP c ↑
AP f ↑
FP16
Full Precision (Baseline)
55.7
76.2
61.4
48.5
41.2
47.5
52.6
W8A8
MinMax
53.8
74.5
59.3
45.5
36.2
45.0
50.5
PTQ4ViT [ 55 ]
54.6 (+0.8)
75.4 (+0.9)
60.2 (+0.9)
46.8 (+1.3)
38.1 (+1.9)
46.1 (+1.1)
51.5
PTQ4SAM [ 30 ]
55.2 (+1.4)
75.8 (+1.3)
60.9 (+1.6)
47.9 (+2.4)
39.6 (+3.4)
46.9 (+1.9)
52.0
SPQ (Ours)
55.6 (+1.8)
76.1 (+1.6)
61.2 (+1.9)
48.4 (+2.9)
40.5 (+4.3)
47.3 (+2.3)
52.4
Table 1: Structural Integrity on COCO and LVIS. We report mask Average Precision (AP). The FP16 entries follow the official SAM 3 reported metrics. PTQ4SAM and SPQ both preserve much of the structural segmentation accuracy under low-bit quantization.
Table 2: Zero-shot referring segmentation. Reconstruction PTQ loses substantially more prompt-grounding accuracy than structural AP.
Figure 3: Qualitative comparison under W4A8. Top: instance-segmentation and text-prompted segmentation cases, with each text prompt shown in its input image. Bottom: additional prompt-labeled cases, including “orange” and “zebra,” that isolate prompt-conditioned target selection. SPQ selects the intended objects and returns more coherent masks than PTQ4SAM in the shown examples.
Figure 4: Temporal degradation and prompt-alignment diagnostics. (a) SA-V long-horizon tracking mIoU: PTQ4SAM under W4A8 decays over time, while SPQ remains close to FP16. (b) Angle between the estimated FP16 and quantized residual bases across transformer blocks: standard PTQ shows a larger angle, while SPQ shows a smaller one. (c) Vision-language alignment across blocks: PTQ4SAM drops with depth, while SPQ better preserves alignment. Shaded regions denote sample variability.
Bit-width
Method
Progressive Frame mIoU
Video J&F ↑
Consistency Score ( Tcs ) ↑
Frame 1 ↑
Frame 50 ↑
Frame 100 ↑
FP16
Full Precision (Baseline)
85.0
83.5
82.2
83.5
96.7%
W8A8
MinMax
80.1
72.5
63.8
71.5
79.6%
PTQ4ViT [ 55 ]
82.2 (+2.1)
76.5 (+4.0)
69.8 (+6.0)
76.2 (+4.7)
84.9%
PTQ4SAM [ 30 ]
83.6 (+3.5)
78.8 (+6.3)
72.5 (+8.7)
78.5 (+7.0)
86.7%
SPQ (Ours)
84.2 (+4.1)
83.1 (+10.6)
82.2 (+18.4)
83.2 (+11.7)
97.6%
Table 3: Temporal stability on SA-V. Progressive mIoU, J&F, and consistency under the shared 32-clip calibration protocol.
Table S1 : Sensitivity Analysis of α and λ on RefCOCO. The combination α=0.1 and λ=1.0 achieves the highest grounding accuracy among these settings.
Basis
k/D
AP ↑
Ref ↑
PCA residual
0.25
54.4
71.8
PCA residual
0.50
54.8
72.5
PCA residual
0.75
54.6
71.9
Text-gradient, equal-rank
0.50
54.7
72.8
Appendix
Table S2 : Rank and basis sensitivity under W4A8. AP is COCO mask AP and Ref is RefCOCO [email protected]. The text-gradient basis stacks and orthogonalizes held-out gradients of normalized visual–text concept scores and retains the same rank as the default PCA basis.
Intervention
Δ AP
Δ Ref
Ures , perturb
−1.3
−22.4
Udom , perturb
−8.7
−4.6
Random, rank-matched
−3.0
−7.2
Remove Ures
−2.1
−26.8
Remove Udom
−10.2
−5.1
Appendix
Table S3 : Matched functional interventions. Each Δ is relative to the same unperturbed quantized checkpoint. Perturbations use matched rank and norm; Ref denotes RefCOCO [email protected].
Table S6 : Image-calibration size under W4A8. “A/B images” is the only quantity varied; Design C always uses the same 32 video clips.
Method
Bit-width
Flickr30k (Zero-shot)
MSCOCO (Zero-shot)
I2T R@1 ↑
I2T R@5 ↑
T2I R@1 ↑
T2I R@5 ↑
I2T R@1 ↑
I2T R@5 ↑
T2I R@1 ↑
T2I R@5 ↑
FP16
Baseline
81.2
96.5
62.4
85.1
58.2
82.4
38.5
64.2
MinMax
W4A8
42.5
68.2
31.2
55.4
28.5
54.1
18.2
35.6
PTQ4ViT
W4A8
58.4 (+15.9)
81.5 (+13.3)
44.8 (+13.6)
68.2 (+12.8)
41.2 (+12.7)
65.5 (+11.4)
26.4 (+8.2)
45.1 (+9.5)
SPQ (Ours)
W4A8
78.5 (+36.0)
94.2 (+26.0)
59.8 (+28.6)
82.4 (+27.0)
55.8 (+27.3)
79.5 (+25.4)
36.7 (+18.5)
61.8 (+26.2)
Appendix
Table S7 : Generalization to CLIP (ViT-B/16) on Flickr30k and MSCOCO. We evaluate zero-shot image–text retrieval under W4A8. SPQ recovers retrieval accuracy on both datasets, extending alignment-aware calibration to a separate vision–language model.
Method
Bit-width
Model Size
Peak Memory
Latency
Throughput
(Weight / Act. / Memory)
(GB) ↓
(GB) ↓
(ms / frame) ↓
(FPS) ↑
Full Precision
FP16 / FP16 / FP16
1.35
14.8
42.5
23.5
PTQ4SAM
INT8 / INT8 / INT8
0.68 ( 2.0× )
8.5
25.4
39.3 ( 1.6× )
SPQ (Ours)
INT8 / INT8 / INT8
0.68 ( 2.0× )
8.5
25.4
39.3 ( 1.6× )
SPQ (Ours)
INT4 / INT8 / INT8
0.38 ( 3.5× )
7.9
18.2
54.9 ( 2.3× )
Appendix
Table S8 : Hardware Efficiency Analysis on an NVIDIA RTX 3090 GPU. Measurements use 1024 × 1024 video frames. SPQ achieves up to 3.5× compression and nearly 2.3× speedup over FP16. At the same bit-width, SPQ preserves the inference cost of standard PTQ.