SACQ: Structure-Aware Codebook Quantization for Ultra-Low-Bit Vision-Language-Action Models
Organizations: KAIST
Abstract
Vision-Language-Action (VLA) models enable capable robot behavior, but their large memory footprint limits deployment on resource-constrained robotic platforms. Weight quantization is a natural remedy, yet post-training quantization (PTQ) recipes inherited from large language models degrade when applied to VLAs at ultra-low precision. Existing low-bit PTQ methods often fail to explicitly account for the structure of downstream operations and allocate precision without considering its effect on the policy actions. We introduce SACQ, an ultra- low-bit weight-only vector-codebook quantization framework for VLAs. SACQ combines (1) structure-aware codebook fitting with role-specific quadratic metrics that reflect input statistics and downstream sensitivity; (2) action-aware mixed-bit allocation based on the increase in final-action error as individual layers are compressed; and (3) two-stage post-quantization refinement that reassigns codeword indices with second-order error compensation and calibrates codebooks and row-group scales against the decoded action. Using GR00T N1.7, SACQ achieves 93.85% average success across four LIBERO suites at 1.5 bits per language/action weight while reducing model memory by 4.83x. With INT8 activation quantization and low-precision attention, it retains 93.55% success while running 1.51x faster than BF16. On SimplER, SACQ further achieves 63.76% and 52.27% success under Visual Matching and Variant Aggregation at 2 bits per language/action weight, compared to 58.90% and 53.45% for BF16, respectively.
Figures & tables
| Method | L/A bpw | Spatial | Object | Goal | Long | Avg. | Memory (GB) |
|---|---|---|---|---|---|---|---|
| BF16 | 16.00 | 94.2 | 97.6 | 95.2 | 89.4 | 94.10 | 5.677 |
| HQQ | 3.00 | 75.0 | 93.6 | 76.4 | 73.8 | 79.70 | 1.660 |
| GPTQ | 3.00 | 95.2 | 97.4 | 96.6 | 85.6 | 93.70 | 1.982 |
| ActQuant | 3.00 | 94.8 | 96.2 | 96.4 | 90.4 | 94.45 | 1.562 |
| AQLM | 3.00 | 93.6 | 97.4 | 92.4 | 79.4 | 90.70 | 1.551 |
| SACQ | 3.00 | 94.8 | 96.8 | 96.6 | 88.2 | 94.10 | 1.902 |
| Method | L/A bpw | Pick | Move | Drawer | Apple | Avg. | Memory (GB) |
|---|---|---|---|---|---|---|---|
| Visual Matching (VM) | |||||||
| BF16 | 16.00 | 86.00 | 91.25 | 45.37 | 12.96 | 58.90 | 5.677 |
| GPTQ | 2.00 | 70.33 | 25.00 | 6.94 | 0.00 | 25.57 | 1.524 |
| ActQuant | 2.00 | 90.67 | 62.92 | 32.87 | 0.00 | 46.61 | 1.331 |
| TWLA | 2.06 | 84.00 | 72.08 | 47.22 | 7.41 | 52.68 | 2.782 |
| AQLM | 2.00 | 73.33 | 26.25 | 24.54 | 0.00 | 31.03 | 1.595 |
| Configuration | Spatial | Object | Goal | Long | Avg. | Latency (ms) | Relative speed |
|---|---|---|---|---|---|---|---|
| BF16 | 94.2 | 97.6 | 95.2 | 89.4 | 94.10 | 21.28 | 1.00 |
| SACQ (W1.5A16) | 93.2 | 97.0 | 96.2 | 89.0 | 93.85 | 20.20 | 1.05 |
| SACQ (W1.5A8) | 93.8 | 96.2 | 95.4 | 88.8 | 93.55 | 14.07 | 1.51 |
| Setting | L/A bpw | Alloc. | Struct. | Refine. | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| BF16 | 16.0 | – | – | – | 94.2 | 97.6 | 95.2 | 89.4 | 94.10 |
| Uniform Euclidean VQ | 1.50 | – | – | – | 0.0 | 0.0 | 0.0 | 0.0 | 0.00 |
| + Mixed-bit allocation | 1.50 | ✓ | – | – | 0.0 | 0.0 | 0.0 | 0.0 | 0.00 |
| + Structure-aware fitting | 1.50 | ✓ | ✓ | – | 44.8 | 73.4 | 31.4 | 40.4 | 47.50 |
| Full SACQ | 1.50 | ✓ | ✓ | ✓ | 93.2 | 97.0 | 96.2 | 89.0 | 93.85 |
| w/o Struct. (Euclidean) | 1.50 | ✓ | – | ✓ | 95.2 | 95.8 | 93.8 | 85.6 | 92.60 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Projection | Input-side Gram | Output-side weighting |
|---|---|---|
| q proj | ||
| k proj | ||
| v proj | ||
| o proj | ||
| down proj | ||
| Direct diagonal metric | ||
| Projection | Input-side Gram | Output-side weighting |
|---|---|---|
| Q | ||
| K | ||
| V | ||
| O | ||
| FF-up | ||
| FF-down |
| Method | V/L bpw | Spatial | Object | Goal | Long | Avg. | Memory (GB) |
|---|---|---|---|---|---|---|---|
| BF16 | 16.00 | 93.8 | 98.2 | 97.2 | 94.0 | 95.80 | 15.203 |
| SACQ | 3.00 | 91.8 | 97.4 | 97.8 | 95.2 | 95.55 | 5.128 |
| GPTQ | 2.00 | 67.6 | 55.8 | 73.6 | 49.6 | 61.65 | 3.625 |
| ActQuant | 2.00 | 79.6 | 60.8 | 77.2 | 24.8 | 60.60 | 2.953 |
| TWLA | 2.11 | 91.0 | 96.4 | 96.2 | 93.8 | 94.35 | 8.960 |
| SACQ | 2.00 | 92.0 | 96.0 | 98.0 | 90.2 | 94.05 | 2.926 |
| Method | L/A bpw | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|---|
| BF16 | 16.0 | 44.60 | 41.28 | 45.52 | 43.28 | 43.67 |
| SACQ | 3.00 | 44.44 | 41.04 | 45.56 | 44.00 | 43.76 |
| SACQ | 2.00 | 44.60 | 41.08 | 44.48 | 44.32 | 43.62 |
| SACQ | 1.50 | 43.64 | 40.72 | 43.24 | 42.88 | 42.62 |
| Calib. data | Fitting | Spatial | Object | Goal | Long | Avg. | vs. Struct. |
|---|---|---|---|---|---|---|---|
| – | BF16 | 94.2 | 97.6 | 95.2 | 89.4 | 94.10 | – |
| 90 | Structure-aware | 76.2 | 94.6 | 88.0 | 77.8 | 84.15 | 0.00 |
| Layer-local | 61.8 | 95.0 | 71.8 | 69.6 | 74.55 | -9.60 | |
| Euclidean | 42.2 | 48.6 | 35.0 | 33.2 | 39.75 | -44.40 | |
| 180 | Structure-aware | 86.0 | 95.8 | 90.6 | 83.0 | 88.85 | 0.00 |
| Layer-local | 78.8 | 96.4 | 87.6 | 73.4 | 84.05 | -4.80 |
VLM Quantization
Vision-language model quantization reduces model precision to lower memory and computation while preserving multimodal capabilities.
See all 57 papers