LoopQ: Fair, Closed-Loop Evaluation of Quantized Vision-Language-Action Models
Organizations: Chalmers University of Technology and University of Gothenburg Computer and Network Systems
Abstract
Post-training quantization (PTQ) for vision-language-action (VLA) models promises smaller and faster VLA deployments. However, when analyzing existing VLA quantization works, we realised that it is hard to reason about the performance of the suggested quantization method compared to existing, similar, or in many cases, simpler PTQ methods. We present LoopQ, a closed-loop benchmark that compares multiple quantization policies on the same checkpoint, layers and engine, varying only the method and the kernels. We evaluate RTN, AWQ, GPTQ, ActQuant and QuantVLA on four VLAs with more than 180,000 episodes of LIBERO and LIBERO-plus, compare against QVLA's reported results on the same layers, and benchmark whether the four public VLA-quantization releases can be reproduced. We benchmark the methods on an NVIDIA H100, an L4, and a Jetson AGX Thor. Our results show that while quantization reliably saves memory, 2-3x for real low-bit models, VLA-specific PTQ methods underperform compared to simpler generic baselines. At 4 bits, RTN, AWQ and GPTQ match or exceed ActQuant and QuantVLA, plain RTN matches the calibrated methods, and plain AWQ matches the success QVLA reports for itself. Standard LIBERO hides the losses that do exist: configurations with no significant LIBERO loss lose 1.5-4.3 points under paired LIBERO-plus perturbations. Speed is a weaker story. At batch 1 on the H100, no quantized pi0.5 or OpenVLA-OFT configuration is faster than its 16-bit model (1.06-1.58x slower on library kernels); only the L4 with int8 kernels gains (4-14%), the Thor is slower on every 4-bit configuration, and the engine alone changes latency by 2.1x. Profiling explains why: pi0.5's quantized backbone is compute-bound, while its memory-bound action expert, 63% of GPU time and 13% of the parameters, stays in 16 bits.
Figures & tables
| Model | Execution | Configurations | LIBERO | LIBERO-plus |
| real | 18 | 2,000 | 2,016 a | |
| GR00T N1.5 | sim. | 6 | 2,000 | 2,016 |
| UniVLA | sim. | 5 | 1,000 | 5,014 |
| OpenVLA-OFT | sim. | 5 | 1,000 | 5,014 |
| OpenVLA-OFT | real | 10 b | 2,000 | 2,016 |
| Configuration | File GB | Peak GiB | ms/call | base | J/call |
| , GGML engine | |||||
| F16 | 6.71 | 7.04 | 66.8 | 1.00 | 13.8 |
| Q8_0 cuBLAS / MMQ | 3.97 | 4.55 / 4.48 | 71.0 / 79.7 | 1.06 / 1.19 | 15.5 / 16.6 |
| Q4_0 cuBLAS / MMQ † | 2.51 | 3.19 / 3.12 | 83.1 / 88.7 | 1.24 / 1.33 | 16.5 / 17.5 |
| AWQ cuBLAS / MMQ | 2.70 | 3.36 / 3.30 | 74.9 / 83.9 | 1.12 / 1.26 | 15.9 / 16.9 |
| GPTQ cuBLAS / MMQ | 2.70 | 3.36 / 3.30 | 74.1 / 83.0 | 1.11 / 1.24 | 15.5 / 16.8 |
| LIBERO | LIBERO-plus | ||||
| Model | Config | % | % | ||
| GGML F16 | 96.5 | – | 86.6 | – | |
| RTN Q8_0 | 96.7 | +0.2 | 87.7 | +1.1 | |
| RTN Q4_0 | 96.9 | +0.4 | 86.9 | +0.2 | |
| AWQ W4 | 97.0 | +0.5 | 86.5 | 0.1 | |
| GPTQ W4 | 97.1 | +0.6 | 86.4 | 0.2 | |
| LIBERO (%) | LIBERO-plus (%) | |||||
| Configuration | H100 | L4 | Thor | H100 | L4 | Thor |
| (real) | ||||||
| F16 | 95.0 | 94.2 | 97.5 | 87.3 | 86.5 | 87.3 |
| RTN Q4_0 | 95.0 | 99.2 | 96.7 | 88.5 (+1.2) | 86.9 (+0.4) | 85.7 ( 1.6) |
| AWQ W4 | 96.7 | 95.8 | 97.5 | 87.3 (+0.0) | 85.3 ( 1.2) | 86.5 ( 0.8) |
| GPTQ W4 | 97.5 | 97.5 | 95.8 | 87.3 (+0.0) | 84.5 ( 2.0) | 86.5 ( 0.8) |
| Method | Real low-bit | What we found |
| ActQuant | ✓ | Reproduces its LIBERO results (96.6/96.0/94.5 vs. 96.6/96.3/94.8%), but runs – slower than F16 on its own engine; its reported speed-up (up to ) is measured on OpenVLA against PyTorch. Model cards name a different base checkpoint than the files were built from. |
| QuantVLA | ✗ | GR00T code only. Simulated W4A8 with floating-point weight copies, so larger and slower; memory computed from bit widths, speed-up only in the repository; activation scales frozen after the first request. |
| QVLA | ✗ | Simulated. Code needs fixes to run on OpenVLA-OFT; the released sensitivity proxy is not the paper’s estimator and does not regenerate its results. |
| HoloQ-VLA | ✗ | Inspected, not run. Published checkpoints are FP16 with no INT4 kernel (simulated W4A4); the release quantizes 252 of the paper’s 414 layers. |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Scheme | Methods | Calib. | Format | Exec. |
| W8A16 | RTN Q8_0 | none | int8, block 32, 8.5 bpw a | real / sim. |
| W4A16 | RTN Q4_0 | none | int4, block 32, 4.5 bpw a | real / sim. |
| W4A16 | AWQ, GPTQ | 512 fr. | uint4, group 128, 4.19 bpw; run as Q4_1, 5.0 bpw b | real / sim. |
| W A8-dyn | RTN/AWQ/GPTQ on MMQ | none | int8 act. per 32 values | real |
| W3–4 A8-dyn | ActQuant 4/3.5/3 | 60 ep. | k-quant mix, 3–4 bpw nominal | real |
| W4A8 static | QuantVLA | first requests | DuQuant W4, A8 per channel | sim. |
| Configuration | SigLIP | PaliGemma | Total | +ms | Faster |
| Q8_0 cuBLAS | 1.16 | 1.33 | 1.24 | 3.4 | 0/283 |
| Q8_0 MMQ | 1.85 | 2.17 | 2.00 | 14.1 | 0/283 |
| Q4_0 cuBLAS / MMQ | 1.36 / 1.88 | 1.79 / 2.26 | 1.57 / 2.06 | 8.0 / 14.9 | 0/283 |
| AWQ, GPTQ cuBLAS | 1.36 | 1.78 | 1.56 | 7.8 | 0/283 |
| AWQ, GPTQ MMQ | 1.96 | 2.49 | 2.21 | 17.1 | 0/283 |
| AWQ, GPTQ fused | 14.0 | 44.8 | 28.7 | 389.5 | 0/283 |
| Component | Quantized | Tokens | Calls | GPU ms (share) | GEMM | Non-GEMM | AI | Class | TFLOP/s | GB/s | % roof |
| SigLIP | yes | 256 | 3 | 7.6 (13%) | 4.0 | 3.6 | 177 | near ridge | 110 (168) | 621 | 16% |
| PaliGemma | yes | 968 | 1 | 13.2 (22%) | 7.2 | 5.9 | 498 | compute-bound | 509 (530) | 904 | 61% |
| Action expert | no | 10 | 10 | 37.5 (63%) | 10.5 | 27.0 | 10 | memory-bound | 7 (6) | 931 | 24% |
| Total | 59.4 | 21.8 | 37.6 |
| # | Method | Scheme (path) | File GB | Success % (95% CI) | Spatial | Object | Goal | Long | Lat. ms | base | Peak GiB |
| GGML engine (ours) | |||||||||||
| 1 | F16 | W16A16 | 6.71 | 96.5 (95.6–97.2) | 98.4 | 99.2 | 97.0 | 91.4 | 81 | 1.00 | 8.36 |
| 2 | RTN Q8_0 | W8A8-dyn (MMQ) | 3.97 | 96.7 (95.8–97.4) | 98.2 | 98.4 | 97.0 | 93.0 | 102 | 1.27 | 5.81 |
| 3 | RTN Q8_0 | W8A16 (cuBLAS) | 3.97 | 96.7 (95.8–97.4) | 98.8 | 97.8 | 98.0 | 92.0 | 94 | 1.17 | 5.87 |
| 16 | RTN Q4_0 | W4A8-dyn (MMQ) | 2.51 | 96.8 (95.9–97.5) | 97.8 | 98.0 | 97.4 | 94.0 | 113 | 1.39 | 4.44 |
| 17 | RTN Q4_0 | W4A16 (cuBLAS) | 2.51 | 96.9 (96.0–97.6) | 99.0 | 98.2 | 97.8 | 92.6 | 93 | 1.15 | 4.51 |
| Model | Config | L1 | L2 | L3 | L4 | L5 |
| GGML F16 / Q8_0 / Q4_0 / AWQ / GPTQ | 97.6 / 97.6 / 95.8 / 95.5 / 96.7 | 94.7 / 94.9 / 94.9 / 95.4 / 95.2 | 90.7 / 93.1 / 91.8 / 92.0 / 90.7 | 83.8 / 84.6 / 85.9 / 85.6 / 85.1 | 65.5 / 67.1 / 64.8 / 62.7 / 63.0 | |
| AQ F16 / AQ 4 / AQ 3 bpw | 94.2 / 94.8 / 93.9 | 92.4 / 91.2 / 89.2 | 90.0 / 86.5 / 81.6 | 79.9 / 75.5 / 72.6 | 57.5 / 54.4 / 50.8 | |
| GR00T | FP / QuantVLA | 87.0 / 88.8 | 87.3 / 87.1 | 83.0 / 79.0 | 75.2 / 70.2 | 56.2 / 43.8 |
| UniVLA | FP / W8 / RTN W4 / AWQ / GPTQ | 79.6 / 80.7 / 74.1 / 72.5 / 79.0 | 60.2 / 59.7 / 57.2 / 54.5 / 59.0 | 54.2 / 54.4 / 52.6 / 53.0 / 52.2 | 44.7 / 45.5 / 45.0 / 41.0 / 41.6 | 25.6 / 25.7 / 25.2 / 21.0 / 25.1 |
| OpenVLA-OFT | FP / W8 / RTN W4 / AWQ / GPTQ | 92.1 / 91.6 / 90.3 / 90.5 / 92.1 | 90.2 / 90.3 / 88.7 / 89.0 / 89.1 | 77.6 / 78.3 / 77.9 / 77.0 / 78.2 | 69.4 / 69.6 / 67.3 / 69.1 / 69.6 | 37.1 / 38.1 / 37.6 / 36.0 / 36.7 |
| All | Camera | Robot | Language | Light | Backgr. | Noise | Layout | |
| GGML F16, noise varies per call | 86.6 | 75.3 | 77.8 | 86.8 | 95.8 | 92.4 | 90.6 | 87.5 |
| GGML F16, fixed noise | 84.8 | 70.8 | 74.0 | 88.9 | 95.1 | 89.6 | 90.6 | 84.7 |
| ActQuant-engine F16 (fixed by design) | 82.9 | 69.4 | 71.9 | 82.6 | 94.1 | 91.0 | 86.1 | 85.4 |
| fixed varying (paired) | ∗ | |||||||
| ActQuant engine fixed (paired) | ∗ | ∗ | ∗ |
| Method | vs. PyTorch FP (16 frames) | vs. GGML F16 (MMQ / cuBLAS / fused) | vs. ActQuant-engine F16 |
| RTN Q8_0 | 0.58% | 0.21% / 0.15% / – | – |
| GPTQ W4 | 2.05% (linears) / 2.96% (+ emb.) | 2.26% / 2.21% / 2.21% | – |
| AWQ W4 | 3.03% | 3.63% / 3.63% / 3.61% | – |
| RTN Q4_0 | 12.0% (one outlier frame; median 2.0%) | 2.49% / 2.46% / – | – |
| ActQuant 4 / 3.5 / 3 bpw | – | – | 3.3% / 8.1% / 8.8% |
| Model | Tasks | AWQ RTN | GPTQ RTN |
| 2,016 | 0.4 ( ) | 0.5 ( ) | |
| GR00T N1.5 | 2,016 | +0.3 ( ) | 0.6 ( ) |
| UniVLA | 5,014 | ( ) | +0.5 ( ) |
| OpenVLA-OFT | 5,014 | 0.1 ( ) | +0.7 ( ) |
| LIBERO (Sp/Ob/Go/Lo) | LIBERO-plus (Sp/Ob/Go/Lo) | LIBERO-plus category | |||||||||||||
| Method | Sp | Ob | Go | Lo | Sp | Ob | Go | Lo | Cam. | Robot | Lang. | Light | Backg. | Noise | Layout |
| FP | 95.6 | 95.8 | 94.8 | 86.0 | 83.9 | 81.5 | 75.8 | 71.6 | 72.2 | 45.5 | 83.7 | 94.8 | 94.4 | 89.2 | 67.7 |
| QuantVLA | 96.2 | 97.6 | 92.0 | 85.2 | 83.7 | 77.0 | 63.5 | 71.8 | 10.8 ∗ | 5.2 ∗ | 6.6 ∗ | 5.6 ∗ | 4.2 | 2.1 | +4.9 |
| Q8_0 | 94.2 | 95.6 | 95.6 | 86.8 | 84.3 | 83.7 | 74.2 | 71.6 | 0.3 | 1.4 | 0.3 | 1.0 | 0.3 | +3.1 | +2.1 |
| Q4_0 | 95.6 | 96.0 | 94.6 | 86.2 | 81.7 | 81.5 | 75.8 | 73.8 | 1.7 | 0.7 | 1.7 | 2.4 | 0.7 | +2.8 | +4.5 ∗ |
| AWQ | 95.4 | 95.6 | 94.0 | 85.8 | 81.3 | 84.1 | 75.6 | 72.8 | 0.0 | +1.0 | +1.4 | 1.4 | 2.4 | +0.3 | +2.8 |
| Model | Configuration | Spatial | Object | Goal | Long | Avg (95% CI) | Storage GB |
| UniVLA | FP (BF16) | 95.2 | 94.8 | 90.8 | 90.0 | 92.7 (90.9–94.2) | 15.11 |
| RTN W8 (per channel) | 96.0 | 94.4 | 92.8 | 92.0 | 93.8 (92.1–95.1) | 7.91 | |
| RTN W4 (g128) | 97.6 | 91.6 | 93.2 | 86.0 | 92.1 (90.3–93.6) | 4.47 | |
| AWQ W4 (g128) | 94.8 | 94.0 | 91.2 | 88.0 | 92.0 (90.2–93.5) | 4.47 | |
| GPTQ W4 (g128) | 96.0 | 96.0 | 94.0 | 90.8 | 94.2 (92.6–95.5) | 4.47 | |
| OpenVLA-OFT (ours) | FP (BF16) | 98.8 | 96.8 | 99.6 | 94.8 | 97.5 (96.3–98.3) | 15.42 |
| H100 NVL | L4 (AWS g6.2xlarge) | Jetson AGX Thor | |
| Architecture | Hopper (sm_90) | Ada (sm_89) | Blackwell (sm_110), integrated GPU |
| Memory | 94 GB, 3.9 TB/s | 24 GB, 0.30 TB/s | 128 GB unified, 0.27 TB/s |
| FP16 TFLOP/s / ridge | 835 / 214 | 121 / 403 | – a |
| Power | 400 W limit | 72 W limit | MAXN mode, locked clocks |
| Host CPU | 64 cores | 8 vCPU | 14 Arm cores |
| CUDA / PyTorch | 12 / 2.2–2.7 | 12 / 2.2–2.7 | 13.2 / 2.9 |
| Configuration | H100 NVL | L4 | Jetson AGX Thor |
| (real low-bit) | |||
| GGML F16 | 67 | 167 | 186 |
| RTN Q4_0 MMQ | 89 (1.33 ) | 154 ( 0.93 ) | 247 (1.33 ) |
| RTN Q4_0 cuBLAS | 83 (1.25 ) | 189 (1.13 ) | 213 (1.15 ) |
| AWQ W4 MMQ | 84 (1.26 ) | 160 ( 0.96 ) | 260 (1.40 ) |
| AWQ W4 cuBLAS | 75 (1.12 ) | 190 (1.14 ) | 216 (1.16 ) |
| Energy per call (J) | Memory (GiB) | ||||||
| Configuration | H100 | L4 | Thor GPU | Thor VIN | H100 | L4 | Thor (unified) |
| GGML F16 | 13.8 | 11.8 | 7.0 | 14.7 | 7.0 | 6.6 | 7.2 |
| RTN Q4_0 MMQ | 17.5 | 11.0 | 8.5 | 17.1 | 3.1 | 2.7 | 3.3 |
| RTN Q4_0 cuBLAS | 16.5 | 13.4 | 8.0 | 16.9 | 3.2 | 2.7 | 3.3 |
| AWQ W4 MMQ | 16.9 | 11.4 | 8.8 | 17.7 | 3.3 | 2.8 | 3.4 |
| Model | Component | H100 NVL | L4 | Jetson AGX Thor |
| SigLIP | 7.6 (13%) | 18.4 (10%) | 19.0 (11%) | |
| PaliGemma prefix | 13.2 (22%) | 86.5 (48%) | 71.4 (43%) | |
| action expert | 37.5 (63%) | 75.1 (41%) | 74.9 (45%) | |
| end to end (host overhead) | 130.5 (71.1) | 329.8 (148.8) | 232.2 (65.8) | |
| OpenVLA-OFT | Llama-2 7B | 30.7 (76%) | 172.0 (86%) | 157.5 (86%) |
| end to end | 59.3 | 217.4 | 181.7 |
Post-Training Quantization
Post-Training Quantization compresses trained neural networks into low-bit weights, activations, or caches to reduce memory use and inference cost.
See all 395 papers