Adaptive Residual Quantization for Memory-Efficient Temporal Action Segmentation
Organizations: Honda Research Institute USA · Northeastern University · Purdue University · Johns Hopkins University
Abstract
Temporal action segmentation (TAS) is commonly trained on large banks of pre-extracted frame features, but storing and replaying those features becomes a practical bottleneck for long-form video, especially in incremental TAS where past-task data must remain available to mitigate catastrophic forgetting. Recent TAS replay and condensation methods reduce this burden, but they rely on dense frame-wise annotations and allocate storage through fixed segment-level structures, limiting their flexibility for long procedural videos whose frame complexity varies substantially over time. We introduce Adaptive Residual-Quantized Variational AutoEncoder (ARQ-VAE), a compact and label-free memory representation for TAS features that learns a discrete residual quantization of frame-level video features and then adaptively truncates the residual chain on a per-frame basis, retaining only the residual prefix that best reconstructs each frame feature. This yields a practical compact feature memory that does not depend on frame-wise labels and can therefore support multiple TAS settings with the same stored representation, while a lightweight fine-tuning stage further improves reconstruction quality for the truncated codes that are actually retained. Across standard TAS benchmarks, ARQ-VAE achieves a substantially improved storage-performance trade-off over prior replay and condensation baselines. On Breakfast, our final representation reduces training-set storage from 28 GB for the original I3D feature bank to 10 MB while retaining strong downstream segmentation performance; with ASFormer, it achieves 71.5 Edit and 51.2 F1@50 in the supervised setting. We further show that the same label-free memory remains effective for incremental and unsupervised TAS. We position ARQ-VAE as a practical memory representation for TAS and a promising direction for broader dataset condensation applications.
Figures & tables
| GTEA [ 9 ] | 50Salads | Breakfast | ||||||||||
| Acc | Edit | F1@{10,25,50} | Storage | Acc | Edit | F1@{10,25,50} | Storage | Acc | Edit | F1@{10,25,50} | Storage | |
| MS-TCN [ 8 ] | ||||||||||||
| Original | 75.8 | 81.2 | 86.4 / 83.5 / 71.4 | 245 MB | 79.3 | 65.3 | 73.2 / 70.9 / 61.9 | 4.5 GB | 66.5 | 62.8 | 55.7 / 51.0 / 40.1 | 28 GB |
| Mean | 71.2 | 73.3 | 77.1 / 73.7 / 59.4 | 7.2 MB | 69.0 | 42.7 | 50.0 / 46.1 / 37.4 | 7.8 MB | 47.6 | 31.8 | 27.8 / 23.3 / 15.6 | 96 MB |
| Coreset | 66.7 | 66.1 | 72.4 / 68.9 / 53.2 | 7.2 MB | 61.7 | 43.3 | 49.9 / 46.3 / 35.4 | 7.8 MB | 49.7 | 36.8 | 32.3 / 27.5 / 19.3 | 96 MB |
| TCA [ 6 ] | 60.9 | 54.1 | 59.2 / 55.3 / 39.3 | – | 56.4 | 33.6 | 39.8 / 35.8 / 25.9 | – | 34.2 | 20.7 | 17.9 / 13.8 / 8.4 | – |
| Features | GTEA | 50 Salads | Breakfast | |||||||||
| MoF | F1 | mIoU | Size | MoF | F1 | mIoU | Size | MoF | F1 | mIoU | Size | |
| I3D | 56.9 | 52.7 | 36.1 | 245 MB | 65.2 | 51.2 | 32.8 | 4.5 GB | 60.0 | 58.7 | 40.0 | 28 GB |
| RQ-VAE | 58.1 | 51.9 | 36.8 | 1.3 MB | 60.4 | 43.6 | 27.3 | 4.9 MB | 65.2 | 55.0 | 36.5 | 25 MB |
| ARQ-VAE | 61.0 | 55.2 | 38.3 | 1.1 MB | 59.1 | 43.4 | 26.1 | 2.7 MB | 64.1 | 53.3 | 34.7 | 11 MB |
| ARQ-VAE ∗ | 59.0 | 55.9 | 40.6 | 1.1 MB | 61.9 | 45.1 | 28.0 | 2.7 MB | 64.6 | 54.5 | 36.1 | 11 MB |
| Dataset | MSE | Storage (MB) | |||
| RQ-VAE | ARQ-VAE | ARQ-VAE ∗ | RQ-VAE | ARQ-VAE | |
| GTEA | 0.104 | 0.026 (-75.0%) | 0.022 (-78.8%) | 0.989 | 0.451 (-54.4%) |
| 50Salads | 0.076 | 0.024 (-68.4%) | 0.022 (-70.3%) | 4.512 | 1.471 (-67.4%) |
| Breakfast | 0.042 | 0.025 (-41.5%) | 0.023 (-45.2%) | 21.901 | 8.392 (-61.7%) |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Complete storage (MB) | Compression | MSE | MAE | Acc | Edit | F1@{10,25,50} |
| Original FP32 | 3777.877 | 1.00 | 0 | 0 | 81.61 | 70.79 | 76.39 / 73.61 / 64.81 |
| FP16 | 1888.947 | 2.00 | 0.00004 | 81.43 | 73.34 | 77.86 / 75.52 / 67.13 | |
| INT8 | 944.487 | 4.00 | 0.00002 | 0.00312 | 80.30 | 63.42 | 69.62 / 66.67 / 58.23 |
| Lossless | 3331.176 | 1.13 | 0 | 0 | 81.61 | 70.79 | 76.39 / 73.61 / 64.81 |
| PCA, | 3.725 | 1014.18 | 0.02948 | 0.11284 | 46.92 | 39.90 | 37.83 / 33.48 / 25.22 |
| PCA, | 7.431 | 508.41 | 0.02732 | 0.10837 | 67.58 | 44.69 | 49.71 / 46.21 / 37.67 |
| Dataset | Metric | Mean P | Mean S | % neg. vids | Pooled P |
| 50Salads | Pred. | -0.2057 | -0.1966 | 100.0 | -0.2163 |
| Var. | -0.1828 | -0.1847 | 97.5 | -0.1951 | |
| GTEA | Pred. | -0.2856 | -0.2675 | 100.0 | -0.2999 |
| Var. | -0.2688 | -0.2524 | 100.0 | -0.2860 | |
| Breakfast | Pred. | -0.2135 | -0.2031 | 90.9 | -0.2587 |
| Var. | -0.2046 | -0.1913 | 88.0 | -0.2511 |
| Criterion | Mean depth | Storage (MB) | Complete (MB) | MSE | MAE | Acc | Edit | F1@{10,25,50} |
| Fixed-8 (RQ-VAE) | 8.00 | 5.206 | 7.574 | 0.02605 | 0.10664 | 69.84 | 59.87 | 67.88 / 63.78 / 54.67 |
| L2 argmin (ARQ-VAE) | 3.55 | 3.152 | 5.521 | 0.02068 | 0.09296 | 74.92 | 62.01 | 69.20 / 65.18 / 57.59 |
| L2-permuted | 3.55 | 3.152 | 5.521 | 0.02380 | 0.09900 | 70.50 | 58.98 | 64.41 / 60.81 / 53.15 |
| Label-aware probe | 5.13 | 3.880 | 6.249 | 0.02420 | 0.10087 | 68.45 | 51.11 | 57.52 / 54.10 / 44.57 |
| Criterion | Mean depth | Storage (MB) | Complete (MB) | MSE | Acc | Edit | F1@50 |
| Fixed-8 (RQ-VAE) | 8.00 | 5.206 | 7.574 | 0.02605 | 69.84 | 59.87 | 54.67 |
| (lower rate) | 2.06 | 2.466 | 4.835 | 0.02126 | 68.54 | 53.75 | 50.65 |
| (mild rate penalty) | 3.30 | 3.037 | 5.406 | 0.02069 | 72.64 | 57.87 | 56.05 |
| (L2 argmin, ARQ-VAE) | 3.55 | 3.152 | 5.521 | 0.02068 | 74.92 | 62.01 | 57.59 |
| (mild high-rate bias) | 3.82 | 3.277 | 5.646 | 0.02069 | 75.28 | 60.57 | 58.21 |
| (strong high-rate bias) | 6.15 | 4.353 | 6.722 | 0.02191 | 74.67 | 56.75 | 54.07 |
| Representation | Codec prep (s) | Decode / epoch (s) | Decode, 50 epochs (s) | Codec total (s) |
| RQ-VAE | 102.91 | 3.78 | 189.00 | 291.91 |
| ARQ-VAE | 119.38 | 3.43 | 171.50 | 290.88 |
| ARQ-VAE ∗ | 333.49 | 3.54 | 177.00 | 510.49 |
| Method | Reconstruction | TAS metrics | ||||||
| MSE | MAE | Storage | Acc | Edit | F1@10 | F1@25 | F1@50 | |
| Original | – | – | 4.5 GB | 70.4 | 67.9 | 69.6 | 66.8 | 54.3 |
| RQ-VAE | 0.034 | 0.143 | 4.1 MB | 60.3 | 53.9 | 59.1 | 55.8 | 41.9 |
| ARQ-VAE | 0.017 | 0.099 | 1.8 MB | 65.1 | 56.6 | 57.9 | 53.2 | 41.1 |
| ARQ-VAE ∗ | 0.014 | 0.092 | 1.8 MB | 67.2 | 61.6 | 63.8 | 57.9 | 45.2 |
| Method | Acc | Edit | F1@10 | F1@25 | F1@50 |
| Original | 85.10 | 79.29 | 84.77 | 83.08 | 77.61 |
| RQ-VAE | 77.65 | 70.99 | 76.30 | 74.16 | 66.28 |
| ARQ-VAE | 79.13 | 73.02 | 79.06 | 76.30 | 66.52 |
| ARQ-VAE ∗ | 80.23 | 73.38 | 79.26 | 77.19 | 67.28 |
Temporal Action Segmentation
Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.
See all 50 papers