cs.CVNov 9, 2026

Adaptive Residual Quantization for Memory-Efficient Temporal Action Segmentation

Authors: Gerard Donahue, Guven Gergerli, Ayush Gupta, Reza Ghoddoosian, Enna Sachdeva, Faizan Siddiqui

Organizations: Honda Research Institute USA · Northeastern University · Purdue University · Johns Hopkins University

Abstract

Temporal action segmentation (TAS) is commonly trained on large banks of pre-extracted frame features, but storing and replaying those features becomes a practical bottleneck for long-form video, especially in incremental TAS where past-task data must remain available to mitigate catastrophic forgetting. Recent TAS replay and condensation methods reduce this burden, but they rely on dense frame-wise annotations and allocate storage through fixed segment-level structures, limiting their flexibility for long procedural videos whose frame complexity varies substantially over time. We introduce Adaptive Residual-Quantized Variational AutoEncoder (ARQ-VAE), a compact and label-free memory representation for TAS features that learns a discrete residual quantization of frame-level video features and then adaptively truncates the residual chain on a per-frame basis, retaining only the residual prefix that best reconstructs each frame feature. This yields a practical compact feature memory that does not depend on frame-wise labels and can therefore support multiple TAS settings with the same stored representation, while a lightweight fine-tuning stage further improves reconstruction quality for the truncated codes that are actually retained. Across standard TAS benchmarks, ARQ-VAE achieves a substantially improved storage-performance trade-off over prior replay and condensation baselines. On Breakfast, our final representation reduces training-set storage from 28 GB for the original I3D feature bank to 10 MB while retaining strong downstream segmentation performance; with ASFormer, it achieves 71.5 Edit and 51.2 F1@50 in the supervised setting. We further show that the same label-free memory remains effective for incremental and unsupervised TAS. We position ARQ-VAE as a practical memory representation for TAS and a promising direction for broader dataset condensation applications.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Improving Temporal Action Segmentation via Constraint-Aware Decoding

    May 11, 2026Yeo Keat Ee, Debaditya Roy, Chen Li +2Temporal Action SegmentationConstrained Decoding

  2. Adaptive Latent Trajectory Anchoring for Action Segmentation Dataset Condensation

    Jul 10, 2026Artheme Gauthier-Villar, Guodong Ding, Angela YaoTemporal Action SegmentationDataset Condensation

  3. Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation

    Apr 2, 2026Hinako Mitsuoka, Kazuhiro HottaTemporal Action SegmentationBoundary-Aware Segmentation