Temporal action segmentation (TAS) is commonly trained on large banks of pre-extracted frame features, but storing and replaying those features becomes a practical bottleneck for long-form video, especially in incremental TAS where past-task data must remain available to mitigate catastrophic forgetting. Recent TAS replay and condensation methods reduce this burden, but they rely on dense frame-wise annotations and allocate storage through fixed segment-level structures, limiting their flexibility for long procedural videos whose frame complexity varies substantially over time. We introduce Adaptive Residual-Quantized Variational AutoEncoder (ARQ-VAE), a compact and label-free memory representation for TAS features that learns a discrete residual quantization of frame-level video features and then adaptively truncates the residual chain on a per-frame basis, retaining only the residual prefix that best reconstructs each frame feature. This yields a practical compact feature memory that does not depend on frame-wise labels and can therefore support multiple TAS settings with the same stored representation, while a lightweight fine-tuning stage further improves reconstruction quality for the truncated codes that are actually retained. Across standard TAS benchmarks, ARQ-VAE achieves a substantially improved storage-performance trade-off over prior replay and condensation baselines. On Breakfast, our final representation reduces training-set storage from 28 GB for the original I3D feature bank to 10 MB while retaining strong downstream segmentation performance; with ASFormer, it achieves 71.5 Edit and 51.2 F1@50 in the supervised setting. We further show that the same label-free memory remains effective for incremental and unsupervised TAS. We position ARQ-VAE as a practical memory representation for TAS and a promising direction for broader dataset condensation applications.
Figures & tables
Figure 1: Storage efficiency across TAS settings. ARQ-VAE achieves a better storage–performance trade-off while growing memory more slowly than feature replay and GNI.
Figure 2: Residual-Quantized Variational AutoEncoder (RQ-VAE) for compact TAS feature memory. A pretrained video backbone extracts a frame feature xt from video frame vt . The encoder Eψ maps xt into latent space, where residual quantization selects a sequence of discrete indices kt(m) and code vectors qt(m) from codebooks C(1),…,C(M) . These stage-wise codes are summed to form the quantized latent z~t=∑m=1Mqt(m) , which is decoded by Dθ to reconstruct x^t . The stored representation is the sequence of discrete stage indices.
Figure 3: Adaptive prefix selection in ARQ-VAE. After training a fixed-depth RQ-VAE, we decode every prefix of the residual chain. For frame t , the partial latents z~t(m) yield partial reconstructions x^t(m) . ARQ-VAE retains the prefix length mt∗ with the lowest reconstruction error (shown in green ) and discards the remaining later-stage indices (shown in red ). The resulting memory therefore stores a variable number of stage indices per frame.
GTEA [ 9 ]
50Salads
Breakfast
Acc
Edit
F1@{10,25,50}
Storage
Acc
Edit
F1@{10,25,50}
Storage
Acc
Edit
F1@{10,25,50}
Storage
MS-TCN [ 8 ]
Original
75.8
81.2
86.4 / 83.5 / 71.4
245 MB
79.3
65.3
73.2 / 70.9 / 61.9
4.5 GB
66.5
62.8
55.7 / 51.0 / 40.1
28 GB
Mean
71.2
73.3
77.1 / 73.7 / 59.4
7.2 MB
69.0
42.7
50.0 / 46.1 / 37.4
7.8 MB
47.6
31.8
27.8 / 23.3 / 15.6
96 MB
Coreset
66.7
66.1
72.4 / 68.9 / 53.2
7.2 MB
61.7
43.3
49.9 / 46.3 / 35.4
7.8 MB
49.7
36.8
32.3 / 27.5 / 19.3
96 MB
TCA [ 6 ]
60.9
54.1
59.2 / 55.3 / 39.3
–
56.4
33.6
39.8 / 35.8 / 25.9
–
34.2
20.7
17.9 / 13.8 / 8.4
–
Table 1: Temporal Action Segmentation results on three TAS benchmark datasets.
Figure 5
Features
GTEA
50 Salads
Breakfast
MoF
F1
mIoU
Size
MoF
F1
mIoU
Size
MoF
F1
mIoU
Size
I3D
56.9
52.7
36.1
245 MB
65.2
51.2
32.8
4.5 GB
60.0
58.7
40.0
28 GB
RQ-VAE
58.1
51.9
36.8
1.3 MB
60.4
43.6
27.3
4.9 MB
65.2
55.0
36.5
25 MB
ARQ-VAE
61.0
55.2
38.3
1.1 MB
59.1
43.4
26.1
2.7 MB
64.1
53.3
34.7
11 MB
ARQ-VAE ∗
59.0
55.9
40.6
1.1 MB
61.9
45.1
28.0
2.7 MB
64.6
54.5
36.1
11 MB
Table 2: Unsupervised action segmentation with ASOT [ 39 ] . Gray rows use original I3D features; other rows replace I3D with reconstructed training features.
Dataset
MSE ↓
Storage (MB) ↓
RQ-VAE
ARQ-VAE
ARQ-VAE ∗
RQ-VAE
ARQ-VAE
GTEA
0.104
0.026 (-75.0%)
0.022 (-78.8%)
0.989
0.451 (-54.4%)
50Salads
0.076
0.024 (-68.4%)
0.022 (-70.3%)
4.512
1.471 (-67.4%)
Breakfast
0.042
0.025 (-41.5%)
0.023 (-45.2%)
21.901
8.392 (-61.7%)
Table 3: Summary reconstruction error and storage averaged across various (K,M) settings.
Figure 6: Hyperparameter ablations on the number of codes ( K ) and residual stages ( M ). (a) Ablation varying K . (b) Ablation varying M .
Figure 7: Prefix-selection criteria on a single 50Salads split with MS-TCN. Left: F1@50 vs. storage without decoder; the gray curve sweeps λ ( λ=0 is L2 argmin). Right: TAS metrics per criterion; MB is storage without decoder. Full results in Tables 8 and 9 .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Complete storage (MB)
Compression
MSE ↓
MAE ↓
Acc
Edit
F1@{10,25,50}
Original FP32
3777.877
1.00 ×
0
0
81.61
70.79
76.39 / 73.61 / 64.81
FP16
1888.947
2.00 ×
≈0
0.00004
81.43
73.34
77.86 / 75.52 / 67.13
INT8
944.487
4.00 ×
0.00002
0.00312
80.30
63.42
69.62 / 66.67 / 58.23
Lossless
3331.176
1.13 ×
0
0
81.61
70.79
76.39 / 73.61 / 64.81
PCA, d=2
3.725
1014.18 ×
0.02948
0.11284
46.92
39.90
37.83 / 33.48 / 25.22
PCA, d=4
7.431
508.41 ×
0.02732
0.10837
67.58
44.69
49.71 / 46.21 / 37.67
Appendix
Table 4: Comparison with standard feature compression on 50Salads with MS-TCN. Storage is the complete footprint, including all method-specific parameters (e.g., the decoder for RQ-VAE and ARQ-VAE).
Figure 8: Storage–performance trade-off against standard feature compression on 50Salads with MS-TCN. Storage is the complete footprint, including all method-specific parameters, on a log scale. In the high-compression regime, ARQ-VAE retains far more downstream TAS performance than PCA at comparable storage.
Figure 9: Storage–performance plots for the width and depth ablations, using reconstruction mean absolute error and downstream temporal action segmentation performance.
Figure 10: Distribution of selected prefix lengths across GTEA, 50Salads, and Breakfast.
Dataset
Metric
Mean P ↓
Mean S ↓
% neg. vids ↑
Pooled P ↓
50Salads
Pred.
-0.2057
-0.1966
100.0
-0.2163
Var.
-0.1828
-0.1847
97.5
-0.1951
GTEA
Pred.
-0.2856
-0.2675
100.0
-0.2999
Var.
-0.2688
-0.2524
100.0
-0.2860
Breakfast
Pred.
-0.2135
-0.2031
90.9
-0.2587
Var.
-0.2046
-0.1913
88.0
-0.2511
Appendix
Table 7: Frame-wise prefix length is inversely correlated with feature redundancy. Pred. denotes local-context predictability and Var. denotes local temporal variance. Negative correlation means that more redundant frames tend to use shorter prefixes, and larger redundancy scores indicate greater temporal redundancy.
Figure 11: Example video-level relationship between prefix length and feature redundancy. Top: normalized trajectories of the selected prefix length and the feature redundancy score over time, with Pearson and Spearman correlations shown in the inset. Bottom: ground-truth action segmentation for the same video. Regions with sustained actions such as spoon_powder , pour_milk , and stir_milk tend to exhibit higher redundancy and shorter selected prefixes, while lower-redundancy portions of the sequence coincide with longer prefixes. This example illustrates the inverse relationship quantified in Table 7 : ARQ-VAE allocates more residual depth to temporally harder, less redundant frames and fewer stages to stable action interiors.
Figure 12: Residual code reuse on a GTEA example. The top row shows ground-truth action segments, while lower rows show frequently reused code indices at residual stages s0,…,s4 . Colored intervals indicate retained codes, and greyed-out regions indicate later-stage codes removed by ARQ-VAE prefix truncation. Reused codes often align with repeated actions such as take , open , and pour , suggesting a weak correspondence between discrete codes and recurring procedural structure.
Figure 13: Residual code reuse on a second GTEA example. As in Fig. 12 , early residual stages contain longer reused code intervals that overlap with major action segments, while later stages are more fragmented and are often discarded by adaptive prefix selection. Repeated codes appear across multiple instances of actions such as take , open , pour , and fold .
Criterion
Mean depth
Storage (MB)
Complete (MB)
MSE ↓
MAE ↓
Acc
Edit
F1@{10,25,50}
Fixed-8 (RQ-VAE)
8.00
5.206
7.574
0.02605
0.10664
69.84
59.87
67.88 / 63.78 / 54.67
L2 argmin (ARQ-VAE)
3.55
3.152
5.521
0.02068
0.09296
74.92
62.01
69.20 / 65.18 / 57.59
L2-permuted
3.55
3.152
5.521
0.02380
0.09900
70.50
58.98
64.41 / 60.81 / 53.15
Label-aware probe
5.13
3.880
6.249
0.02420
0.10087
68.45
51.11
57.52 / 54.10 / 44.57
Appendix
Table 8: Prefix-selection criteria on 50Salads with MS-TCN. Storage excludes the decoder; Complete includes it.
Criterion
Mean depth
Storage (MB)
Complete (MB)
MSE ↓
Acc
Edit
F1@50
Fixed-8 (RQ-VAE)
8.00
5.206
7.574
0.02605
69.84
59.87
54.67
λ=10−3 (lower rate)
2.06
2.466
4.835
0.02126
68.54
53.75
50.65
λ=10−4 (mild rate penalty)
3.30
3.037
5.406
0.02069
72.64
57.87
56.05
λ=0 (L2 argmin, ARQ-VAE)
3.55
3.152
5.521
0.02068
74.92
62.01
57.59
λ=−10−4 (mild high-rate bias)
3.82
3.277
5.646
0.02069
75.28
60.57
58.21
λ=−10−3 (strong high-rate bias)
6.15
4.353
6.722
0.02191
74.67
56.75
54.07
Appendix
Table 9: Rate–distortion prefix selection , Eq. 5 , on 50Salads with MS-TCN. Storage excludes the decoder; Complete includes it.
Representation
Codec prep (s)
Decode / epoch (s)
Decode, 50 epochs (s)
Codec total (s)
RQ-VAE
102.91
3.78
189.00
291.91
ARQ-VAE
119.38
3.43
171.50
290.88
ARQ-VAE ∗
333.49
3.54
177.00
510.49
Appendix
Table 10: Codec-related computational cost on 50Salads. Codec preparation is a one-time offline cost; decoding reconstructs the complete 40-video training bank once per epoch, and the 50-epoch columns follow the MS-TCN schedule.
Figure 14: Computational cost on 50Salads. (a) Codec-related time over the 50-epoch MS-TCN schedule: one-time preparation (solid) plus on-the-fly decoding (light). (b) End-to-end MS-TCN training wall-clock, including feature access and on-the-fly reconstruction; the hatched segment adds ARQ-VAE’s one-time preparation, paid only on the first run.
Method
Reconstruction
TAS metrics
MSE ↓
MAE ↓
Storage ↓
Acc ↑
Edit ↑
F1@10 ↑
F1@25 ↑
F1@50 ↑
Original
–
–
4.5 GB
70.4
67.9
69.6
66.8
54.3
RQ-VAE
0.034
0.143
4.1 MB
60.3
53.9
59.1
55.8
41.9
ARQ-VAE
0.017
0.099
1.8 MB
65.1
56.6
57.9
53.2
41.1
ARQ-VAE ∗
0.014
0.092
1.8 MB
67.2
61.6
63.8
57.9
45.2
Appendix
Table 11: Results on DINOv3 frame features under the same condensation pipeline. Lower is better for reconstruction metrics and storage; higher is better for TAS metrics. The results show that the proposed method is not specific to I3D features and extends effectively to a different pretrained feature space.
Method
Acc
Edit
F1@10
F1@25
F1@50
Original
85.10
79.29
84.77
83.08
77.61
RQ-VAE
77.65
70.99
76.30
74.16
66.28
ARQ-VAE
79.13
73.02
79.06
76.30
66.52
ARQ-VAE ∗
80.23
73.38
79.26
77.19
67.28
Appendix
Table 12: FACT [ 23 ] on 50Salads , averaged over the five standard splits. Original trains on the full I3D feature bank; the other rows train on reconstructed features.
Figure 15: FACT on 50Salads (5-split average). The ordering RQ-VAE < ARQ-VAE < ARQ-VAE ∗ observed with MS-TCN and ASFormer is unchanged.
Temporal action segmentation (TAS) divides untrimmed videos into labeled action segments. While fully supervised methods have advanced the field, challenges such as action variability, ambiguous boundaries, and high annotation costs remain, especially in new or low-resource domains. Grammar-based approaches improve segmentation with structural priors but rely on complex parsing limiting scalability. In this work, we propose a lightweight, constraint-based refinement framework that enhances TAS predictions by integrating statistical structural priors such as transition confidence, action boundary sets, and per-class duration, that can be directly extracted from annotated data. These constraints are integrated into a modified Viterbi decoding algorithm, allowing inference-time refinement without retraining or added model complexity. Our approach improves both fully and semi-supervised TAS models by correcting structural prediction errors while maintaining high efficiency. Code is available at https://github.com/LUNAProject22/CAD
Yeo Keat Ee, Debaditya Roy, Chen Li +2
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1
Dataset condensation for action segmentation synthesizes compact, informative representations of long, untrimmed video datasets. The existing approach relies on Variational Autoencoders and an iterative latent optimization; it is computationally expensive and suffers from over-smoothed reconstructions and rigid temporal constraints. This paper proposes to shift the condensation paradigm from optimization-based inversion to deterministic latent mapping. By leveraging Denoising Diffusion Implicit Models, we represent action segments as continuous trajectories anchored by sparse latent points in the noise manifold. To maximize representational efficiency, we introduce an adaptive allocation mechanism that dynamically redistributes the anchoring budget based on segment-wise reconstruction difficulty. Extensive experiments demonstrate that our framework significantly outperforms state-of-the-art methods in segmentation performance across common datasets. Notably, our approach achieves performance parity with real data training while maintaining a condensation ratio of 2.4% on Breakfast dataset.
Artheme Gauthier-Villar, Guodong Ding, Angela Yao
ETH Zurich, Switzerland · National University of Singapore, Singapore
Recent progress in Temporal Action Segmentation (TAS) has increasingly relied on complex architectures, which can hinder practical deployment. We present a lightweight dual-loss training framework that improves fine-grained segmentation quality with only one additional output channel and two auxiliary loss terms, requiring minimal architectural modification. Our approach combines a boundary-regression loss that promotes accurate temporal localization via a single-channel boundary prediction and a CDF-based segment-level regularization loss that encourages coherent within-segment structure by matching cumulative distributions over predicted and ground-truth segments. The framework is architecture-agnostic and can be integrated into existing TAS models (e.g., MS-TCN, C2F-TCN, FACT) as a training-time loss function. Across three benchmark datasets, the proposed method improves segment-level consistency and boundary quality, yielding higher F1 and Edit scores across three different models. Frame-wise accuracy remains largely unchanged, highlighting that precise segmentation can be achieved through simple loss design rather than heavier architectures or inference-time refinements.
Hinako Mitsuoka, Kazuhiro Hotta
Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan