Organizations: Guangdong University of Technology · Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence · University of Hong Kong
Humans perceive actions through key transitions that structure actions across multiple abstraction levels, whereas machines, relying on visual features, tend to over-segment. This highlights the difficulty of enabling hierarchical reasoning in video understanding. Interestingly, we observe that lower-level visual and high-level action latent variables evolve at different rates, with low-level visual variables changing rapidly, while high-level action variables evolve more slowly, making them easier to identify. Building on this insight, we propose the Hierarchical Action Learning (HAL) model for weakly-supervised action segmentation. Our approach introduces a hierarchical causal data generation process, where high-level latent action governs the dynamics of low-level visual features. To model these varying timescales effectively, we introduce deterministic processes to align these latent variables over time. The HAL model employs a hierarchical pyramid transformer to capture both visual features and latent variables, and a sparse transition constraint is applied to enforce the slower dynamics of high-level action variables. This mechanism enhances the identification of these latent variables over time. Under mild assumptions, we prove that these latent action variables are strictly identifiable. Experimental results on several benchmarks show that the HAL model significantly outperforms existing methods for weakly-supervised action segmentation, confirming its practical effectiveness in real-world applications.
Figures & tables
Figure 1: An action segmentation example in the CrossTask dataset. (a) Real-world action videos exhibit different levels of representation, where action representations change more smoothly than visual representations. (b) A hierarchical data generation process of action videos, where high-level latent action variables govern the evolution of low-level visual variables. (c) Action segmentation results show that action-level segmentation better aligns with the ground truth than visual-level segmentation.
Figure 2: Illustration of the augmented data generation process and the framework of the HAL model. (a) The original data generation process is augmented by introducing pseudo-states and aligning the number of latent action with visual variables. The dashed arrows denote the unknown deterministic transitions. (b) The overall framework of HAL consists of a pyramidal transformer-based backbone for feature extraction, visual and action encoders for latent visual and action variables, visual and action decoder for reconstruction, and the smoothness transition constraint that enforces the identification of latent action variables.
Figure 3: Intuitions of the theoretical results, where the solid and dashed arrows denote the stochastic and deterministic transitions, respectively. (a) The identification of (vt,ct) can be achieved by leveraging five consecutive temporal observations. (b) By introducing the independent noises ϵt−1v,ϵtv,ϵt+1v , the stochastic transition processes vt−2→vt−1 and vt→vt+1 can be effectively transformed into corresponding deterministic transition processes.
Breakfast
Method
MoF
MoF-Bg
IoU
IoD
HMM+RNN [ 64 ]
33.3
-
-
-
TCFPN+ISBA [ 12 ]
38.4/36.4±1.0
38.4
24.2
40.6
NN-Viterbi [ 66 ]
43.0/39.7±2.4
-
-
-
D3TW [ 7 ]
45.7
-
-
-
CDFL [ 44 ]
50.2/48.1±2.5
48.0
33.7
45.4
Table 1: Experiment results on Breakfast dataset.
CrossTask
Method
MoF
MoF-Bg
IoU
IoD
NN-Viterbi [ 66 ]
26.5
-
10.7
24.0
CDFL [ 44 ]
31.9
-
11.5
23.8
TASL [ 54 ]
40.7
27.4
14.5
25.1
POC [ 55 ]
42.8
17.6
15.6
-
ATBA [ 78 ]
50.6±1.3
31.3±0.7
20.9±0.4
44.6 ±0.7
Table 2: Experiment results on the CrossTask dataset.
Hollywood Extended
Method
MoF
MoF-Bg
IoU
IoD
HMM+RNN [ 64 ]
-
-
11.9
-
TCFPN+ISBA [ 12 ]
28.7
34.5
12.6
18.3
D3TW [ 7 ]
33.6
-
-
-
CDFL [ 44 ]
45.0
40.6
19.5
25.8
TASL [ 54 ]
42.1
27.2
23.3
33.0
Table 3: Experiment results on the Hollywood dataset.
GTEA
Method
MoF
MoF-Bg
IoU
IoD
NN-Viterbi [ 66 ] ♯
25.7±2.5
25.9±2.8
13.0±1.8
18.9±2.3
CDFL [ 44 ] ♯
31.4±3.1
31.9±4.0
15.5±1.4
22.8±2.6
POC [ 55 ] ♯
18.4±4.5
22.2±6.1
7.3±2.7
11.2±3.6
ATBA [ 78 ] ♯
42.2±4.3
42.7±5.6
24.4±1.9
47.2±4.1
CtrlNS [ 71 ] ♯
33.3±3.9
33.1±5.9
18.9±2.1
38.3±4.4
Table 4: Experiment results on the GTEA dataset.
Figure 4: Qualitative results of P04-cam01-P04-pancake on the Breakfast dataset.
EXP
Lr
Ls
LKL
δ
MoF
IoU
IoD
1
-
-
-
-
53.3
40.1
58.7
2
✓
-
-
-
54.3
38.4
61.6
3
-
✓
-
-
54.6
40.3
61.6
4
-
-
✓
-
54.5
42.0
61.0
5
-
-
-
✓
53.9
41.1
59.4
6
✓
✓
✓
✓
56.6
42.6
62.1
Table 5: Ablation studies on the split 1 of Breakfast dataset.
Figure 5: Illustration of T-SNE Visualization of latent variables. (a) and (b) show scatter plots of the latent action and visual variables of the propsoed method, respectively. (c) shows the output of ATBA after dimensionality reduction.
Figure 6: Comparison of training and inference time between the proposed framework and baseline methods. The framework universally outperforms baselines in terms of faster inference speed and shorter training time across all experimental settings.
Figure 7: Sensitivity analysis of the hyperparameter for loss term Ls across datasets. Performance rises notably within a specific hyperparameter range and stabilizes thereafter, verifying the existence of a robust optimal hyperparameter range.
Configuration
Description
Output
1.ϕ
Transformer
Input x1:T
Input Feature
BatchSize×T×∣xt∣
Dense
256 neurons
BatchSize×T×256
2.ψ
Visual Encoder
Input z1:T
Feature Latent Variables
BatchSize×T×256
Dense
256 neurons; ReLU
BatchSize×T×256
Table 6: Architecture details. L:length of input feature, ∣xt∣ :the dimension of xt , ∣y∣ : the numbers of action class, ReLU:Rectified Linear Unit.
EXP
Lr
Ls
LKL
δ
MoF
IoU
IoD
1
53.3
40.9
58.7
2
✓
54.3
38.4
61.6
3
✓
54.6
40.3
61.6
4
✓
54.5
42.0
61.0
5
✓
53.9
41.1
59.4
6
✓
✓
54.7
39.3
61.4
Table 7: Experiment results on the GTEA dataset.
Figure 8: The F1 score with linear probing on Breakfast.
Figure 9: Frame-to-frame visual feature distance and ground-truth action boundary distribution. Visual features show large frequent fluctuations, and action boundaries arise after prolonged visual variations rather than coinciding with them.
Latent Action Models (LAMs) enable learning from actionless data for applications ranging from robotic control to interactive world models. However, existing LAMs typically focus on short-horizon frame transitions and capture low-level motion while overlooking longer-term temporal structure. In contrast, actionless videos often contain temporally extended and high-level skills. We present HiLAM, a hierarchical latent action model that discovers latent skills by modeling long-term temporal information. To capture these dependencies across long horizons, we utilize a pretrained LAM as a low-level extractor. This architecture aggregates latent action sequences, which contain the underlying dynamic patterns of the video, into high-level latent skills. Our experiments demonstrate that HiLAM improves over the baseline and exhibits robust dynamic skill discovery.
In this paper, we propose an HOI-aware adaptive network named AdaAct for weakly-supervised action segmentation. Most existing methods learn a fixed network to predict the action of each frame with the neighboring frames. However, this would result in ambiguity when estimating similar actions, such as pouring juice and pouring coffee. To address this, we aim to exploit temporally global but spatially local human-object interactions (HOI) as video-level prior knowledge for action segmentation. The long-term HOI sequence provides crucial contextual information to distinguish ambiguous actions, where our network dynamically adapts to the given HOI sequence at test time. More specifically, we first design a video HOI encoder that extracts, selects, and integrates the most representative HOI throughout the video. Then, we propose a two-branch HyperNetwork to learn an adaptive temporal encoder, which automatically adjusts the parameters based on the HOI information of various videos on the fly. Extensive experiments on two widely-used datasets including Breakfast and 50Salads demonstrate the effectiveness of our method under different evaluation metrics.
Runzhong Zhang, Suchen Wang, Yueqi Duan +3
1Nanyang Technological University · 2Tsinghua University · 3Beijing Jiaotong University
Temporal action segmentation (TAS) divides untrimmed videos into labeled action segments. While fully supervised methods have advanced the field, challenges such as action variability, ambiguous boundaries, and high annotation costs remain, especially in new or low-resource domains. Grammar-based approaches improve segmentation with structural priors but rely on complex parsing limiting scalability. In this work, we propose a lightweight, constraint-based refinement framework that enhances TAS predictions by integrating statistical structural priors such as transition confidence, action boundary sets, and per-class duration, that can be directly extracted from annotated data. These constraints are integrated into a modified Viterbi decoding algorithm, allowing inference-time refinement without retraining or added model complexity. Our approach improves both fully and semi-supervised TAS models by correcting structural prediction errors while maintaining high efficiency. Code is available at https://github.com/LUNAProject22/CAD
Yeo Keat Ee, Debaditya Roy, Chen Li +2
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1