Hierarchical Action Learning for Weakly-Supervised Action Segmentation

Junxian Huang; Ruichu Cai; Hao Zhu; Juntao Fang; Boyan Xu; Weilin Chen; Zijian Li; Shenghua Gao

cs.CVFeb 27, 2026

Hierarchical Action Learning for Weakly-Supervised Action Segmentation

Authors: Junxian Huang, Ruichu Cai, Hao Zhu, Juntao Fang, Boyan Xu, Weilin Chen, Zijian Li, Shenghua Gao

Organizations: Guangdong University of Technology · Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence · University of Hong Kong

Abstract

Humans perceive actions through key transitions that structure actions across multiple abstraction levels, whereas machines, relying on visual features, tend to over-segment. This highlights the difficulty of enabling hierarchical reasoning in video understanding. Interestingly, we observe that lower-level visual and high-level action latent variables evolve at different rates, with low-level visual variables changing rapidly, while high-level action variables evolve more slowly, making them easier to identify. Building on this insight, we propose the Hierarchical Action Learning (HAL) model for weakly-supervised action segmentation. Our approach introduces a hierarchical causal data generation process, where high-level latent action governs the dynamics of low-level visual features. To model these varying timescales effectively, we introduce deterministic processes to align these latent variables over time. The HAL model employs a hierarchical pyramid transformer to capture both visual features and latent variables, and a sparse transition constraint is applied to enforce the slower dynamics of high-level action variables. This mechanism enhances the identification of these latent variables over time. Under mild assumptions, we prove that these latent action variables are strictly identifiable. Experimental results on several benchmarks show that the HAL model significantly outperforms existing methods for weakly-supervised action segmentation, confirming its practical effectiveness in real-world applications.

Figures & tables

Figure 1: An action segmentation example in the CrossTask dataset. (a) Real-world action videos exhibit different levels of representation, where action representations change more smoothly than visual representations. (b) A hierarchical data generation process of action videos, where high-level latent action variables govern the evolution of low-level visual variables. (c) Action segmentation results show that action-level segmentation better aligns with the ground truth than visual-level segmentation.

Figure 2: Illustration of the augmented data generation process and the framework of the HAL model. (a) The original data generation process is augmented by introducing pseudo-states and aligning the number of latent action with visual variables. The dashed arrows denote the unknown deterministic transitions. (b) The overall framework of HAL consists of a pyramidal transformer-based backbone for feature extraction, visual and action encoders for latent visual and action variables, visual and action decoder for reconstruction, and the smoothness transition constraint that enforces the identification of latent action variables.

Figure 3: Intuitions of the theoretical results, where the solid and dashed arrows denote the stochastic and deterministic transitions, respectively. (a) The identification of

(\mathbf{v}_{t},\mathbf{c}_{t})

can be achieved by leveraging five consecutive temporal observations. (b) By introducing the independent noises

\epsilon^{v}_{t-1},\epsilon^{v}_{t},\epsilon^{v}_{t+1}

, the stochastic transition processes

\mathbf{v}_{t-2}\rightarrow\mathbf{v}_{t-1}

and

\mathbf{v}_{t}\rightarrow\mathbf{v}_{t+1}

can be effectively transformed into corresponding deterministic transition processes.

Breakfast
Method	MoF	MoF-Bg	IoU	IoD
HMM+RNN [ 64 ]	33.3	-	-	-
TCFPN+ISBA [ 12 ]	38.4/36.4±1.0	38.4	24.2	40.6
NN-Viterbi [ 66 ]	43.0/39.7±2.4	-	-	-
D3TW [ 7 ]	45.7	-	-	-
CDFL [ 44 ]	50.2/48.1±2.5	48.0	33.7	45.4

Table 1: Experiment results on Breakfast dataset.

CrossTask
Method	MoF	MoF-Bg	IoU	IoD
NN-Viterbi [ 66 ]	26.5	-	10.7	24.0
CDFL [ 44 ]	31.9	-	11.5	23.8
TASL [ 54 ]	40.7	27.4	14.5	25.1
POC [ 55 ]	42.8	17.6	15.6	-
ATBA [ 78 ]	50.6±1.3	31.3±0.7	20.9±0.4	44.6 ±0.7

Table 2: Experiment results on the CrossTask dataset.

Hollywood Extended
Method	MoF	MoF-Bg	IoU	IoD
HMM+RNN [ 64 ]	-	-	11.9	-
TCFPN+ISBA [ 12 ]	28.7	34.5	12.6	18.3
D3TW [ 7 ]	33.6	-	-	-
CDFL [ 44 ]	45.0	40.6	19.5	25.8
TASL [ 54 ]	42.1	27.2	23.3	33.0

Table 3: Experiment results on the Hollywood dataset.

GTEA
Method	MoF	MoF-Bg	IoU	IoD
NN-Viterbi [ 66 ] $\sharp$	25.7±2.5	25.9±2.8	13.0±1.8	18.9±2.3
CDFL [ 44 ] $\sharp$	31.4±3.1	31.9±4.0	15.5±1.4	22.8±2.6
POC [ 55 ] $\sharp$	18.4±4.5	22.2±6.1	7.3±2.7	11.2±3.6
ATBA [ 78 ] $\sharp$	42.2±4.3	42.7±5.6	24.4±1.9	47.2±4.1
CtrlNS [ 71 ] $\sharp$	33.3±3.9	33.1±5.9	18.9±2.1	38.3±4.4

Table 4: Experiment results on the GTEA dataset.

Figure 4: Qualitative results of P04-cam01-P04-pancake on the Breakfast dataset.

EXP	$\mathcal{L}_{r}$	$\mathcal{L}_{s}$	$\mathcal{L}_{KL}$	$\delta$	MoF	IoU	IoD
1	-	-	-	-	53.3	40.1	58.7
2	✓	-	-	-	54.3	38.4	61.6
3	-	✓	-	-	54.6	40.3	61.6
4	-	-	✓	-	54.5	42.0	61.0
5	-	-	-	✓	53.9	41.1	59.4
6	✓	✓	✓	✓	56.6	42.6	62.1

Table 5: Ablation studies on the split 1 of Breakfast dataset.

Figure 5: Illustration of T-SNE Visualization of latent variables. (a) and (b) show scatter plots of the latent action and visual variables of the propsoed method, respectively. (c) shows the output of ATBA after dimensionality reduction.

Figure 6: Comparison of training and inference time between the proposed framework and baseline methods. The framework universally outperforms baselines in terms of faster inference speed and shorter training time across all experimental settings.

Figure 7: Sensitivity analysis of the hyperparameter for loss term

\mathcal{L}_{s}

across datasets. Performance rises notably within a specific hyperparameter range and stabilizes thereafter, verifying the existence of a robust optimal hyperparameter range.

Configuration	Description	Output
$1.\phi$	Transformer
Input $\mathbf{x}_{1:T}$	Input Feature	$BatchSize\times T\times\|\mathbf{x}_{t}\|$
Dense	256 neurons	$BatchSize\times T\times 256$
$2.\psi$	Visual Encoder
Input $\mathbf{z}_{1:T}$	Feature Latent Variables	$BatchSize\times T\times 256$
Dense	256 neurons; ReLU	$BatchSize\times T\times 256$

Table 6: Architecture details. L:length of input feature,

|\mathbf{x}_{t}|

:the dimension of

\mathbf{x}_{t}

|y|

: the numbers of action class, ReLU:Rectified Linear Unit.

EXP	$\mathcal{L}_{r}$	$\mathcal{L}_{s}$	$\mathcal{L}_{KL}$	$\delta$	MoF	IoU	IoD
1					53.3	40.9	58.7
2	✓				54.3	38.4	61.6
3		✓			54.6	40.3	61.6
4			✓		54.5	42.0	61.0
5				✓	53.9	41.1	59.4
6		✓	✓		54.7	39.3	61.4

Table 7: Experiment results on the GTEA dataset.

Figure 8: The F1 score with linear probing on Breakfast.

Figure 9: Frame-to-frame visual feature distance and ground-truth action boundary distribution. Visual features show large frequent fluctuations, and action boundaries arise after prolonged visual variations rather than coinciding with them.

Panel of Figure 10 — Figure 10: More qualitative results.

Temporal Action Segmentation

Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.

See all 49 papers

Hierarchical Latent Action Model
Mar 6, 2026Hanjung Kim, Lerrel Pinto, Seon Joo KimLatent Action Learning Agent Skill Learning
Yonsei University · New York University
HOI-aware Adaptive Network for Weakly-supervised Action Segmentation
Apr 29, 2026Runzhong Zhang, Suchen Wang, Yueqi Duan +3Temporal Action Segmentation Human Activity Recognition
1Nanyang Technological University · 2Tsinghua University · 3Beijing Jiaotong University
Improving Temporal Action Segmentation via Constraint-Aware Decoding
May 11, 2026Yeo Keat Ee, Debaditya Roy, Chen Li +2Temporal Action Segmentation Constrained Decoding
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1

Mar 6, 2026J/K move · Enter open · S save

Hierarchical Latent Action Model

Hanjung Kim, Lerrel Pinto, Seon Joo Kim

Yonsei University · New York University

Latent Action Learning Agent Skill Learning Hierarchical Representation Learning

Open paper AskPDF arXiv

Latent Action Models (LAMs) enable learning from actionless data for applications ranging from robotic control to interactive world models. However, existing LAMs typically focus on short-horizon frame transitions and capture low-level motion while overlooking longer-term temporal structure. In contrast, actionless videos often contain temporally extended and high-level skills. We present HiLAM, a hierarchical latent action model that discovers latent skills by modeling long-term temporal information. To capture these dependencies across long horizons, we utilize a pretrained LAM as a low-level extractor. This architecture aggregates latent action sequences, which contain the underlying dynamic patterns of the video, into high-level latent skills. Our experiments demonstrate that HiLAM improves over the baseline and exhibits robust dynamic skill discovery.

Abstract

Temporal Action Segmentation

Explore similar work

Hierarchical Latent Action Model

HOI-aware Adaptive Network for Weakly-supervised Action Segmentation

Improving Temporal Action Segmentation via Constraint-Aware Decoding