ConsensusTAS: Self-Supervised Temporal Action Segmentation for Long-Horizon Construction Videos

Xiaoshan Zhou; Yafei Sun

cs.CVAug 25, 2026

ConsensusTAS: Self-Supervised Temporal Action Segmentation for Long-Horizon Construction Videos

Authors: Xiaoshan Zhou, Yafei Sun

Organizations: School of Project Management, University of Sydney, Australia · School of Civil and Environmental Engineering, University of New South Wales, Australia

Abstract

Recognizing sequential construction activities is important for collaborative human-robot work; for example, robots are able to understand workers' current and upcoming actions and provide timely tool delivery or physical support. However, despite extensive research on construction worker activity recognition, existing studies have been limited to classifying activity categories, such as climbing, lifting, and walking, instead of recognizing fine-grained activity transitions from long-horizon sequences. Addressing this problem is challenging because annotating action temporal boundaries in long construction videos is time-consuming. In this study, we propose ConsensusTAS, a label-free, self-supervised learning approach to segment continuous video streams into distinct activity phases by exploiting the internal consensus of candidate segmentations. We evaluated our algorithm on three public datasets, where it outperformed state-of-the-art methods, achieving an F1@10 of 73.08 on GTEA, an F1@10 of 64.33 on Breakfast, and an F1@50 of 33.50 on static-camera videos from Assembly101. We also tested it on real-world construction videos, where post-hoc evaluation showed that the model successfully recognized and segmented actions within the composite activity of bricklaying, such as spreading mortar on a brick, placing the brick, pressing, and aligning. Compared with other temporal action segmentation models that require computationally intensive large vision-language models, our method can run on a CPU, which provides practical value for video surveillance and human-robot collaboration on mobile robotic platforms.

Figures & tables

Panel of Figure 1 — Fig. 1: Sample images from the GETA dataset.

Method	F1@10
ConsensusTAS	$\mathbf{73.08\pm 2.18}$
ASESM, manually selected interval	70.4
ASESM, automatic interval	57.0
SemiTAS comparison, 5% labels	59.8
SemiTAS comparison, 10% labels	71.5

TABLE I: GTEA results over three seeds and four folds. Baseline results are reported by Aouaidjia et al. [ 9 ] .

Method	F1@10	F1@25	F1@50
TSA, published	58.0	–	–
SaM, published	55.9	–	–
Uniform 300, local	63.09	58.41	33.60
Uniform 350, local	63.36	59.18	33.38
ASESM, local	63.12	58.59	32.77
ASESM, published	63.2	–	–

TABLE II: Breakfast results over seeds and official folds.

Method	F1@10	F1@25	F1@50
Uniform	48.94	40.06	18.05
Change point	47.69	43.81	23.78
KMeans + smoothing	$40.62\pm 0.04$	$37.01\pm 0.06$	$20.41\pm 0.27$
Adapted ASOT	$42.56\pm 0.37$	$39.50\pm 0.39$	$22.25\pm 0.37$
Local full ASOT	$28.84\pm 4.32$	$23.60\pm 4.92$	$11.90\pm 2.88$
ConsensusTAS	$54.79\pm 0.10$	$52.60\pm 0.08$	$\mathbf{33.50\pm 0.24}$

TABLE III: Assembly101 test results on 167 eligible activity crops.

ASESM configuration	F1@10	F1@25	F1@50
Label-free selected: $K=16$ , gap 30	11.40	4.55	0.97
Grid sensitivity: $K=16$ , gap 225	$\mathbf{59.81}$	$\mathbf{53.55}$	26.21
ConsensusTAS	54.79	52.60	$\mathbf{33.50}$

TABLE IV: Assembly101 test results on 167 eligible activity crops.

Variant	F1@10	F1@25	F1@50	Interpretation
Full ConsensusTAS	54.79	52.60	33.50	Reference
No consistency reward	54.80	52.60	32.96	Small strict-overlap loss
No foreground suppression	55.49	53.42	34.38	Foreground term unfavorable
Population 12	55.00	52.83	33.18	Large population not essential
Unweighted density	54.76	52.59	33.52	Weighting has little effect
No density decoder	48.27	45.73	28.42	Large degradation

TABLE V: Assembly101 ablations. Test-set analyses are diagnostic.

Fig. 4: Timeline and consensus density for the bricklaying video.

Fig. 5: Qualitative visualization of the nine anonymous phases discovered by ConsensusTAS in an unlabeled bricklaying video.

Phase	Interval	Visual interpretation
1	0.0–7.0 s	Holding the brick, collecting and spreading mortar
2	7.0–9.3 s	Completing mortar application
3	9.3–10.0 s	Transition into placement
4	10.0–13.2 s	Inserting and initially positioning the closure brick
5	13.2–15.6 s	Pressing and aligning the brick
6	15.6–16.7 s	Short joint or edge adjustment

TABLE VI: Tentative visual interpretation of the identified phases.

Temporal Action Segmentation

Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.

See all 49 papers

May 11, 2026

Improving Temporal Action Segmentation via Constraint-Aware Decoding

Temporal action segmentation (TAS) divides untrimmed videos into labeled action segments. While fully supervised methods have advanced the field, challenges such as action variability, ambiguous boundaries, and high annotation costs remain, especially in new or low-resource domains. Grammar-based approaches improve segmentation with structural priors but rely on complex parsing limiting scalability. In this work, we propose a lightweight, constraint-based refinement framework that enhances TAS predictions by integrating statistical structural priors such as transition confidence, action boundary sets, and per-class duration, that can be directly extracted from annotated data. These constraints are integrated into a modified Viterbi decoding algorithm, allowing inference-time refinement without retraining or added model complexity. Our approach improves both fully and semi-supervised TAS models by correcting structural prediction errors while maintaining high efficiency. Code is available at https://github.com/LUNAProject22/CAD

Yeo Keat Ee, Debaditya Roy, Chen Li +2

Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1

Temporal Action Segmentation Constrained Decoding

Ask PDF

Apr 16, 2026

Boundary-Centric Clip-Budgeted Active Learning for Temporal Action Segmentation

Temporal action segmentation (TAS) in untrimmed videos requires dense temporal supervision. However, most of the annotation cost is spent identifying action transitions where segmentation errors concentrate and small temporal shifts can disproportionately degrade segment-level metrics. We introduce B-ACT, a clip-budgeted active learning framework that explicitly allocates supervision to these error-prone boundary regions. B-ACT operates in a hierarchical two-stage loop: (i) it ranks and queries unlabeled videos using predictive uncertainty, and (ii) within each selected video, it detects candidate transitions from the current model predictions and selects the top-

K

boundaries via a novel boundary score. The boundary score fuses neighborhood uncertainty, class ambiguity, and temporal prediction dynamics to reveal the underlying importance of each frame. Importantly, our annotation protocol requests labels only at the boundary frames while still training on boundary-centered clips to exploit temporal context through the model's receptive field. Extensive experiments on GTEA, 50Salads, and Breakfast demonstrate that boundary-centric supervision delivers strong label efficiency and consistently surpasses representative TAS active learning baselines and prior state of the art under sparse budgets. Gains are largest on datasets where performance is highly sensitive to boundary placement, as measured by edit and overlap-based F1 metrics.

Halil Ismail Helvaci, Sen-ching Samson Cheung

Department of Electrical and Computer Engineering, University of Kentucky, Lexington, KY, USA

Temporal Action Segmentation Active Learning Boundary-Aware Segmentation

Ask PDF

Aug 30, 2026

See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning

Unsupervised action segmentation aims to discover latent action categories and their temporal organization without action annotations. Optimal transport-based methods provide structured frame-to-action assignments, however, their pseudo-label quality is fundamentally conditioned on the representation space used to construct the transport cost. We argue that reliable OT pseudo-labeling requires a representation geometry that is simultaneously sensitive to discriminative action changes and coherent along local temporal progressions. Based on this insight, we propose SpecT-OT, a spectral-temporal representation learning framework built upon an unbalanced optimal transport pseudo-labeling concept. SpecT-OT introduces a Spectral Reparameterization Projector (SRP), which parameterizes projector weights with fixed Fourier bases and learnable coefficients to improve the modeling of rapidly varying discriminative features, and Temporal Affinity Regularization (TAR), which imposes distance-aware, label-free constraints on pairwise frame affinities to stabilize local temporal structure. The two components jointly produce more discriminative and temporally stable transport costs, yielding more reliable pseudo-labels for iterative representation learning. Experiments on four benchmarks demonstrate strong performance compared with state-of-the-art methods. SpecT-OT achieves the best results on 13 of 15 metrics, including 4.1-point MoF and 7.4-point F1 gains over the baseline on Breakfast and Desktop Assembly, respectively.

Yun Li, Jun Xiao, Cong Zhang +1

School of Control Science and Engineering, Shandong University

Temporal Action Segmentation Temporal Representation Learning Unsupervised Representation Learning Video Representation Learning Optimal Transport

Ask PDF

Abstract

Temporal Action Segmentation

Explore similar work

Improving Temporal Action Segmentation via Constraint-Aware Decoding

Boundary-Centric Clip-Budgeted Active Learning for Temporal Action Segmentation

See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning