Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation

Hinako Mitsuoka; Kazuhiro Hotta

cs.CVApr 2, 2026

Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation

Authors: Hinako Mitsuoka, Kazuhiro Hotta

Organizations: Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan

Abstract

Recent progress in Temporal Action Segmentation (TAS) has increasingly relied on complex architectures, which can hinder practical deployment. We present a lightweight dual-loss training framework that improves fine-grained segmentation quality with only one additional output channel and two auxiliary loss terms, requiring minimal architectural modification. Our approach combines a boundary-regression loss that promotes accurate temporal localization via a single-channel boundary prediction and a CDF-based segment-level regularization loss that encourages coherent within-segment structure by matching cumulative distributions over predicted and ground-truth segments. The framework is architecture-agnostic and can be integrated into existing TAS models (e.g., MS-TCN, C2F-TCN, FACT) as a training-time loss function. Across three benchmark datasets, the proposed method improves segment-level consistency and boundary quality, yielding higher F1 and Edit scores across three different models. Frame-wise accuracy remains largely unchanged, highlighting that precise segmentation can be achieved through simple loss design rather than heavier architectures or inference-time refinements.

Figures & tables

Figure 1: Overview of the proposed method and its performance improvement. The left shows that the proposed method adds a single boundary output channel and introduces two auxiliary training losses that can be integrated into conventional TAS models with minimal modification. The right shows the relationship between the F1 score and the number of parameters on GTEA, illustrating improved performance with minimal parameter overhead.

Figure 2: Overview of the proposed training framework. In addition to the model-specific loss, our method introduces two complementary auxiliary losses: a boundary-regression loss using an additional boundary output channel and a CDF-based segment shape regularization loss. The losses are applied selectively to different temporal regions (boundary vs non-boundary) to reduce optimization conflicts, and can be combined with existing TAS models with minimal architectural modification.

Figure 3: Visualization of Boundary-Regression Loss $\mathcal{L}_{B}$ . The additional output channel predicts a boundary probability curve ( orange ), which is supervised by a binary ground-truth boundary mask (top). The loss encourages high responses around class transitions while maintaining low values elsewhere. Segment labels (bottom) are color-coded for clarity.

Figure 4: Visualization of the CDF-based segment shape regularization loss $\mathcal{L}_{\text{S}}$ . Each ground-truth segment (top) is compared with its corresponding predicted region (bottom) using cumulative distributions. The loss penalizes structural mismatches such as over-segmentation or fragmented predictions by measuring discrepancies between cumulative probability distributions within each segment.

Methods	Dataset	$\lambda_{\text{B}}$	$\lambda_{\text{S}}$	$E_{\text{start}}$	$E_{\text{max}}$
MS-TCN	GTEA	$1.0\times 10^{-4}$	$1.0\times 10^{-3}$	20	50
	50Salads	$1.0\times 10^{-3}$
	Breakfast	$1.0\times 10^{-4}$
C2F-TCN	GTEA	$1.0\times 10^{-4}$	$1.0\times 10^{-3}$	200	500
	50Salads	$1.0\times 10^{-3}$
	Breakfast	$1.0\times 10^{-5}$

Table 1: Hyperparameters used for each model and dataset.

Dataset	GTEA					50Salads					Breakfast
Methods	F1@{10,25,50}			Edit	Acc	F1@{10,25,50}			Edit	Acc	F1@{10,25,50}			Edit	Acc
For Standard Architecture
MS-TCN [ 6 ]	86.47	83.73	71.20	79.92	76.53	73.96	71.42	62.12	66.29	79.11	50.61	46.41	36.58	61.75	66.96
+ Ours	88.13	84.62	72.40	83.23	78.20	77.44	75.22	67.31	70.29	81.17	55.96	51.39	40.78	62.29	67.46
For Complex Architecture
C2F-TCN [ 19 ]	87.80	86.38	76.72	81.14	80.32	84.03	82.65	74.37	76.76	86.63	71.41	67.80	57.28	67.84	76.10

Table 2: Performance comparison on GTEA, 50Salads and Breakfast. We apply our method to MS-TCN, C2F-TCN, and FACT.

Methods	F1@{10,25,50}			Edit	Acc	MACs(G)	Params(M)
MS-TCN [ 6 ]	86.47	83.73	71.20	79.92	76.53	0.80	0.80
ASRF [ 8 ]	86.52	85.27	73.68	78.59	76.01	1.30	1.30
BCN [ 22 ]	88.07	85.88	75.73	82.26	79.95	16.64	16.66
MS-TCN + Ours	88.13	84.62	72.40	83.23	78.20	0.80	0.80

Table 3: Comparison with existing boundary-aware methods (ASRF, BCN) and the proposed method on GTEA. We report performance metrics along with computational cost (MACs) and model size (Params).

Figure 5: Qualitative comparison on GTEA, 50Salads, and Breakfast using MS-TCN as the backbone. For each dataset, we show the ground-truth (GT), baseline MS-TCN predictions, and MS-TCN trained with the proposed losses.

Figure 6: The trade-off between Params(M) and F1@10 for conventional boundary-aware methods and our method, on GTEA.

$\mathcal{L}_{\text{B}}$	$\mathcal{L}_{\text{S}}$	F1@{10,25,50}			Edit	Acc
		86.47	83.73	71.20	79.92	76.53
$\surd$		87.84	85.24	72.16	82.83	77.69
	$\surd$	87.36	85.55	72.73	82.42	77.30
$\surd$	$\surd$	88.13	84.62	72.40	83.23	78.20

Table 4: Ablation study on GTEA with MS-TCN. We evaluate the individual and joint contributions of

\mathcal{L}_{\text{B}}

and

\mathcal{L}_{\text{S}}

$E_{\text{start}}$	F1@{10,25,50}			Edit	Acc
0	81.53	77.35	60.63	75.84	75.13
10	86.45	84.43	72.37	83.62	76.16
20	88.13	84.62	72.40	83.23	78.20
30	86.86	84.81	71.71	81.71	77.77

Table 5: Ablation study on the start epoch

E_{\text{start}}

for applying Segment Shape Regularization Loss on GTEA using MS-TCN. We evaluate how the timing of introducing

\mathcal{L}_{\text{S}}

affects performance, while keeping all other settings identical.

Methods	F1@{10,25,50}			Edit	Acc
All frames	87.49	85.08	72.44	81.61	77.64
Ours	88.13	84.62	72.40	83.23	78.20

Table 6: Ablation study on loss assignment strategy using MS-TCN on GTEA . ”All frames” applies both

\mathcal{L}_{\text{B}}

and

\mathcal{L}_{\text{S}}

to all frames uniformly, while ”Ours” follows our proposed boundary-aware strategy:

\mathcal{L}_{\text{B}}

is applied only near ground-truth boundaries, and

\mathcal{L}_{\text{S}}

is applied to non-boundary regions.

Figure 7: Qualitative results of “All frames” vs. “Ours” on GTEA. Boundary-aware loss assignment improves alignment and segmentation quality, especially near action boundaries.

Dataset	GTEA					50Salads					Breakfast
Methods	F1@{10,25,50}			Edit	Acc	F1@{10,25,50}			Edit	Acc	F1@{10,25,50}			Edit	Acc
MS-TCN [ 6 ]	86.47	83.73	71.20	79.92	76.53	73.96	71.42	62.12	66.29	79.11	50.61	46.41	36.58	61.75	66.96
+ Ours	88.13	84.62	72.40	83.23	78.20	77.44	75.22	67.31	70.29	81.17	55.96	51.39	40.78	62.29	67.46
+ ASOT [ 23 ]	87.81	85.49	73.30	81.65	76.61	81.36	79.17	70.16	74.30	79.49	46.91	42.50	32.01	44.94	61.10
+ ASOT + $\mathcal{L}_{\text{B}}$	88.12	85.99	73.95	83.33	78.04	82.78	80.97	71.69	74.77	81.27	45.13	40.89	30.36	42.89	59.41
+ ASOT + $\mathcal{L}_{\text{S}}$	88.03	84.92	73.44	82.01	77.03	79.78	77.79	69.65	71.86	78.43	49.25	44.80	34.28	46.33	61.90

Table 7: Ablation study on synergy with post-processing method using MS-TCN on GTEA, 50Salads and Breakfast. We apply ASOT [ 23 ] as a post-processing step to both the baseline and our model trained with the proposed losses.

Temporal Action Segmentation

Temporal action segmentation labels when actions begin and end in continuous video, enabling fine-grained analysis of human and robotic activity.

See all 49 papers

Improving Temporal Action Segmentation via Constraint-Aware Decoding
May 11, 2026Yeo Keat Ee, Debaditya Roy, Chen Li +2Temporal Action Segmentation Constrained Decoding
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India +1
Boundary-Centric Clip-Budgeted Active Learning for Temporal Action Segmentation
Apr 16, 2026Halil Ismail Helvaci, Sen-ching Samson CheungTemporal Action Segmentation Active Learning
Department of Electrical and Computer Engineering, University of Kentucky, Lexington, KY, USA
Boundary Voting Network for Ambiguity-Aware Timestamp-Supervised Action Segmentation
Sep 8, 2026Runzhong Zhang, Yueqi Duan, Yang Chen +4Temporal Action Segmentation Boundary-Aware Segmentation
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Electronic Engineering, Tsinghua University, Beijing, China · Amazon, Seattle, America

May 11, 2026J/K move · Enter open · S save

Improving Temporal Action Segmentation via Constraint-Aware Decoding

Yeo Keat Ee, Debaditya Roy, Chen Li, Hao Zhang+1

Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Indian Institute of Technology Kharagpur, India+1

Temporal Action Segmentation Constrained Decoding

Open paper AskPDF arXiv

Temporal action segmentation (TAS) divides untrimmed videos into labeled action segments. While fully supervised methods have advanced the field, challenges such as action variability, ambiguous boundaries, and high annotation costs remain, especially in new or low-resource domains. Grammar-based approaches improve segmentation with structural priors but rely on complex parsing limiting scalability. In this work, we propose a lightweight, constraint-based refinement framework that enhances TAS predictions by integrating statistical structural priors such as transition confidence, action boundary sets, and per-class duration, that can be directly extracted from annotated data. These constraints are integrated into a modified Viterbi decoding algorithm, allowing inference-time refinement without retraining or added model complexity. Our approach improves both fully and semi-supervised TAS models by correcting structural prediction errors while maintaining high efficiency. Code is available at https://github.com/LUNAProject22/CAD

Abstract

Temporal Action Segmentation

Explore similar work

Improving Temporal Action Segmentation via Constraint-Aware Decoding

Boundary-Centric Clip-Budgeted Active Learning for Temporal Action Segmentation

Boundary Voting Network for Ambiguity-Aware Timestamp-Supervised Action Segmentation