cs.CVNov 19, 2026

What Places an Action Boundary? Structure-Aware Attribution for Temporal Action Localization

Authors: Ruoshan Lan, Zongru Shao, Chieh Tsai, Mihai Surdeanu

Organizations: Department of Computer Science, University of Arizona · Silicon Austria Labs · Department of Electrical and Computer Engineering, University of Arizona

Abstract

Temporal action localization (TAL) detectors must identify actions and estimate their temporal boundaries, but localization remains a major source of detection errors. While understanding what determines boundary placement requires faithful attribution of temporal evidence, existing attribution methods have not sufficiently addressed the temporal structure of TAL or the need to tailor attribution to boundary localization. We introduce a structure-aware attribution framework that averages contributions across random evidence-restoration orders following the Shapley-Shubik principle. We make it tractable by using the detector's receptive field to limit which time steps are ordered individually, and by approximating each order with a soft gradient path. For boundary predictions, we further introduce cell-structured restoration orders and boundary-anchored references that selectively remove adjacent evidence. We also adapt faithfulness evaluation to assess the joint effects of contiguous temporal evidence and the direction of boundary displacement. Experiments on THUMOS14 and EPIC-KITCHENS-100 with TriDet and ActionFormer demonstrate improved attribution faithfulness over existing methods. Attribution analysis and controlled interventions further reveal that class-score changes do not reliably track boundary displacement, providing input-level evidence that classification relies on content less informative about boundary placement. Code and evaluation artifacts will be publicly released.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Sep 18, 2026

PSEE: Progressive Sensor Event Expansion for Point-Supervised Temporal Action Localization

Temporal action localization (TAL) in wearable sensor streams identifies action classes and temporal boundaries, enabling finer-grained activity understanding than conventional action recognition. However, training typically requires costly start--end annotations for every action instance. To reduce this burden, we study point-supervised TAL, where each instance is labeled with only one timestamp and its class. We propose Progressive Sensor Event Expansion (PSEE), which combines semantic activations, sensor-specific transition evidence, and adaptive temporal ownership to recover point-supervised pseudo segments. These segments supervise standard TAL detectors without modifying their inference procedures. Cross-subject experiments on four inertial-sensing benchmarks demonstrate improved pseudo-boundary quality over adapted point-supervised baselines, compatibility with different TAL detectors, and robustness to point sampling. Code is available at https://github.com/joeeeeyin/PSEE.
Oct 23, 2026

DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization

Point-supervised Temporal Action Localization (PTAL) aims to localize action instances using only a single annotated timestamp for each instance. Such sparse supervision offers limited guidance for modeling the structural organization of an action instance. Existing PTAL methods primarily infer action regions from classification responses, leaving the structural organization of multiple action-related parts within each instance insufficiently modeled. To address this problem, we propose DeformStruct, an instance-level deformable structure modeling framework for PTAL. DeformStruct progressively constructs the structure of each action instance through three stages. First, Structural Anchor Mining discovers multiple anchors from annotated points and action-responsive regions, providing structural nodes grounded in the current instance. Second, Deformable Part Alignment associates latent structural parts with these anchors through deformation-aware matching, allowing their temporal supports to adapt to instance-dependent variations. Third, Relational Structural Organization explicitly models the relative temporal relations among the aligned parts, forming a relational representation that characterizes how different action-related parts are organized together. Extensive experiments on THUMOS14, ActivityNet-v1.3, GTEA, and BEOID demonstrate consistent improvements in point-supervised temporal localization and validate the effectiveness of deformable structural modeling.
May 28, 2026

Masked Diffusion Vision-Language Models for Temporal Action Localization

Temporal action localization (TAL) requires recognizing the target event and localizing its start and end times precisely in untrimmed videos. Recent vision-language formulations improve semantic reasoning and support language-conditioned outputs, but their autoregressive decoders still generate tokens from left to right, preventing later semantic evidence from revising earlier timestamp predictions. We adapt masked diffusion vision-language models (MDVLMs) to TAL so that semantic tokens and boundary tokens remain editable throughout iterative denoising with bidirectional attention, allowing temporal boundaries and semantic content to be refined jointly. Direct adaptation, however, creates two TAL-specific mismatches: standard masked diffusion training corrupts all positions uniformly at random, but the time tokens are more reliable when enough semantic context is available; and token-level cross-entropy does not reflect temporal IoU. To address these mismatches, we introduce a Planned Training Objective that uses boundary-aware masking and step-weighted reconstruction to rehearse the late recovery of time tokens, together with a Step-Level IoU Reward that provides overlap-aware supervision during denoising. A standard sequence-level cross-entropy term provides the base reconstruction signal. Experiments on ActivityNet-RTL, ActivityNet-1.3, and THUMOS-14 show that MDVLM-TAL improves both temporal reasoning and boundary localization over autoregressive vision-language baselines, with especially strong gains under stricter temporal IoU criteria. The source code is available at https://github.com/ycwfs/MDVLM-TAL .