DUGO: Dual-Scale Gaussian Memory for Online Temporal Action Localization in Egocentric Procedural Videos
Authors: Mohammed Aman Bhuiyan, Md. Sazzad Hossain Adib, Samiul Basir Bhuiyan, Mir Faiyaz Hossain, Syed Mohammed Shamsul Islam, Md Zakir Hossain, Shafin Rahman
Organizations: Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh · School of Science, Edith Cowan University, Australia · School of Engineering, Computing and Mathematics, The University of Western Australia, Australia · Curtin Medical School, Curtin University, Australia · School of Systems and Computing, University of New South Wales, Australia
Online Temporal Action Localization (OnTAL) aims to detect and localize action instances from untrimmed streaming videos in real time. Unlike offline approaches, OnTAL models have no access to future frames and cannot go back and refine earlier predictions, which makes the task considerably harder given the strict real-time constraints and narrow temporal window available. Most existing OnTAL methods lean on either brief anchor-based context or a single-scale memory of past frames, which limits how well they can capture actions unfolding across multiple time scales or requiring long-range reasoning in Procedural Videos. To address these gaps, we present DUGO (Dual-Scale Encoding and Gaussian Memory Online), a new OnTAL framework that jointly models local motion dynamics and long-range temporal dependencies. At its core, DUGO uses a Dual-Scale Temporal Encoder, pairing depth-wise local representations with global self-attention so it can pick up both fine-grained temporal patterns and broader contextual cues. We also introduce a Gaussian Memory Bank, which represents action memory as stochastic, input-conditioned Gaussian prototypes, allowing for richer temporal memory through gated cross-attention applied at both the encoder and anchor-query stages. Extensive experiments on procedural egocentric datasets (EGTEA, EPIC-Kitchens-100) demonstrate that DUGO achieves state-of-the-art performance while also providing substantial improvements on non-egocentric benchmarks such as Muses. The project page is available at: https://aman554-eq.github.io/DUGO
Figures & tables
Fig. 1: Motivation. Egocentric procedural videos require fine-grained motion discrimination, memory of prior steps, and procedure-aware framing. Prior OnTAL methods rely on single-scale encoding with absent or one-shot deterministic memory; DUGO introduces dual-scale encoding and a twice-injected stochastic Gaussian memory that preserves multi-modal procedural ambiguity.
Fig. 2: DUGO Architecture. DSE replaces the single-granularity encoder with dual branches consisting of depthwise temporal convolutions for local modeling and self-attention for global temporal reasoning, followed by fusion through a learned projection. GMB replaces deterministic history compression with a stochastic mixture of Gaussian prototypes injected at two stages: encoder memory refinement (Pass 1) and decoded anchor query enhancement (Pass 2) through per-head gated cross-attention.
EGTEA [ 26 ]
EPIC-Kitchens-100 [ 27 ]
Model
0.1
0.2
0.3
0.4
0.5
Avg.
0.1
0.2
0.3
0.4
0.5
Avg.
MATR [ 24 ]
21.5
19.5
15.9
12.6
8.6
15.6
17.2
14.9
12.8
10.0
8.7
12.7
OAT [ 20 ]
24.9
23.1
20.6
17.1
12.2
19.6
17.8
16.3
14.3
12.8
10.1
14.2
HAT [ 21 ]
27.5
25.4
22.6
18.3
13.5
21.5
18.3
17.0
15.8
13.9
11.5
15.3
Ours
30.9
28.8
25.6
21.1
14.9
24.3 ∗
20.1
18.9
17.0
15.0
11.7
16.6 ∗
TABLE I: Comparison of mAP (%) across tIoU thresholds on EGTEA and EPIC-Kitchens-100 datasets. ∗ Indicates best result.
Model
0.3
0.4
0.5
0.6
0.7
Avg.
CAG-QIL [ 19 ]
8.5
6.5
4.2
2.8
1.9
4.8
OAT [ 20 ]
16.7
13.0
10.0
6.3
3.2
9.8
HAT [ 21 ]
19.1
14.7
10.1
6.3
3.7
10.8
MATR [ 24 ]
23.5
19.3
14.3
9.4
5.7
14.4
Ours
24.7
20.0
15.5
9.8
4.5
14.9 ∗
TABLE II: mAP (%) comparison on MUSES [ 28 ] . ∗ Our final configuration.
TABLE IV: Loss function analysis on EGTEA dataset. ∗ Our final configuration.
Fig. 3: Gaussian prototypes Kg analysis of DUGO on EGTEA and EPIC-Kitchens-100 datasets across various tIoU thresholds.
0.1
0.2
0.3
0.4
0.5
Avg
MATR [ 24 ]
21.5
19.5
15.9
12.6
8.6
15.6
No-History
24.2
22.7
20.5
16.3
10.9
18.9
LSTR [ 10 ]
25.3
23.3
20.4
16.4
12.0
19.5
GateHUB [ 15 ]
26.2
24.0
21.0
16.6
11.7
19.9
FSHM(HAT) [ 21 ]
27.3
25.3
21.6
16.9
12.8
20.8
GMB
29.2
27.1
24.0
18.1
12.7
22.2 ∗
TABLE V: Additional baselines on EGTEA dataset, re-implemented under a controlled setting. ∗ Our final configuration. HAT figures here differ from Table I due to this controlled re-implementation.
Temporal Action Localization (TAL) typically relies on segment annotations or offline access to full videos, limiting scalability and online use. We introduce Point-Supervised Online TAL (POTAL), which localizes actions in streaming videos using only one temporal point per instance. To solve POTAL, we propose OnPoint, an offline-to-online multi-level distillation framework that transfers knowledge from a point-supervised offline teacher to an online student via (i) pseudo-segment instance distillation, (ii) class-activation sequence distillation, and (iii) anticipatory window-level distillation. We further improve robustness by incorporating the original point labels into student training and by refining anchor decoding with actionness-guided attention calibration. Experiments on five datasets show OnPoint consistently outperforms strong baselines, establishing a solid foundation for POTAL.
Sakib Reza, Gauri Jagatap, Mohsen Moghaddam +2
Northeastern University, Boston, MA 02115, USA · Georgia Institute of Technology, Atlanta, GA 30332, USA · Dolby Laboratories +1
Open-Vocabulary Temporal Action Localization (OV-TAL) aims to recognize and localize instances of any desired action categories in videos without explicitly curating training data for all categories. Existing methods mostly recognize action categories at a single granularity, which degrades the recognition accuracy of both base and novel action categories. To address these issues, we propose a Multi-Grained Category-Aware Network (MGCA-Net) comprising a localizer, an action presence predictor, a conventional classifier, and a coarse-to-fine classifier. Specifically, the localizer localizes category-agnostic action proposals. For these action proposals, the action presence predictor estimates the probability that they belong to an action instance. At the same time, the conventional classifier predicts the probability of each action proposal over base action categories at the snippet granularity. Novel action categories are recognized by the coarse-to-fine classifier, which first identifies action presence at the video granularity. Finally, it assigns each action proposal to one category from the coarse categories at the proposal granularity. Through coarse-to-fine category awareness for novel actions and the conventional classifier's awareness of base actions, multi-grained category awareness is achieved, effectively enhancing localization performance. Comprehensive evaluations on the THUMOS'14 and ActivityNet-1.3 benchmarks demonstrate that our method achieves state-of-the-art performance. Furthermore, our MGCA-Net achieves state-of-the-art results under the Zero-Shot Temporal Action Localization setting.
Zhenying Fang, Richang Hong
School of Computer Science and Information Engineering, Hefei University of Technology, 188065, China
Online Temporal Action Localization (On-TAL) aims to detect the occurrence time and category of actions in untrimmed streaming videos immediately upon their completion. Recent advancements in this field focus on developing more sophisticated frameworks, shifting from Online Action Detection (OAD)-based aggregation paradigm to instance-level understanding. However, existing approaches are typically trained on specific domains and often exhibit limited generalization capabilities when applied to arbitrary videos, particularly in the presence of previously unseen actions. In this paper, we introduce a new task called Online Zero-shot Temporal Action Localization (OZ-TAL), which aims to detect previously unseen actions in an online fashion. Furthermore, we propose a training-free framework that leverages off-the-shelf Vision-Language Models (VLMs) while introducing additional mechanisms to enhance visual representations and mitigate their inherent biases. We establish new benchmarks and representative baselines for OZ-TAL on THUMOS14 and ActivityNet-1.3, and extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches under both offline and online zero-shot settings.
Chaolei Han, Hongsong Wang, Xin Gong +1
School of Cyber Science and Engineering, Southeast University, Nanjing 211102, China · Engineering Research Center of Blockchain Application, Supervision and Management (Southeast University), Ministry of Education; Purple Mountain Laboratories, Nanjing 211111, China · School of Computer Science and Engineering, Southeast University, Nanjing 211102, China