CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
Authors: Hieu Hoang, Dung Trung Tran, Hong Nguyen, Nam-Phong Nguyen
Organizations: Viettel Group, Vietnam · School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam · Signal Analysis and Interpretation Lab, University of Southern California, United States
Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.
Figures & tables
Fig. 1 : Overview of the proposed CAKE framework and DMA. (a) During training, an optical-flow teacher guides the DMA via cross-modal distillation. At inference, the RGB backbone splits into a static branch and a DMA motion branch, whose features are fused before temporal modeling with a GRU. (b) ODConv3D mechanism used in DMA, where base kernels Wi are modulated by dynamic attention weights along kernel ( aw ), channel ( af,ac ), spatial ( as ), and temporal ( at ) dimensions.
Fig. 2 : Overview of the MoCo mechanism. The query encoder is updated by backpropagation, while the momentum encoder is updated using Exponential Moving Average (EMA) to maintain a consistent dictionary queue.
Method
Backbone
Params (M)
GFLOPs
THUMOS’14
TVSeries
Back.
Head
mAP (%)
mcAP (%)
RED [ 18 ]
-
-
-
-
45.3
79.2
LAP [ 19 ]
-
-
-
-
53.3
85.3
TFN [ 20 ]
-
-
-
-
55.7
85.0
FATS [ 21 ]
-
-
-
-
59.0
84.6
IDN [ 7 ]
-
-
-
-
60.3
86.1
TABLE I : Performance and efficiency comparison of CAKE and SOTA methods on THUMOS’14 and TVSeries. BNI : BN-Inception.
Fig. 3 : Qualitative analysis using Grad-CAM. From left to right: (a) Original RGB frames, (b) Teacher trained on Optical Flow, (c) RGB Student with static Conv3D, and (d) Our DMA with ODConv3D.
TABLE III : Ablation study of individual components.
Method
Learning Mechanism
mAP (%)
MiniROAD [ 10 ]
Supervised (RNN)
71.40
CAKE (Baseline)
Standard SupCon
70.80 (-0.6)
CAKE (Ours)
Floating SupCon
72.00 (+0.6)
TABLE IV : Performance comparison between MiniROAD, Standard SupCon, and the proposed Floating SupCon on THUMOS’14.
Model
Input
Top-1 Acc. (%)
Top-5
GFLOPs × views
Params (M)
RGB
Flow
Comb.
Single-stream Models
I3D [ 15 ]
R
71.10
-
-
90.30
108.0×N/A
12.0
C2D R50 [ 27 ]
R
71.46
-
-
89.68
25.89×30
24.3
Slow R50 ( 4×16 ) [ 28 ]
R
72.40
-
-
90.18
27.55×30
32.4
X3D-S (Baseline) [ 16 ]
R
73.33
-
-
91.27
1.96×30
3.8
TABLE V : Performance comparison on Kinetics-400. ‘*’ denotes optical flow reconstruction accuracy. R: RGB, OF: Optical Flow.
Fig. 4 : t-SNE visualization of the learned feature space on THUMOS’14. (a) Training features showing clear semantic clustering produced by Floating SupCon. (b) Test features demonstrating generalization to unseen samples.
Temporal Action Detection (TAD) requires precise localization of action boundaries within long, untrimmed video sequences. While current high-performing methods achieve strong accuracy, they are often characterized by excessive parameter counts, substantial computational overhead, and a reliance on specialized operators that hinder deployment across diverse hardware platforms. This paper presents LiquidTAD, a framework that distills the exponential relaxation prior of liquid neural dynamics into a parallel temporal operator, rather than reproducing full Liquid Neural Network (LNN) dynamics. By introducing a Parallel Liquid-inspired Relaxation mechanism, sequential ODE solving is avoided through a fully vectorized, non-recursive formulation built entirely upon standard neural operations, enabling hardware-agnostic deployment with linear complexity with respect to the temporal length. A complementary Hierarchical Decay-Rate Sharing Strategy further adapts this relaxation prior across feature pyramid levels, stabilizing optimization and implicitly compensating for temporal compression in deeper layers. Experimental evaluations on THUMOS-14 and ActivityNet-1.3 demonstrate that LiquidTAD achieves accuracy competitive with strong baselines while substantially lowering the model footprint. Specifically, on THUMOS-14, LiquidTAD achieves 69.46% average mAP with only 10.82M parameters and 27.17G FLOPs, reducing the parameter count by over 60% compared with ActionFormer.
Knowledge Distillation (KD) offers a promising yet underexplored path for compressing large action recognition models. However, existing KD methods suffer from two key limitations: 1) reliance on fixed input samples leads to suboptimal feature alignment between the frozen teacher (larger model) and the learnable student (smaller model), and 2) applying a uniform distillation strength for all channels fails to account for their varying importance in capturing distinct knowledge (e.g., motion tempo or magnitude) across training epochs. This motivates us to develop an Adaptive Sample-aware Channel-wise Dynamic (ASCD) KD approach, which operates in two stages. First, we use an adaptive sample generation module to create updated samples by incorporating semantics from sample gradients, which are derived by minimizing a feature loss weighted by channel centroid frequency differences at each layer. Meanwhile, crucial motion-related details are preserved by applying a Gaussian mask to frequency features. Second, we employ a channel-wise dynamic distillation module to train student on these generated samples, guided by sample gradients and feature frequencies. For efficiency, samples are updated periodically rather than per epoch. Extensive experiments on three video benchmarks (UCF101, Kinetics-400, Something-Something-v2) and two image datasets (CIFAR-100, ImageNet) demonstrate the state-of-the-art performance of our method. Code is available at https://github.com/mlvccn/ASCD_KD_Action.
Ping Li, Chenhao Ping, Jie Song +1
School of Computer Science, Hangzhou Dianzi University, Hangzhou, China · State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China
Viewpoint change invariance and action temporal consistency are critical aspects for the effective deployment of human action detection of untrimmed videos. Existing appearance-based video detection methods often struggle with limited viewpoint diversity during training, while motion-based detection approaches frequently fail to model fine-grained temporal relationships across consecutive motion windows. This paper introduces a novel two-stage action detection approach designed to improve both view-invariance and global temporal coherence properties. In the first stage, we extract motion features from augmented virtual viewpoints, solely used at training. Then, the second stage introduces a new view-invariant, multi-scale temporal encoder based on selective state-space sequence modelling to aggregate information across viewpoints and time scales. Experiments on PKU-MMD and BABEL benchmarks demonstrate that this approach significantly outperforms state-of-the-art methods in all considered splits. Code and trained models are available at: https://icb-vision-ai.github.io/HydraView-TAD
Yannick Porto, Renato Martins, Thomas Chalumeau +1