CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
Authors: Hieu Hoang, Dung Trung Tran, Hong Nguyen, Nam-Phong Nguyen
Organizations: Viettel Group, Vietnam · School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam · Signal Analysis and Interpretation Lab, University of Southern California, United States
Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.
Figures & tables
Fig. 1 : Overview of the proposed CAKE framework and DMA. (a) During training, an optical-flow teacher guides the DMA via cross-modal distillation. At inference, the RGB backbone splits into a static branch and a DMA motion branch, whose features are fused before temporal modeling with a GRU. (b) ODConv3D mechanism used in DMA, where base kernels Wi are modulated by dynamic attention weights along kernel ( aw ), channel ( af,ac ), spatial ( as ), and temporal ( at ) dimensions.
Fig. 2 : Overview of the MoCo mechanism. The query encoder is updated by backpropagation, while the momentum encoder is updated using Exponential Moving Average (EMA) to maintain a consistent dictionary queue.
Method
Backbone
Params (M)
GFLOPs
THUMOS’14
TVSeries
Back.
Head
mAP (%)
mcAP (%)
RED [ 18 ]
-
-
-
-
45.3
79.2
LAP [ 19 ]
-
-
-
-
53.3
85.3
TFN [ 20 ]
-
-
-
-
55.7
85.0
FATS [ 21 ]
-
-
-
-
59.0
84.6
IDN [ 7 ]
-
-
-
-
60.3
86.1
TABLE I : Performance and efficiency comparison of CAKE and SOTA methods on THUMOS’14 and TVSeries. BNI : BN-Inception.
Fig. 3 : Qualitative analysis using Grad-CAM. From left to right: (a) Original RGB frames, (b) Teacher trained on Optical Flow, (c) RGB Student with static Conv3D, and (d) Our DMA with ODConv3D.
TABLE III : Ablation study of individual components.
Method
Learning Mechanism
mAP (%)
MiniROAD [ 10 ]
Supervised (RNN)
71.40
CAKE (Baseline)
Standard SupCon
70.80 (-0.6)
CAKE (Ours)
Floating SupCon
72.00 (+0.6)
TABLE IV : Performance comparison between MiniROAD, Standard SupCon, and the proposed Floating SupCon on THUMOS’14.
Model
Input
Top-1 Acc. (%)
Top-5
GFLOPs × views
Params (M)
RGB
Flow
Comb.
Single-stream Models
I3D [ 15 ]
R
71.10
-
-
90.30
108.0×N/A
12.0
C2D R50 [ 27 ]
R
71.46
-
-
89.68
25.89×30
24.3
Slow R50 ( 4×16 ) [ 28 ]
R
72.40
-
-
90.18
27.55×30
32.4
X3D-S (Baseline) [ 16 ]
R
73.33
-
-
91.27
1.96×30
3.8
TABLE V : Performance comparison on Kinetics-400. ‘*’ denotes optical flow reconstruction accuracy. R: RGB, OF: Optical Flow.
Fig. 4 : t-SNE visualization of the learned feature space on THUMOS’14. (a) Training features showing clear semantic clustering produced by Floating SupCon. (b) Test features demonstrating generalization to unseen samples.
School of Computer Science, Hangzhou Dianzi University, Hangzhou, China · State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China