Online action detection (OAD) identifies ongoing actions in streaming videos. Many existing OAD systems combine RGB appearance with optical flow, but dense-flow extraction remains a major computational bottleneck. Codec motion vectors (MVs) are available during decoding, avoiding a separate optical-flow computation. On THUMOS'14, optical flow exceeds MVs by 11.2 mAP points with motion-only inputs, whereas the gap narrows to 3.7 points after RGB fusion, with the largest reduction in action-category discrimination. We propose CodecMotion-OAD, which replaces optical flow with decoder-provided MVs while retaining existing temporal detector architectures. Its two-stage training first aligns MV features with a pretrained flow-feature interface without action labels and then trains RGB+MV detectors using ground-truth labels and soft predictions from a frozen RGB+Flow teacher. To the best of our knowledge, this is the first application of codec MVs to frame-level OAD. On THUMOS'14 with MiniROAD, CodecMotion-OAD achieves 68.3 mAP versus 70.9 mAP for matched-distillation RGB+Flow, while reducing decoder-to-prediction processing time by 11.48-fold on CPU and 2.04-fold on GPU.
Figures & tables
Figure 1: Accuracy–processing trade-off of MiniROAD on THUMOS’14 under matched soft-label distillation. CodecMotion-OAD achieves 68.3 mAP compared with 70.9 mAP for RGB+TV-L1, while reducing decoder-to-prediction processing time by 2.04 × on an NVIDIA Tesla V100 GPU and 11.48 × on an Intel Xeon Gold 6240 CPU with eight threads. Filled markers denote GPU measurements, and hollow markers denote CPU measurements. The horizontal axis uses a logarithmic scale.
Figure 2: Qualitative comparison of motion representations on THUMOS’14. The columns show RGB center frames, TV-L1 optical flow, spatially coarsened Flow (16 × 16 block averaging, with block means tiled back to the original resolution), and codec motion vectors. Rows (a)–(d) show ThrowDiscus, HammerThrow, and two distinct Shotput examples. The three motion visualizations within each row share the same direction-to-hue mapping and magnitude-to-saturation scale, with a common magnitude cap for that row. Block averaging removes fine spatial detail, whereas codec MVs exhibit additional sparsity and prediction-dependent spatial structure.
Figure 3: Flow–MV performance-gap contraction after RGB fusion on THUMOS’14, without soft-label distillation. (a) Overall mAP gaps for MiniROAD and frame-wise linear probes. (b) MiniROAD gaps in action identity and action presence. Identity macro-AP evaluates category discrimination on ground-truth action frames; presence AP evaluates action-versus-background discrimination. All gaps are expressed in percentage points.
Figure 4: Overview of Interface-Preserving Motion Transfer (IPMT). (a) Stage I adapts a codec-MV encoder initialized from Flow-ResNet50 by matching its 2048-dimensional per-step features to those of a frozen Flow-ResNet50, without action labels. (b) Stage II freezes the RGB and adapted MV encoders and trains the original RGB+MV temporal detectors with ground-truth labels mixed with soft labels from a shared, frozen CMeRT RGB+Flow teacher. (c) Under the H.264 IPPP protocol, decoding supplies RGB frames and codec MVs to the student detector, without optical-flow extraction or the teacher at inference.
THUMOS’14
Charades
Detector
RGB
RGB+Flow
RGB+MV (ours)
RGB
RGB+Flow
RGB+MV (ours)
CMeRT
59.7
70.8
68.6
14.7
15.4
15.5
MiniROAD
54.6
67.5
68.3
13.9
14.9
16.5
LSTR
59.1
69.6
67.9
13.3
13.7
14.9
TeSTra
60.1
70.5
68.3
12.1
12.5
13.8
Table 3: Frame-level mAP on THUMOS’14 and Charades. The RGB+Flow references use ground-truth supervision; CodecMotion-OAD uses both IPMT stages. The matched-distillation MiniROAD comparison is in Table 4.
Source/state
Raw cos.
Ctr. cos.
R2
mAP
RAFT fixed
0.578
0.306
0.290
64.2
RAFT adapted
0.721
0.277
0.232
60.2
MV fixed
0.456
0.066
0.206
63.7
MV adapted
0.828
0.623
0.407
68.3
Table 8: Effect of source-dependent feature adaptation on THUMOS’14. Centered cosine subtracts training-set means; R2 is the test-set ridge prediction score. CMeRT uses ground-truth supervision without soft labels.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
mAP
ms/step
TV-L1 (persistent) Original Flow-ResNet50 Ground truth
70.8
102.2
RAFT-Large Fixed pretrained Flow-ResNet50 Ground truth
Table A3: THUMOS’14 CMeRT accuracy and GPU processing time across three source configurations. Each entry specifies the motion source, feature encoder, and decision supervision, respectively. Table 8 compares source adaptation under a common ground-truth CMeRT objective.
Figure B1: Qualitative comparison of aligned RGB frames, TV-L1 optical flow, and codec motion vectors from THUMOS’14 under the shared IPPP protocol. The examples show (a) body motion (SoccerPenalty), (b) camera motion (CliffDiving), (c) localized motion (TennisSwing), and (d) an I picture without motion vectors (CleanAndJerk). For visualization, codec MVs are sign-flipped to match the forward-flow direction convention. Within each row, Flow and MV use the same magnitude cap, determined by the 99th percentile of Flow magnitude and labeled in pixels. The displayed motion fields are aligned at ct and each constitutes one field of the five-field encoder input.
Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.
Hieu Hoang, Dung Trung Tran, Hong Nguyen +1
Viettel Group, Vietnam · School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam · Signal Analysis and Interpretation Lab, University of Southern California, United States
Viewpoint change invariance and action temporal consistency are critical aspects for the effective deployment of human action detection of untrimmed videos. Existing appearance-based video detection methods often struggle with limited viewpoint diversity during training, while motion-based detection approaches frequently fail to model fine-grained temporal relationships across consecutive motion windows. This paper introduces a novel two-stage action detection approach designed to improve both view-invariance and global temporal coherence properties. In the first stage, we extract motion features from augmented virtual viewpoints, solely used at training. Then, the second stage introduces a new view-invariant, multi-scale temporal encoder based on selective state-space sequence modelling to aggregate information across viewpoints and time scales. Experiments on PKU-MMD and BABEL benchmarks demonstrate that this approach significantly outperforms state-of-the-art methods in all considered splits. Code and trained models are available at: https://icb-vision-ai.github.io/HydraView-TAD
Yannick Porto, Renato Martins, Thomas Chalumeau +1
Temporal Action Detection (TAD) requires precise localization of action boundaries within long, untrimmed video sequences. While current high-performing methods achieve strong accuracy, they are often characterized by excessive parameter counts, substantial computational overhead, and a reliance on specialized operators that hinder deployment across diverse hardware platforms. This paper presents LiquidTAD, a framework that distills the exponential relaxation prior of liquid neural dynamics into a parallel temporal operator, rather than reproducing full Liquid Neural Network (LNN) dynamics. By introducing a Parallel Liquid-inspired Relaxation mechanism, sequential ODE solving is avoided through a fully vectorized, non-recursive formulation built entirely upon standard neural operations, enabling hardware-agnostic deployment with linear complexity with respect to the temporal length. A complementary Hierarchical Decay-Rate Sharing Strategy further adapts this relaxation prior across feature pyramid levels, stabilizing optimization and implicitly compensating for temporal compression in deeper layers. Experimental evaluations on THUMOS-14 and ActivityNet-1.3 demonstrate that LiquidTAD achieves accuracy competitive with strong baselines while substantially lowering the model footprint. Specifically, on THUMOS-14, LiquidTAD achieves 69.46% average mAP with only 10.82M parameters and 27.17G FLOPs, reducing the parameter count by over 60% compared with ActionFormer.