CodecMotion-OAD: Efficient Online Action Detection with Compressed-Domain Motion

Zepeng Sun; Naichuan Zheng; Hailun Xia; Junjie Wu

cs.CVOct 17, 2026

CodecMotion-OAD: Efficient Online Action Detection with Compressed-Domain Motion

Authors: Zepeng Sun, Naichuan Zheng, Hailun Xia, Junjie Wu

Organizations: School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China

Abstract

Online action detection (OAD) identifies ongoing actions in streaming videos. Many existing OAD systems combine RGB appearance with optical flow, but dense-flow extraction remains a major computational bottleneck. Codec motion vectors (MVs) are available during decoding, avoiding a separate optical-flow computation. On THUMOS'14, optical flow exceeds MVs by 11.2 mAP points with motion-only inputs, whereas the gap narrows to 3.7 points after RGB fusion, with the largest reduction in action-category discrimination. We propose CodecMotion-OAD, which replaces optical flow with decoder-provided MVs while retaining existing temporal detector architectures. Its two-stage training first aligns MV features with a pretrained flow-feature interface without action labels and then trains RGB+MV detectors using ground-truth labels and soft predictions from a frozen RGB+Flow teacher. To the best of our knowledge, this is the first application of codec MVs to frame-level OAD. On THUMOS'14 with MiniROAD, CodecMotion-OAD achieves 68.3 mAP versus 70.9 mAP for matched-distillation RGB+Flow, while reducing decoder-to-prediction processing time by 11.48-fold on CPU and 2.04-fold on GPU.

Figures & tables

Figure 1: Accuracy–processing trade-off of MiniROAD on THUMOS’14 under matched soft-label distillation. CodecMotion-OAD achieves 68.3 mAP compared with 70.9 mAP for RGB+TV-L1, while reducing decoder-to-prediction processing time by 2.04 $\times$ on an NVIDIA Tesla V100 GPU and 11.48 $\times$ on an Intel Xeon Gold 6240 CPU with eight threads. Filled markers denote GPU measurements, and hollow markers denote CPU measurements. The horizontal axis uses a logarithmic scale.

Figure 2: Qualitative comparison of motion representations on THUMOS’14. The columns show RGB center frames, TV-L1 optical flow, spatially coarsened Flow (16 $\times$ 16 block averaging, with block means tiled back to the original resolution), and codec motion vectors. Rows (a)–(d) show ThrowDiscus, HammerThrow, and two distinct Shotput examples. The three motion visualizations within each row share the same direction-to-hue mapping and magnitude-to-saturation scale, with a common magnitude cap for that row. Block averaging removes fine spatial detail, whereas codec MVs exhibit additional sparsity and prediction-dependent spatial structure.

Figure 3: Flow–MV performance-gap contraction after RGB fusion on THUMOS’14, without soft-label distillation. (a) Overall mAP gaps for MiniROAD and frame-wise linear probes. (b) MiniROAD gaps in action identity and action presence. Identity macro-AP evaluates category discrimination on ground-truth action frames; presence AP evaluates action-versus-background discrimination. All gaps are expressed in percentage points.

Figure 4: Overview of Interface-Preserving Motion Transfer (IPMT). (a) Stage I adapts a codec-MV encoder initialized from Flow-ResNet50 by matching its 2048-dimensional per-step features to those of a frozen Flow-ResNet50, without action labels. (b) Stage II freezes the RGB and adapted MV encoders and trains the original RGB+MV temporal detectors with ground-truth labels mixed with soft labels from a shared, frozen CMeRT RGB+Flow teacher. (c) Under the H.264 IPPP protocol, decoding supplies RGB frames and codec MVs to the student detector, without optical-flow extraction or the teacher at inference.

	THUMOS’14			Charades
Detector	RGB	RGB+Flow	RGB+MV (ours)	RGB	RGB+Flow	RGB+MV (ours)
CMeRT	59.7	70.8	68.6	14.7	15.4	15.5
MiniROAD	54.6	67.5	68.3	13.9	14.9	16.5
LSTR	59.1	69.6	67.9	13.3	13.7	14.9
TeSTra	60.1	70.5	68.3	12.1	12.5	13.8

Table 3: Frame-level mAP on THUMOS’14 and Charades. The RGB+Flow references use ground-truth supervision; CodecMotion-OAD uses both IPMT stages. The matched-distillation MiniROAD comparison is in Table 4.

Source/state	Raw cos.	Ctr. cos.	$R^{2}$	mAP
RAFT fixed	0.578	0.306	0.290	64.2
RAFT adapted	0.721	0.277	0.232	60.2
MV fixed	0.456	0.066	0.206	63.7
MV adapted	0.828	0.623	0.407	68.3

Table 8: Effect of source-dependent feature adaptation on THUMOS’14. Centered cosine subtracts training-set means;

R^{2}

is the test-set ridge prediction score. CMeRT uses ground-truth supervision without soft labels.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	mAP	ms/step
TV-L1 (persistent) Original Flow-ResNet50 Ground truth	70.8	102.2
RAFT-Large Fixed pretrained Flow-ResNet50 Ground truth	64.2	120.5
CodecMotion-OAD Adapted codec encoder Mixed teacher targets	68.6	60.5

Appendix

Table A3: THUMOS’14 CMeRT accuracy and GPU processing time across three source configurations. Each entry specifies the motion source, feature encoder, and decision supervision, respectively. Table 8 compares source adaptation under a common ground-truth CMeRT objective.

Figure B1: Qualitative comparison of aligned RGB frames, TV-L1 optical flow, and codec motion vectors from THUMOS’14 under the shared IPPP protocol. The examples show (a) body motion (SoccerPenalty), (b) camera motion (CliffDiving), (c) localized motion (TennisSwing), and (d) an I picture without motion vectors (CleanAndJerk). For visualization, codec MVs are sign-flipped to match the forward-flow direction convention. Within each row, Flow and MV use the same magnitude cap, determined by the 99th percentile of Flow magnitude and labeled in pixels. The displayed motion fields are aligned at $c_{t}$ and each constitutes one field of the five-field encoder input.

Temporal Action Detection

Temporal action detection identifies action categories and their start and end times in untrimmed video.

See all 41 papers

CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
Mar 25, 2026Hieu Hoang, Dung Trung Tran, Hong Nguyen +1Temporal Action Detection Human Activity Recognition
Viettel Group, Vietnam · School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam · Signal Analysis and Interpretation Lab, University of Southern California, United States
Improving Viewpoint-Invariance and Temporal Consistency for Action Detection
May 21, 2026Yannick Porto, Renato Martins, Thomas Chalumeau +1Video Action Recognition Invariant Representation Learning
⋆Universit Bourgogne Europe, CNRS, ICB · † TEB Group, Prynel SAS
LiquidTAD: Efficient Temporal Action Detection via Parallel Liquid-Inspired Temporal Relaxation
Apr 20, 2026Zepeng Sun, Naichuan Zheng, Hailun Xia +3Liquid Neural Networks Human Activity Recognition

Mar 25, 2026J/K move · Enter open · S save

CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning

Hieu Hoang, Dung Trung Tran, Hong Nguyen, Nam-Phong Nguyen

Viettel Group, Vietnam · School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam · Signal Analysis and Interpretation Lab, University of Southern California, United States

Temporal Action Detection Human Activity Recognition Knowledge Distillation

Open paper AskPDF arXiv

Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.

Abstract

Temporal Action Detection

Explore similar work

CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning

Improving Viewpoint-Invariance and Temporal Consistency for Action Detection

LiquidTAD: Efficient Temporal Action Detection via Parallel Liquid-Inspired Temporal Relaxation