HuC-VideoMAE: Human-Centric Video Masked Autoencoding from synthetic data
Authors: Ricardo Pizarro, Roberto Valle, José M. Buenaposada, Luis M. Bergasa, Luis Baumela
Organizations: Universidad de Alcal´a, Alcal´a de Henares, Spain · Universidad Polit´ecnica de Madrid, Madrid, Spain · Universidad Rey Juan Carlos, M´ostoles, Spain
Modern action recognition models rely on video transformers pretrained on massive collections of web-crawled videos, such as Kinetics-700. However, the use of such data raises ethical concerns, as subjects' consent is typically not obtained. Recent high-quality synthetic video datasets generated from motion-capture data, such as BEDLAM2.0, offer a promising ethical alternative. In this work, we investigate self-supervised pretraining of video transformers on synthetic human-motion datasets. We first show that directly applying the standard VideoMAE masking strategy leads to substantially worse performance than pretraining on Kinetics. To address this limitation, we propose a human-centric masking scheme that leverages body keypoints and person bounding box regions. Our approach encourages the model to focus on the structure and dynamics of human motion during pretraining. Experiments on NTU RGB+D and Toyota-Smarthome demonstrate that our method significantly outperforms standard VideoMAE pretraining on synthetic data, closing 49% of the gap to Kinetics pretraining on NTU RGB+D cross-view-subject without using a single real frame during pretraining. To promote the use of ethical action recognition models, we will publicly release our pretrained models.
Figures & tables
Figure 1: Body-keypoint-guided masking. From a synthetic frame with 2 D keypoint and person-box annotations (a), every patch is labelled as keypoint , body (inside the box) or background (b), and receives a masking score ρ (c): background scores above the body, and keypoints lowest of all, so that the sparse joint structure survives the mask. The exception are the joints selected for identity masking ( ∈S , red), with score above the background and are therefore always masked.
Figure 2: Keypoint masking is temporally consistent. A keypoint selected for masking (here the right wrist) is hidden in every frame of the clip (bottom row), because the choice is made at the body part identity level. A naive per-frame choice (top row) would leave the same joint visible in some frames ( t1,t4 ), letting the decoder copy it from a neighbouring frame. Green: visible keypoints; red: the masked joint and its patch.
Pretrain data
Masking
0∘
45∘
90∘
Avg
Kinetics
random tube
92.2
91.8
91.2
91.7
BEDLAM2.0
random tube
81.6
79.6
79.1
80.1
BEDLAM2.0
HuC-VideoMAE
86.6
85.9
84.8
85.8
Table 1: Ablation of the masking policy on NTU RGB+D, cross-view-subject (ViT-B). The backbone, optimizer, schedule and augmentation are identical across rows. Only the pretraining source and the masking scheme change. All numbers are downstream fine-tuning accuracy (in % ) after synthetic-to-real transfer (S → R), except the Kinetics row (R → R).
Pretrain data
NTU (Avg)
Toyota (Avg)
AMARV (synth.)
55.0
42.3
BEDLAM2.0 (synth.)
80.1
56.5
Table 2: Strength of the pretraining source, independent of masking policy. Random tube masking, same backbone and schedule. Only the synthetic pretraining corpus changes. Downstream accuracy averaged per dataset (NTU: 0∘/45∘/90∘ . Toyota: CV1/CV2).
Model
Pretrain Data
0∘
45∘
90∘
SURREACT [ 25 ]
R
-
86.9
74.5
53.6
X3D-S [ 19 ]
R
-
86.4
77.8
60.4
ViewCLR [ 6 ]
R
-
84.2
77.0
75.8
SURREACT [ 25 ]
S → R
SURREACT
84.1
77.5
66.2
X3D-S [ 19 ]
S → R
AMARV
89.9
81.8
68.0
MViTv2-S [ 19 ]
S → R
AMARV
94.4
84.5
65.1
Table 3: Cross-view-subject evaluation on NTU RGB+D, comparison with SURREACT [ 25 ] and our new VideoMAE-based results. S → R, Pretrained on Synth and then fine-tuned on Real.
Method
CV1
CV2
mCA. ( ↑ )
mCA. ( ↑ )
MotionFormer [ 16 ]
45.2
51.0
LTN [ 29 ]
-
54.6
TimeSFormer [ 2 ]
50.0
60.6
VPN++ [ 5 ]
-
54.9
Video Swing [ 15 ]
36.6
48.6
Table 4: Test results on Toyota-Smarthome over the CV1 and CV2 protocols. Comparison of our HuC models, against previous methods pretrained on Kinetics.