Micro-actions, fleeting and low-amplitude motions, such as glances, nods, or minor posture shifts, carry rich social meaning but remain difficult for current action recognition models to recognize due to their subtlety, short duration, and high inter-class ambiguity. In this paper, we introduce B-MoE, a Body-part-aware Mixture-of-Experts framework designed to explicitly model the structured nature of human motion. In B-MoE, each expert specializes in a distinct body region (head, body, upper limbs, lower limbs), and is based on the lightweight Macro-Micro Motion Encoder (M3E) that captures long-range contextual structure and fine-grained local motion. A cross-attention routing mechanism learns inter-region relationships and dynamically selects the most informative regions for each micro-action. B-MoE uses a dual-stream encoder that fuses these region-specific semantic cues with global motion features to jointly capture spatially localized cues and temporally subtle variations that characterize micro-actions. Experiments on three challenging benchmarks (MA-52, SocialGesture, and MPII-GroupInteraction) show consistent state-of-theart gains, with improvements in ambiguous, underrepresented, and low amplitude classes.
Figures & tables
Figure 1: Visualization of MA categories illustrating fine-grained motion variations and the challenges of class imbalance and inter-category ambiguity. Example actions (E1: Scratching or touching chest , C11: Pointing oneself ) that appear visually similar but belongs to different semantic categories [ 10 ] .
Figure 2: Semantic Branch: Using SAPIENS [ 14 ] , we segment each frame, derive the crop around the target body part (upper limb in this example), and apply the corresponding mask to the cropped region. The resulting cropped and masked video is then processed by VideoMAE-V2, pretrained on Kinetics.
Figure 3: B-MoE: A dual-stream encoder extracts region-conditioned semantic features using semantic encoder and global motion encoder. The semantic stream is routed through a region-aware MoE, where each expert specializes in modeling micro-movements within a specific body region. A cross-attention fusion head integrates expert outputs with motion saliency from the global stream, and a transformer-MLP classifier produces the final predictions.
Figure 4: Macro-Micro Motion Encoder (M3E). The input sequence is processed with multi-head self-attention to capture global temporal dependencies, followed by an SGP module [ 31 ] for fine-grained local motion reasoning. During pre-training, a semantic alignment loss ( Lemb ) aligns learned features with word embeddings of action labels.
Method
Top-1
F1 macro
TSN [ECCV’16] [ 34 ]
34.46
28.52
TIN [AAAI’20] [ 30 ]
52.81
39.82
TSM [ICCV’19] [ 19 ]
56.75
40.19
C3D [ICCV’15] [ 32 ]
52.22
40.86
I3D [CVPR’17] [ 5 ]
57.07
39.84
SlowFast [ICCV’19] [ 9 ]
59.60
44.96
Table 1: Performance comparison of MAR on the MA-52 dataset.
Method
MPII-GI
SocialGesture
Top-1
F1 macro
Top-1
F1 macro
I3D [CVPR’17]
62.24
34.62
83.54
25.54
VSwinT [CVPR’22]
57.16
30.73
85.45
42.58
TimesFormer [ICML’21]
49.87
17.79
84.88
45.38
MANet [TCSVT’24]
66.86
41.63
85.74
47.76
MoNE [NeurIPS’24]
30.85
4.16
82.34
22.63
Table 2: Comparison of MAR performance on MPII-GI and SocialGesture.
Semantic
Motion
Top-1
F1 macro
✓
✗
59.31
47.82
✗
✓
60.90
48.98
✓
✓
64.54
53.30
Table 3: Ablation study on the impact of the dual-stream encoder (semantic and motion) on MA-52 dataset. A ✓ indicates that the corresponding stream is included in the model, while an ✗ indicates that it is removed.
#
Head
Body
Upper-Limb
Lower-Limb
Top-1
F1 macro
1
✗
✓
✓
✓
62.01
51.72
2
✓
✗
✓
✓
64.21
52.14
3
✓
✓
✗
✓
63.76
51.94
4
✓
✓
✓
✗
63.94
52.62
5
✓
✓
✓
✓
64.54
53.30
Table 4: Ablation experiments on the impact of the experts on MA-52. A ✓ indicates that the corresponding expert is included in the architecture, while ✗ denotes its removal.
Figure 5: Cross-attention heatmap showing expert importance per micro-action class on MA-52. Brighter colors indicate higher expert activation.
Figure 6: Per-class comparison of B-MoE and MaNet on the validation set for MA-52. Classes enclosed by black, blue, and red boxes correspond to ambiguous , underrepresented , and subtle actions, respectively. The column Train_N indicates the percentage of samples for each class.