Micro-actions, fleeting and low-amplitude motions, such as glances, nods, or minor posture shifts, carry rich social meaning but remain difficult for current action recognition models to recognize due to their subtlety, short duration, and high inter-class ambiguity. In this paper, we introduce B-MoE, a Body-part-aware Mixture-of-Experts framework designed to explicitly model the structured nature of human motion. In B-MoE, each expert specializes in a distinct body region (head, body, upper limbs, lower limbs), and is based on the lightweight Macro-Micro Motion Encoder (M3E) that captures long-range contextual structure and fine-grained local motion. A cross-attention routing mechanism learns inter-region relationships and dynamically selects the most informative regions for each micro-action. B-MoE uses a dual-stream encoder that fuses these region-specific semantic cues with global motion features to jointly capture spatially localized cues and temporally subtle variations that characterize micro-actions. Experiments on three challenging benchmarks (MA-52, SocialGesture, and MPII-GroupInteraction) show consistent state-of-theart gains, with improvements in ambiguous, underrepresented, and low amplitude classes.
Figures & tables
Figure 1: Visualization of MA categories illustrating fine-grained motion variations and the challenges of class imbalance and inter-category ambiguity. Example actions (E1: Scratching or touching chest , C11: Pointing oneself ) that appear visually similar but belongs to different semantic categories [ 10 ] .
Figure 2: Semantic Branch: Using SAPIENS [ 14 ] , we segment each frame, derive the crop around the target body part (upper limb in this example), and apply the corresponding mask to the cropped region. The resulting cropped and masked video is then processed by VideoMAE-V2, pretrained on Kinetics.
Figure 3: B-MoE: A dual-stream encoder extracts region-conditioned semantic features using semantic encoder and global motion encoder. The semantic stream is routed through a region-aware MoE, where each expert specializes in modeling micro-movements within a specific body region. A cross-attention fusion head integrates expert outputs with motion saliency from the global stream, and a transformer-MLP classifier produces the final predictions.
Figure 4: Macro-Micro Motion Encoder (M3E). The input sequence is processed with multi-head self-attention to capture global temporal dependencies, followed by an SGP module [ 31 ] for fine-grained local motion reasoning. During pre-training, a semantic alignment loss ( Lemb ) aligns learned features with word embeddings of action labels.
Method
Top-1
F1 macro
TSN [ECCV’16] [ 34 ]
34.46
28.52
TIN [AAAI’20] [ 30 ]
52.81
39.82
TSM [ICCV’19] [ 19 ]
56.75
40.19
C3D [ICCV’15] [ 32 ]
52.22
40.86
I3D [CVPR’17] [ 5 ]
57.07
39.84
SlowFast [ICCV’19] [ 9 ]
59.60
44.96
Table 1: Performance comparison of MAR on the MA-52 dataset.
Method
MPII-GI
SocialGesture
Top-1
F1 macro
Top-1
F1 macro
I3D [CVPR’17]
62.24
34.62
83.54
25.54
VSwinT [CVPR’22]
57.16
30.73
85.45
42.58
TimesFormer [ICML’21]
49.87
17.79
84.88
45.38
MANet [TCSVT’24]
66.86
41.63
85.74
47.76
MoNE [NeurIPS’24]
30.85
4.16
82.34
22.63
Table 2: Comparison of MAR performance on MPII-GI and SocialGesture.
Semantic
Motion
Top-1
F1 macro
✓
✗
59.31
47.82
✗
✓
60.90
48.98
✓
✓
64.54
53.30
Table 3: Ablation study on the impact of the dual-stream encoder (semantic and motion) on MA-52 dataset. A ✓ indicates that the corresponding stream is included in the model, while an ✗ indicates that it is removed.
#
Head
Body
Upper-Limb
Lower-Limb
Top-1
F1 macro
1
✗
✓
✓
✓
62.01
51.72
2
✓
✗
✓
✓
64.21
52.14
3
✓
✓
✗
✓
63.76
51.94
4
✓
✓
✓
✗
63.94
52.62
5
✓
✓
✓
✓
64.54
53.30
Table 4: Ablation experiments on the impact of the experts on MA-52. A ✓ indicates that the corresponding expert is included in the architecture, while ✗ denotes its removal.
Figure 5: Cross-attention heatmap showing expert importance per micro-action class on MA-52. Brighter colors indicate higher expert activation.
Figure 6: Per-class comparison of B-MoE and MaNet on the validation set for MA-52. Classes enclosed by black, blue, and red boxes correspond to ambiguous , underrepresented , and subtle actions, respectively. The column Train_N indicates the percentage of samples for each class.
Micro-actions are subtle, short, low-amplitude body movements, such as a fidgeting hand or a slight head tilt, that humans perform with little conscious intent yet that reliably leak emotional and psychological state. Understanding them goes beyond assigning a label: a model must also describe which body parts move and reason, faithfully, about why a clip warrants a particular fine-grained category. We present the training-free, prompt-only system that won first place in the fine-grained understanding track (MA-Bench) of the MAC~2026 Micro-Action Challenge, where both fine-tuning and ground-truth supervision are disallowed. Built entirely upon frozen multimodal large language models (MLLMs), the system dynamically routes each of the eight sub-tasks to the MLLM empirically best suited for that task: a discriminative MLLM for closed-ended recognition tasks and a generative MLLM for open-ended description and reasoning tasks. This architecture achieves a statistically significant performance advantage on open-ended tasks, attaining an average score of 2.68 (on a five-point scale) compared to 1.44 for the second-best approach.
With the rapid development of Multimodal Large Language Models (MLLMs), their potential in Micro-Action understanding, a vital role in human emotion analysis, remains unexplored due to the absence of specialized benchmarks. To tackle this issue, we present MA-Bench, a benchmark comprising 1,000 videos and a three-tier evaluation architecture that progressively examines micro-action perception, relational comprehension, and interpretive reasoning. MA-Bench contains 12,000 structured question-answer pairs, enabling systematic assessment of both recognition accuracy and action interpretation. The results of 23 representative MLLMs reveal that there are significant challenges in capturing motion granularity and fine-grained body-part dynamics. To address these challenges, we further construct MA-Bench-Train, a large-scale training corpus with 20.5K videos annotated with structured micro-action captions for fine-tuning MLLMs. The results of Qwen3-VL-8B fine-tuned on MA-Bench-Train show clear performance improvements across micro-action reasoning and explanation tasks. Our work aims to establish a foundation benchmark for advancing MLLMs in understanding subtle micro-action and human-related behaviors. Project Page: https://MA-Bench.github.io
Kun Li, Jihao Gu, Fei Wang +3
CVLab, College of Information Technology, United Arab Emirates University · University College London · Hefei University of Technology +2
Micro-actions are subtle, localized movements lasting 1-3 seconds such as scratching one's head or tapping fingers. Such subtle actions are essential for social communication, ubiquitously used in natural interactions, and thus critical for fine-grained video understanding, yet remain poorly understood by current computer vision systems. We identify a fundamental challenge: micro-actions exhibit diverse spatio-temporal characteristics where some are defined by spatial configurations while others manifest through temporal dynamics. Existing methods that commit to a single spatio-temporal decomposition cannot accommodate this diversity. We propose a dual-path network that processes anatomically-grounded spatial entities through parallel Spatial-Temporal (ST) and Temporal-Spatial (TS) pathways. The ST path captures spatial configurations before modeling temporal dynamics, while the TS path inverts this order to prioritize temporal dynamics. Rather than fixed fusion, we introduce entity-level adaptive routing where each body part learns its optimal processing preference, complemented by Mutual Action Consistency (MAC) loss that enforces cross-path coherence. Extensive experiments demonstrate competitive performance on MA-52 dataset and state-of-the-art results on iMiGUE dataset. Our work reveals that architectural adaptation to the inherent complexity of micro-actions is essential for advancing fine-grained video understanding.
Naga VS Raviteja Chappa, Evangelos Sariyanidi, Lisa Yankowitz +4
The Children’s Hospital of Philadelphia, USA · University of Pennsylvania, USA