Orthogonal Ensembles and Tested Explanations for Performer-Independent Body-Motion Emotion Recognition

Naoto Nishida; Yoshio Ishiguro

cs.CVSep 2, 2026

Orthogonal Ensembles and Tested Explanations for Performer-Independent Body-Motion Emotion Recognition

Authors: Naoto Nishida, Yoshio Ishiguro

Organizations: The University of Tokyo Tokyo, Japan

Abstract

We study body-only, 12-class acted-emotion classification from skeleton motion under leave-performer-out (LPO) evaluation, a hard, underdetermined setting: chance is 8.3%, and a protocol-matched reproduced STGCN++ baseline reaches only 25.73 +/- 4.03% Macro-F1. We show that reliable gains come not from a new architecture but from combining eleven models with orthogonal error modes: under 10-fold LPO cross-validation on the labeled training performers, an equal-weight logit-mean ensemble reaches 36.80 +/- 4.00% per-fold Macro-F1, a protocol-matched +11.07 pp (+43% relative) over the same-split reproduced baseline. Our central contribution is a tested explanation suite: for a strong ensemble member, part-masking and counterfactual edits show (rather than assert) that its decisions depend on motion-grounded body-region evidence, and this region saliency aligns with rule-based Laban Movement Analysis (LMA) attributes far more than with classical kinematics: region-level saliency-LMA Spearman rho = +0.500 versus +0.033, roughly 15x, and the alignment holds for the submitted 11-way ensemble itself at rho = +0.517; the audit is post hoc and needs no retraining. The same suite faithfully reports a negative: within-window temporal saliency is diffuse rather than localized. On the hidden challenge test set the submitted ensemble scored 37.23 % Macro-F1 and received the Best Performance Award of the MMAC Challenge 2026 (Human score is 39 %). Code is available at https://github.com/nawta/diema-challenge and the presentation at https://nawta.github.io/mmac2026/.

Figures & tables

Fig. 1: The DIEM-A 12-emotion body-motion challenge ( $74$ train / $18$ test performer-disjoint performers; BVH/FBX/C3D + text scenarios) and the proposed pipeline: eleven models spanning four inductive biases (GCN, attention, hybrid/MLP, external pretraining) fused by equal-weight logit-mean, reaching $36.80\pm 4.00\%$ per-fold Macro-F1 versus the same-split reproduced STGCN++ baseline ( $25.73\pm 4.03\%$ ). Protocol: 10-fold leave-performer-out on the 74-performer training split; per-fold mean $\pm$ SD throughout.

Approach	Configuration	Outcome ( $\Delta$ F1)	Methodological lesson
Supervised contrastive head	pairwise SupCon, $\lambda$ =0.05, batch 128	-11.53 pp, significant	12 classes at batch 128 give too few positives per class, suggesting a per-class center loss may fit better at this scale
Mixture-of-experts fusion	K=4, hand-crafted gradient-free routing	collapses to 6.55% F1	a shared head with random init and 21-D routing is associated with backbone collapse, suggesting a feature adapter, learned routing, and zero-init are needed
Scenario-text alignment	global InfoNCE, 3-fold	-1.42 pp	the projection collapses to performer identity (cross-group retrieval 10.6%)
Part-rationale alignment	cosine text alignment, 3-fold	-0.64 pp, CI [-1.30, -0.09]	the text bottleneck is associated with an F1 ceiling of 9–12%; shortcut-breaking works but yields no F1 lift
VLM zero-shot distillation	12-class, a recent VLM, preliminary	failed in preliminary zero-shot ranking	the true class ranked last (12/12); a vision-language model did not infer emotion reliably from faceless, scene-free skeleton renderings
Contact-marker late fusion	C3D 22-D contact stats, 3-fold	-0.59 pp	the contact features carry a 69.8% country signal, so some performers go out of distribution under LPO, suggesting a domain-invariance objective is needed

TABLE I: Approaches that did not improve the system, each paired with its methodological lesson (non-causal wording: associated with / regresses / does not improve).

\Delta

F1 is the paired change versus the matched baseline. The full catalogue is in the supplementary material.

Fig. 2: $12\times 12$ row-normalised pooled-OOF confusion of the 11-way logit-mean ensemble ( $n=7{,}992$ clips, 74-performer split; rows = true class, rows sum to $100\%$ ); the five strongest off-diagonal confusions are boxed. The per-class-by-country companion panel appears in the supplementary material.

System	Trainable params (M) ‡	Macro-F1 (mean $\pm$ SD)	Macro-F1 95% CI	Accuracy (mean $\pm$ SD)
STGCN++ official baseline	1.40 $\star$	25.21 $\pm$ 4.49 $\star$	n/a $\star$	27.11 $\pm$ 3.67 $\star$
STGCN++ reproduced	1.41	25.73 $\pm$ 4.03	[25.37, 27.23]	27.54 $\pm$ 3.92
Best single (Region-Aware)	1.03	30.05 $\pm$ 3.48	[29.32, 31.15]	30.87 $\pm$ 3.43
7-way ensemble	12.98	33.86 $\pm$ 2.92	[33.01, 35.04]	34.68 $\pm$ 3.00
11-way logit-mean (submitted)	12.98	36.80 $\pm$ 4.00	[35.90, 37.94]	37.40 $\pm$ 4.06

TABLE II: Main results on DIEM-A (10-fold LPO, 74-performer train split). Final row = submitted model. Macro-F1 / Accuracy = 10-fold LPO per-fold mean

\pm

SD (the official convention); fusion = logit-mean (mean of raw logits); 95% CI = sample-level paired bootstrap, 1000 iter, seed 42, on pooled OOF. Pooled-OOF F1 is a descriptive secondary: 7-way 34.03%, 11-way logit-mean 36.94%.

Component	Metric	Value	95% CI / null	What it shows
Part-masking	Faithfulness AUC gap (important-reverse), zero-mask	+0.124 $\pm$ 0.031	10-fold SD; positive in all folds	Masking important parts lowers F1: attributions are faithful
Temporal saliency	AUC gap (salient-reverse), negative result	+0.002 ( $\approx$ 0)	entropy 98.7% of log T ( $\approx$ uniform) †	Frame importance is diffuse (a reported negative)
Stability	Spearman $\rho$ vs $\sigma$ =0 ref at $\sigma$ =0.02	+0.983 $\pm$ 0.039	10-fold SD; top part never flips	Attribution ranking is stable under input noise
Counterfactual	Most-disruptive edit $\Delta$ p(true): head+freeze	-0.0164	mean over 864 val $\times$ 10 folds	Perturbing motion changes predictions: motion-dependent
Narrator grounding	Unsupported claims in audited cards	0/50 cards	deterministic audit	No unsupported claims in the 50 audited cards
F1 cost	$\Delta$ F1: baseline $\rightarrow$ with post-hoc saliency/LMA export	0.0 pp	post-hoc; no retraining	Explanations add no accuracy cost

TABLE III: Explainability metrics, with a plain reading of each row in the last column. Six rows are positive evidence (faithfulness, stability, counterfactual, narrator, submitted ensemble, LMA); temporal saliency is a deliberately reported negative. All values are recomputed on the same 10 LPO folds and out-of-fold predictions unless otherwise stated (10/10 folds for part-masking/stability). Bold = headline contrast.

Fig. S2: (A) $12\times 12$ row-normalised pooled-OOF confusion of the 11-way logit-mean ensemble ( $n=7{,}992$ clips; rows = true class, rows sum to $100\%$ ); the five strongest off-diagonal confusions are boxed. (B) 11-way per-class F1 by performer country (JP $n{=}40$ , TW $n{=}34$ training performers) with the signed JP $-$ TW per-class strip in percentage points. The JP–TW macro gap ( $35.3$ vs $38.9\%$ ) is a dataset stratum: the five OptiTrack-captured performers are absent from the training split, and excluding the seven earliest-captured Japanese training performers shifts pooled-OOF Macro-F1 by only $-0.44$ pp.

Member	Family	$\Delta$ Macro-F1 (pp)
MAMP-NTU60	External (frozen)	$-0.84$
SkateFormer	Attention	$-0.68$
MAMP-NTU120	External (frozen)	$-0.61$
Region-Aware ConvTr	Graph-conv.	$-0.59$
C3D-marker-stats	External (frozen)	$-0.52$
MotionBERT-Lite	External (frozen)	$-0.47$

TABLE S1: Leave-one-out ablation of the 11-member ensemble: change in pooled-OOF Macro-F1 (pp,

n\,{=}\,7{,}992

, 74-performer split) when one member is removed from the logit-mean fusion (last row: the whole frozen external block). Pooled-OOF base

36.94\%

(per-fold headline:

36.80\,{\pm}\,4.00\%

). All deltas are negative.

Member	Family	All	JP	TW	Excl. earliest JP
MAMP-NTU60	External (frozen)	$-0.84$	$-1.65$	$-0.01$	$-0.84$
SkateFormer	Attention	$-0.68$	$-0.48$	$-0.93$	$-0.78$
MAMP-NTU120	External (frozen)	$-0.61$	$-0.90$	$-0.30$	$-0.62$
Region-Aware ConvTr	Graph-conv.	$-0.59$	$-0.62$	$-0.57$	$-0.65$
C3D-marker-stats	External (frozen)	$-0.52$	$-0.64$	$-0.42$	$-0.54$
MotionBERT-Lite	External (frozen)	$-0.47$	$-0.72$	$-0.23$	$-0.49$

TABLE S2: Per-stratum leave-one-out deltas (pp, pooled OOF) for the 11-member logit-mean ensemble. ‘Excl. earliest JP’ scores after removing the seven earliest-captured Japanese training performers (JP_06–JP_12).

Laban axis	What it captures	Attributes (8 per axis)
Body	posture and bilateral form	head bow, trunk lean, arm openness, left–right speed asymmetry, left–right position asymmetry, stillness ratio, head lateral tilt, shoulder drop
Effort	motion quality (time/weight/flow)	suddenness, sustainedness, strong energy, light energy, bound flow, free flow, intensity peak, intensity variance
Shape	body volume and its change	body volume, shoulder width, head height, contraction change, rise/sink, spread change, arm enclosure, advance/recede
Space	trajectory and locomotion	directness, path curvature, horizontal root displacement, vertical root displacement, lateral sway, cumulative turn, dominant direction, locomotion ratio

TABLE S3: Rule-based Laban Movement Analysis (LMA) attribute schema: 32 clip-level features across the four Laban axes, computed from BVH-24 forward kinematics. The per-emotion z-score signature and the region aggregation

\ell_{c,r}

are defined in the main paper; this is the established movement vocabulary the saliency alignment is measured against.

Fig. S3: Per-emotion temporal and spatial saliency, full grid (companion to the temporal-saliency negative reported in the main paper).

Fig. S4: (A) Temporal saliency (mean over joints) is near-uniform for all 12 emotions; per-sample entropy $\approx 98.7\%$ of $\log(64)$ (class-mean $99.5\%$ ), the load-bearing negative. (B) Spatial saliency (row-normalised, joints in 6 body parts): Head is the top-1 joint for all 12 emotions (Head/Neck/Neck1 dominate). (C) For surprise (least-uniform), a joint $\times$ frame burst shows local structure survives class-averaged temporal uniformity. Colourblind-safe (viridis).

Fig. S5: (A) Mean LMA z-score per emotion ( $32$ attrs: Body/Effort/Shape/Space); right strip = per-emotion saliency–LMA Spearman. Region saliency–LMA $\rho=+0.500$ vs $+0.033$ for classical kinematics ( $15\times$ weaker). (B) Counterfactual motion edits (observational perturbations, not causal): mean $\Delta p_{\text{true}}$ over $864$ val $\times$ 10 folds; head-freeze most disruptive ( $-0.0164$ , $41\%$ flip), l_arm+amplify $23.1\%$ flip. Per-part saliency vs disruption Spearman $\rho=+0.49$ ( $n=6$ , $p=0.33$ ), a positive but modest triangulation.

Approach	Configuration	Outcome ( $\Delta$ F1)	Methodological lesson
Supervised contrastive head	pairwise SupCon, $\lambda$ =0.05, batch 128	-11.53 pp, significant	12 classes at batch 128 give too few positives per class, suggesting a per-class center loss may fit better at this scale
Mixture-of-experts fusion	K=4, hand-crafted gradient-free routing	collapses to 6.55% F1	a shared head with random init and 21-D routing is associated with backbone collapse, suggesting a feature adapter, learned routing, and zero-init are needed
Scenario-text alignment	global InfoNCE, 3-fold	-1.42 pp	the projection collapses to performer identity (cross-group retrieval 10.6%)
Part-rationale alignment	cosine text alignment, 3-fold	-0.64 pp, CI [-1.30, -0.09]	the text bottleneck is associated with an F1 ceiling of 9–12%; shortcut-breaking works but yields no F1 lift
VLM zero-shot distillation	12-class, a recent VLM, preliminary	failed in preliminary zero-shot ranking	the true class ranked last (12/12); a vision-language model did not infer emotion reliably from faceless, scene-free skeleton renderings
Contact-marker late fusion	C3D 22-D contact stats, 3-fold	-0.59 pp	the contact features carry a 69.8% country signal, so some performers go out of distribution under LPO, suggesting a domain-invariance objective is needed

TABLE S4: Supplementary: full catalogue of negative results (extends Table 2).