The practical deployment gap -- transitioning from controlled multi-view 3D skeleton capture to unconstrained monocular 2D pose estimation -- introduces a compound domain shift whose safety implications remain critically underexplored. We present a systematic study of this severe domain shift using a novel Gym2D dataset (style/viewpoint shift) and the UCF101 dataset (semantic shift). Our Skeleton Transformer achieves 63.2% cross-subject accuracy on NTU-120 but drops to 1.6% under zero-shot transfer to the Gym domain and 1.16% on UCF101. Critically, we demonstrate that high Out-Of-Distribution (OOD) detection AUROC does not guarantee safe selective classification. Standard uncertainty methods fail to detect this performance drop: the model remains confidently incorrect with 99.6% risk even at 50% coverage across both OOD datasets. While energy-based scoring (AUROC >= 0.91) and Mahalanobis distance provide reliable distributional detection signals, such high AUROC scores coexist with poor risk-coverage behavior when making decisions. A lightweight finetuned gating mechanism restores calibration and enables graceful abstention, substantially reducing the rate of confident wrong predictions. Our work challenges standard deployment assumptions, providing a principled safety analysis of both semantic and geometric skeleton recognition deployment.
Figures & tables
Fig. 1 : t-SNE visualization of Skeleton Transformer features. NTU-120 (ID) classes form well-separated clusters, while Gym2D (OOD) samples (red) collapse into the interior of NTU clusters despite being systematically mis-classified. This explains why softmax confidence remains high for wrong predictions.
Model
NTU-120 (%)
Gym2D (%)
UCF101 (%)
Skeleton Transformer
63.2 ± 0.3
1.60
0.73 ± 0.04
ST-GCN [ 1 ]
58.9
0.88
0.82
TABLE I : Zero-Shot Transfer: NTU-120 vs. OOD Domains
Fig. 2 : Energy score distributions for NTU-120 (ID, blue) and Gym2D (OOD, red). Clear distributional separation indicates reliable OOD detection , but does not resolve the risk-coverage failure (Fig. 4 ).
OOD Score
AUROC
Max Softmax Probability (MSP)
0.9428
Temperature-Scaled MSP ( T∗=1.324 )
0.9428
MC Dropout Entropy ( N=20 )
0.8108
Deep Ensemble Disagreement ( K=3 )
0.9808
Mahalanobis Distance
0.9015
TABLE II : OOD Detection AUROC (NTU-120 ID vs. Gym2D OOD)
Fig. 3 : ROC curves for Mahalanobis OOD detection. The Skeleton Transformer (AUROC =0.902 ) outperforms ST-GCN, correctly identifying out-of-distribution physical postures where confidence-based metrics fail.
Fig. 4 : Risk-Coverage curves on the Gym2D (OOD) dataset. Left : frozen gating and zero-shot baselines maintain ≈100% risk at all coverage levels, confirming that standard UQ methods fail to identify correct OOD predictions. Right : finetuned gating achieves meaningful risk reduction with decreasing coverage, enabling graceful abstention.
Variant
Gym Acc. (%)
Risk@50%
WSR@50%
Zero-shot (no adapt.)
1.6
0.982
0.491
Frozen gating
1.6
0.986
0.493
Finetuned gating
37.3
0.569
0.285
+ Temperature Scaling
37.3
0.481
0.241
ST-GCN (zero-shot)
0.88
0.994
0.497
Deep Ensemble K=3
0.68
0.994
0.497
TABLE III : Adaptation Results on Gym2D OOD Dataset
Fig. 5 : Reliability diagram for the finetuned gating model on Gym2D. Calibrated confidence now tracks actual accuracy—a necessary pre-condition for meaningful selective classification.
Fig. 6 : Per-class accuracy on Gym2D (finetuned gating). The model achieves >75% accuracy on clear rotational and transitional movements while continuing to fail on visually ambiguous poses ( e.g. , squat vs. RDL).
Fig. 7 : Per-class accuracy on UCF101 (zero-shot Skeleton Transformer). Unlike the purely geometric shift in Gym2D, UCF101 suffers from semantic shift: the model identifies gymnastics and acrobatic movements with unique spatial coordinates ( e.g. , 96% on aerial actions) but fails on subtle terrestrial actions.
Method
Dataset
Acc (mean ± std)
Risk@50%
WSR@50%
AUROC (vs ID)
Zero-Shot (MC Dropout)
Gym2D
1.60%±0.0
99.4%±0.3
49.7%±0.2
0.811
UCF101
1.16%±0.05
99.5%±0.1
49.7%±0.0
0.827
Linear Probe (Gated)
Gym2D
12.3%±0.8
82.1%±1.2
41.0%±0.6
0.844†
UCF101
22.15%±0.41
97.8%±0.4
48.9%±0.2
0.852†
Full Fine-Tune (Gated)
Gym2D
27.0%±0.6
68.2%±0.9
33.0%±0.4
0.845†
UCF101
45.74%±0.39
84.8%±2.7
42.4%±1.3
0.853†
TABLE IV : Unified OOD Robustness: Multi-Seed Statistics across Domains
School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China