De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift
Authors: Mengyuan Liu, Yuhang Wen, Yi Zhang, Songtao Wu, Hong Liu, Junsong Yuan, Beichen Ding
Organizations: State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China. · School of Intelligent Systems Engineering, Sun Yat-sen University, China. · City University of Hong Kong. · R&D Center, Sony China Ltd., China. · University at Buffalo SUNY, USA. · School of Advanced Manufacturing & Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai), Sun Yat-sen University, China.
Skeleton sequences can represent both individual actions and multi-entity interactions, encompassing human bodies, hands, objects, and robots. Existing approaches to recognize skeleton-based actions and interactions usually adopt a late fusion strategy, which expects individuals are independent and identically distributed to train a robust weight-shared entity encoder. However, observed entity bias in various skeletal data violates this assumption, leading to suboptimal optimization of backbone models that might produce wrong recognition results. This bias arises from the world coordinate system's initial configuration, where the choice of origin often creates bias in representation. To this end, we propose a Convex Hull Adaptive Shift based normalization method to reduce Entity bias (CHASE), improving performance across a variety of skeleton-based action and interaction recognition tasks. To adaptively apply plausible shifts to the input skeletons, we formulate a plug-and-play parameterized network that ensures the relocated world origin lies within the skeleton convex hull, which avoids non-convergence by limiting the search space. To further minimize entity bias, we incorporate an auxiliary objective that leverages pair-wise distribution distances to guide network optimization. To support both single- and multi-entity actions, we propose a sub-entity strategy that offers a consistent formulation for both scenarios. Moreover, CHASE demonstrates compatibility with various intra-skeleton modalities, such as bones and velocities, highlighting its adaptability. Essentially, our method works as a normalization approach to reduce entity bias, enabling subsequent classifiers to achieve improved recognition performance across diverse settings. Extensive experiments on 7 datasets verify our approach by seamlessly integrating with various backbones and significantly boosting their performance.
Figures & tables
Figure 1: Overview. (a) Existing methods train their models on raw skeletal data with inherent entity bias, resulting in unsatisfactory performance. (b) Our proposed CHASE works as a normalization method to de-bias the skeleton sequences. (c) It can significantly reduce the inter-entity distribution discrepancy and improve recognition performance. (d) CHASE help backbones achieve state-of-the-art results across a variety of action and interaction learning tasks.
Figure 2: The overall framework of the proposed CHASE for skeleton-based action recognition, including individual actions, person-person interactions, hand-object interactions, human-robot interactions, and group activities. For clarity, we offer an example of person-person interaction skeleton sequence for pipeline illustration. Given a skeleton sequence as input, CHASE leverages Convex Hull Adaptive Shift, a learnable normalization module, to generate plausible and sample-adaptive offset for each input. It also collects pair-wise shifted skeletons within mini-batches, effectively addressing entity bias by introducing an auxiliary objective. Sub-Entity Strategy allows CHASE to be applied in both individual and multi-entity actions. Moreover, CHASE can be adopted in various widely-used skeletal modalities defined using joints and physical topology for de-biasing.
Figure 3: Skeleton convex hulls. A skeleton convex hull, denoted using dark blue dashed lines, is defined by all the spatial and temporal keypoints in a skeleton sequence. All feasible new origins remain within the convex hull, which limits the search space for shift vector and avoids non-convergence.
Figure 4: Overview of sub-entity strategy . Black circles denote joints. A single entity can be viewed as many sub-entities of its own parts (denoted in different colors), which satisfy the subgraph and isomorphism conditions. Examples on human body skeletons are 2-part (2.P.) and 5-part (5.P.) partitions.
Benchmark
Skeleton Sequences
#Category
#Joint
#Entity
#Participant
Type
Example(s)
Body
Hand
Object
Robot
NTU60 ( Shahroudy et al., 2016 )
✓
60
25( × 2)
1 or 2
40
3D
hugging , walking towards
NTU120 ( Liu et al., 2020a )
✓
120
25( × 2)
1 or 2
106
3D
exchange things , whisper
H2O ( Kwon et al., 2021 )
✓
✓
36
21 × 3
3
4
3D
squeeze lotion out from bottle
Assembly101 ( Sener et al., 2022 )
✓
1,380
21 × 2
2
53
3D
screw track with hand
CAD ( Choi et al., 2009 )
✓
4
17 × 5.22
≈ 5.22
Uncontrolled
2D
queuing , walking , talking
Table 1: Details of Benchmarks Used in Experiments
Method*
Venue
NTU26(%)
NTU11(%)
H2O(%)
ASB101(%)
CAD(%)
VD(%)
HARPER(%)
X-Sub
X-Set
X-Sub
X-View
Multi-Entity Backbones
LSTM-IRN ( Perez et al., 2022 )
TMM’22
77.70
79.60
90.50
93.50
-
-
-
-
-
IGFormer ( Pang et al., 2022 )
ECCV’22
85.40
86.50
93.60
96.50
-
-
-
-
-
SkeleTR ( Duan et al., 2023 )
ICCV’23
87.80
88.30
-
-
-
-
-
-
-
ISTA-Net ( Wen et al., 2023b )
IROS’23
90.56
91.72
-
-
89.09
28.01
87.16
91.40
79.63
Table 2: Comparisons with Related Methods on Multi-Entity Action Recognition Datasets. Our proposed CHASE can improve diverse single-entity backbones to achieve state-of-the-art performance across a variety of benchmarks.
Figure 5: Qualitative results of CHASE . Different entity distributions are denoted by blue and orange. By incorporating CHASE, entity distributions become more aligned in terms of both mean and covariance. CHASE effectively mitigates the inherent entity bias, demonstrating its clear effectiveness across a range of data scales and a variety of benchmarks.
Method (w/ CTR-GCN)
Acc (%)
Δ (%)
Vanilla
89.32
-
S2CoM
88.66
-0.67
BatchNorm
89.06
-0.27
ER
89.34
+0.02
Aug
89.72
+0.40
Per-Entity Learnable Offset
89.91
+0.59
Table 3: Comparison with Other Alternatives
Figure 6: More qualitative results . (a) UMAP ( McInnes et al., 2018 ) visualizations of skeleton sequence representations on the test split of NTU26 X-Sub. Compared with Vanilla, our proposed CHASE differentiate similar actions and interactions better by assisting backbones to learn more distinctive representations. (b) Category-level accuracy comparison between the vanilla baseline and CHASE. It demonstrates the effectiveness of CHASE by improving recognition accuracy for most categories. (c) Ablation study on λ for MPMMD objective.
Set
Method
Avg KLD ↓
JSD ↓
BD ↓
HD ↓
MMD ↓
I
Vanilla
1.07
0.19
0.25
0.46
0.94
CHASE
0.39
0.08
0.10
0.30
0.05
II
Vanilla
1.00
0.18
0.23
0.45
1.03
CHASE
0.45
0.10
0.11
0.32
0.07
III
Vanilla
0.72
0.14
0.17
0.39
1.25
CHASE
0.41
0.08
0.10
0.30
0.05
Table 4: Analysis of Inter-entity Distribution Discrepancies
CHAS
MPMMD
LR
Acc (%)
Δ (%)
AS
CHC
CLB
✓
✓
✓
✓
0.1
91.30
-
✓
✓
✓
0.1
22.65*
-68.65
✓
✓
✓
0.01
86.99
-4.32
✓
✓
✓
0.1
91.20
-0.10
✓
✓
0.1
22.75*
-68.56
Table 5: Analysis of Key Components in CHASE. * indicates non-convergence due to large search space of shift vector.
Method
Setting*
# Param.
NTU94(%)
X-Sub
X-Set
CTR-GCN
Full Body
1.46M
82.69 (±0.10)
85.04 (±0.04)
CTR-GCN
2 Parts
1.44M
83.22 (±0.13)
85.81 (±0.03)
+ CHASE
2 Parts
1.46M
83.60 (±0.08)
85.85 (±0.01)
CTR-GCN
5 Parts
1.44M
81.70 (±0.10)
84.08 (±0.05)
+ CHASE
5 Parts
1.45M
81.99 (±0.07)
84.44 (±0.03)
Table 6: Sub-Entity Strategy on Single-Entity Actions
Modality
Method
Acc (%)
Joint
Vanilla
87.07
CHASE
87.42
Bone
Vanilla
88.31
CHASE
88.81
Joint Velocity
Vanilla
83.38
CHASE
83.50
Table 7: Intra-Skeleton Modalities and Ensemble
Method
Avg KLD ↓
JSD ↓
BD ↓
HD ↓
MMD ↓
Vanilla
1.898
0.300
0.473
0.567
0.914
CHASE
1.887
0.276
0.416
0.545
0.901
Table 8: Inter-Sub-Entity Distribution Discrepancies on NTU94
CTR-GCN
InfoGCN
STSA-Net
HD-GCN
DeGCN
Vanilla
1.44M
1.54M
4.13M
1.65M
1.36M
CHASE
1.46M
1.57M
4.16M
1.68M
1.39M
Δ
+1.83%
+1.96%
+0.60%
+1.60%
+1.93%
Table 9: Number of Learnable Parameters
Method
Venue
NTU120(%)
NTU60(%)
X-Sub
X-Set
X-Sub
X-View
CTR-GCN
ICCV’21
88.90
90.60
92.40
96.80
InfoGCN
CVPR’22
89.80
91.20
93.00
97.10
STSA-Net
Neuro.’23
88.50
90.70
92.70
96.70
HD-GCN
ICCV’23
90.10
91.60
93.40
97.20
SkateFormer
ECCV’24
89.80
91.40
93.50
97.80
Table 10: Mixed Recognition of Single- & Multi-Entity Actions. * indicates this model is implemented and trained using the public code.
Skeleton-based action recognition aims to understand human behaviors from body joint sequences and is especially challenging in the one-shot setting, where only a single labeled exemplar is available for each novel action. A key challenge is learning representations that capture the hierarchical and compositional structure of human motion while aligning effectively with high-level action semantics under extreme data scarcity. Existing approaches, largely based on Euclidean embeddings and low-level motion cues, struggle to model the tree-like organization of skeleton data, limiting cross-modal alignment and generalization to unseen action categories. We propose SkelHCC, a unified skeleton hyperbolic CLIP-driven cache adaptation framework for one-shot skeleton-based action recognition. SkelHCC introduces an Explicitly Hierarchical Hyperbolic CLIP (EH-HCLIP) module that embeds skeleton sequences and action language into a shared hyperbolic space. By leveraging the negative curvature and exponential volume growth of hyperbolic geometry, EH-HCLIP naturally encodes the joint-part-body hierarchy of human anatomy and yields structurally consistent cross-modal representations. To support efficient one-shot adaptation, SkelHCC further integrates a training-free LLM-guided Multi-granularity Voting Cache (LMV-Cache) for context-aware inference. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD demonstrate that SkelHCC consistently outperforms state-of-the-art methods.
Yanan Liu, Anqi Zhu, Jingmin Zhu +6
Yunnan University, Kunming, China · Monash University · Lancaster University +1
With the development of robotics, skeleton-based action recognition has become increasingly important, as human-robot interaction requires understanding the actions of humans and humanoid robots. Due to different sources of human skeletons and structures of humanoid robots, skeleton data naturally exhibit heterogeneity. However, previous works overlook the data heterogeneity of skeletons and solely construct models using homogeneous skeletons. Moreover, open-vocabulary action recognition is also essential for real-world applications. To this end, this work studies the challenging problem of heterogeneous skeleton-based action recognition with open vocabularies. We construct a large-scale Heterogeneous Open-Vocabulary (HOV) Skeleton dataset by integrating and refining multiple representative large-scale skeleton-based action datasets. To address universal skeleton-based action recognition, we propose a Transformer-based model that comprises three key components: unified skeleton representation, motion encoder for skeletons, and multi-grained motion-text alignment. The motion encoder feeds multi-modal skeleton embeddings into a two-stream Transformer-based encoder to learn spatio-temporal action representations, which are then mapped to a semantic space to align with text embeddings. Multi-grained motion-text alignment incorporates contrastive learning at three levels: global instance alignment, stream-specific alignment, and fine-grained alignment. Extensive experiments on popular benchmarks with heterogeneous skeleton data demonstrate both the effectiveness and the generalization ability of the proposed method. Code is available at https://github.com/jidongkuang/Universal-Skeleton.
Jidong Kuang, Hongsong Wang, Jie Gui
School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China
Zero-shot skeleton-based action recognition (ZSSAR) is typically treated as a skeleton-text alignment problem: encode joint-coordinate sequences, align them with language, and classify unseen actions. We argue that this alignment is often too late. Skeletons are not complete action observations, but compressed outputs of human pose estimation (HPE); by the time alignment begins, human-object interactions and pose-relative visual cues may no longer be explicit. We call this upstream semantic loss. To address it, we propose PoseBridge, an HPE-aware ZSSAR framework that bridges intermediate HPE representations to skeleton-text alignment. Rather than adding an RGB action branch or object detector, PoseBridge extracts pose-anchored semantic cues from the same HPE process that produces skeletons, then transfers them through skeleton-conditioned bridging and semantic prototype adaptation. Across NTU-RGB+D 60/120, PKU-MMD, and Kinetics-200/400, PoseBridge improves ZSSAR performance under the evaluated protocols. On the Kinetics-200/400 PURLS benchmark, which contains in-the-wild videos with diverse scenes and action contexts, PoseBridge shows the clearest separation, improving the strongest compared baseline by 13.3-17.4 points across all eight splits. Our code will be publicly released.
Sanghyeon Lee, Jinwoo Kim, Jong Taek Lee
School of Computer Science and Engineering Kyungpook National University, Daegu, South Korea