De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift

Mengyuan Liu; Yuhang Wen; Yi Zhang; Songtao Wu; Hong Liu; Junsong Yuan; Beichen Ding

cs.CVSep 26, 2026

De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift

Authors: Mengyuan Liu, Yuhang Wen, Yi Zhang, Songtao Wu, Hong Liu, Junsong Yuan, Beichen Ding

Organizations: State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China. · School of Intelligent Systems Engineering, Sun Yat-sen University, China. · City University of Hong Kong. · R&D Center, Sony China Ltd., China. · University at Buffalo SUNY, USA. · School of Advanced Manufacturing & Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai), Sun Yat-sen University, China.

Abstract

Skeleton sequences can represent both individual actions and multi-entity interactions, encompassing human bodies, hands, objects, and robots. Existing approaches to recognize skeleton-based actions and interactions usually adopt a late fusion strategy, which expects individuals are independent and identically distributed to train a robust weight-shared entity encoder. However, observed entity bias in various skeletal data violates this assumption, leading to suboptimal optimization of backbone models that might produce wrong recognition results. This bias arises from the world coordinate system's initial configuration, where the choice of origin often creates bias in representation. To this end, we propose a Convex Hull Adaptive Shift based normalization method to reduce Entity bias (CHASE), improving performance across a variety of skeleton-based action and interaction recognition tasks. To adaptively apply plausible shifts to the input skeletons, we formulate a plug-and-play parameterized network that ensures the relocated world origin lies within the skeleton convex hull, which avoids non-convergence by limiting the search space. To further minimize entity bias, we incorporate an auxiliary objective that leverages pair-wise distribution distances to guide network optimization. To support both single- and multi-entity actions, we propose a sub-entity strategy that offers a consistent formulation for both scenarios. Moreover, CHASE demonstrates compatibility with various intra-skeleton modalities, such as bones and velocities, highlighting its adaptability. Essentially, our method works as a normalization approach to reduce entity bias, enabling subsequent classifiers to achieve improved recognition performance across diverse settings. Extensive experiments on 7 datasets verify our approach by seamlessly integrating with various backbones and significantly boosting their performance.

Figures & tables

Figure 1: Overview. (a) Existing methods train their models on raw skeletal data with inherent entity bias, resulting in unsatisfactory performance. (b) Our proposed CHASE works as a normalization method to de-bias the skeleton sequences. (c) It can significantly reduce the inter-entity distribution discrepancy and improve recognition performance. (d) CHASE help backbones achieve state-of-the-art results across a variety of action and interaction learning tasks.

Figure 2: The overall framework of the proposed CHASE for skeleton-based action recognition, including individual actions, person-person interactions, hand-object interactions, human-robot interactions, and group activities. For clarity, we offer an example of person-person interaction skeleton sequence for pipeline illustration. Given a skeleton sequence as input, CHASE leverages Convex Hull Adaptive Shift, a learnable normalization module, to generate plausible and sample-adaptive offset for each input. It also collects pair-wise shifted skeletons within mini-batches, effectively addressing entity bias by introducing an auxiliary objective. Sub-Entity Strategy allows CHASE to be applied in both individual and multi-entity actions. Moreover, CHASE can be adopted in various widely-used skeletal modalities defined using joints and physical topology for de-biasing.

Figure 3: Skeleton convex hulls. A skeleton convex hull, denoted using dark blue dashed lines, is defined by all the spatial and temporal keypoints in a skeleton sequence. All feasible new origins remain within the convex hull, which limits the search space for shift vector and avoids non-convergence.

Figure 4: Overview of sub-entity strategy . Black circles denote joints. A single entity can be viewed as many sub-entities of its own parts (denoted in different colors), which satisfy the subgraph and isomorphism conditions. Examples on human body skeletons are 2-part (2.P.) and 5-part (5.P.) partitions.

Benchmark	Skeleton Sequences				#Category	#Joint	#Entity	#Participant	Type	Example(s)
Benchmark	Body	Hand	Object	Robot	#Category	#Joint	#Entity	#Participant	Type	Example(s)
NTU60 ( Shahroudy et al., 2016 )	✓				60	25( $\times$ 2)	1 or 2	40	3D	hugging , walking towards
NTU120 ( Liu et al., 2020a )	✓				120	25( $\times$ 2)	1 or 2	106	3D	exchange things , whisper
H2O ( Kwon et al., 2021 )		✓	✓		36	21 $\times$ 3	3	4	3D	squeeze lotion out from bottle
Assembly101 ( Sener et al., 2022 )		✓			1,380	21 $\times$ 2	2	53	3D	screw track with hand
CAD ( Choi et al., 2009 )	✓				4	17 $\times$ 5.22	$\approx$ 5.22	Uncontrolled	2D	queuing , walking , talking

Table 1: Details of Benchmarks Used in Experiments

Method*	Venue	NTU26(%)		NTU11(%)		H2O(%)	ASB101(%)	CAD(%)	VD(%)	HARPER(%)
Method*	Venue	X-Sub	X-Set	X-Sub	X-View	H2O(%)	ASB101(%)	CAD(%)	VD(%)	HARPER(%)
Multi-Entity Backbones
LSTM-IRN ( Perez et al., 2022 )	TMM’22	77.70	79.60	90.50	93.50	-	-	-	-	-
IGFormer ( Pang et al., 2022 )	ECCV’22	85.40	86.50	93.60	96.50	-	-	-	-	-
SkeleTR ( Duan et al., 2023 )	ICCV’23	87.80	88.30	-	-	-	-	-	-	-
ISTA-Net ( Wen et al., 2023b )	IROS’23	90.56	91.72	-	-	89.09	28.01	87.16	91.40	79.63

Table 2: Comparisons with Related Methods on Multi-Entity Action Recognition Datasets. Our proposed CHASE can improve diverse single-entity backbones to achieve state-of-the-art performance across a variety of benchmarks.

Figure 5: Qualitative results of CHASE . Different entity distributions are denoted by blue and orange. By incorporating CHASE, entity distributions become more aligned in terms of both mean and covariance. CHASE effectively mitigates the inherent entity bias, demonstrating its clear effectiveness across a range of data scales and a variety of benchmarks.

Method (w/ CTR-GCN)	Acc (%)	$\Delta$ (%)
Vanilla	89.32	-
S2CoM	88.66	-0.67
BatchNorm	89.06	-0.27
ER	89.34	+0.02
Aug	89.72	+0.40
Per-Entity Learnable Offset	89.91	+0.59

Table 3: Comparison with Other Alternatives

Figure 6: More qualitative results . (a) UMAP ( McInnes et al., 2018 ) visualizations of skeleton sequence representations on the test split of NTU26 X-Sub. Compared with Vanilla, our proposed CHASE differentiate similar actions and interactions better by assisting backbones to learn more distinctive representations. (b) Category-level accuracy comparison between the vanilla baseline and CHASE. It demonstrates the effectiveness of CHASE by improving recognition accuracy for most categories. (c) Ablation study on $\lambda$ for MPMMD objective.

Set	Method	Avg KLD $\downarrow$	JSD $\downarrow$	BD $\downarrow$	HD $\downarrow$	MMD $\downarrow$
I	Vanilla	1.07	0.19	0.25	0.46	0.94
I	CHASE	0.39	0.08	0.10	0.30	0.05
II	Vanilla	1.00	0.18	0.23	0.45	1.03
II	CHASE	0.45	0.10	0.11	0.32	0.07
III	Vanilla	0.72	0.14	0.17	0.39	1.25
III	CHASE	0.41	0.08	0.10	0.30	0.05

Table 4: Analysis of Inter-entity Distribution Discrepancies

CHAS			MPMMD	LR	Acc (%)	$\Delta$ (%)
AS	CHC	CLB	MPMMD	LR	Acc (%)	$\Delta$ (%)
✓	✓	✓	✓	0.1	91.30	-
✓		✓	✓	0.1	22.65*	-68.65
✓		✓	✓	0.01	86.99	-4.32
✓	✓		✓	0.1	91.20	-0.10
✓			✓	0.1	22.75*	-68.56

Table 5: Analysis of Key Components in CHASE. * indicates non-convergence due to large search space of shift vector.

Method	Setting*	# Param.	NTU94(%)
Method	Setting*	# Param.	X-Sub	X-Set
CTR-GCN	Full Body	1.46M	82.69 (±0.10)	85.04 (±0.04)
CTR-GCN	2 Parts	1.44M	83.22 (±0.13)	85.81 (±0.03)
+ CHASE	2 Parts	1.46M	83.60 (±0.08)	85.85 (±0.01)
CTR-GCN	5 Parts	1.44M	81.70 (±0.10)	84.08 (±0.05)
+ CHASE	5 Parts	1.45M	81.99 (±0.07)	84.44 (±0.03)

Table 6: Sub-Entity Strategy on Single-Entity Actions

Modality	Method	Acc (%)
Joint	Vanilla	87.07
Joint	CHASE	87.42
Bone	Vanilla	88.31
Bone	CHASE	88.81
Joint Velocity	Vanilla	83.38
Joint Velocity	CHASE	83.50

Table 7: Intra-Skeleton Modalities and Ensemble

Method	Avg KLD $\downarrow$	JSD $\downarrow$	BD $\downarrow$	HD $\downarrow$	MMD $\downarrow$
Vanilla	1.898	0.300	0.473	0.567	0.914
CHASE	1.887	0.276	0.416	0.545	0.901

Table 8: Inter-Sub-Entity Distribution Discrepancies on NTU94

	CTR-GCN	InfoGCN	STSA-Net	HD-GCN	DeGCN
Vanilla	1.44M	1.54M	4.13M	1.65M	1.36M
CHASE	1.46M	1.57M	4.16M	1.68M	1.39M
$\Delta$	+1.83%	+1.96%	+0.60%	+1.60%	+1.93%

Table 9: Number of Learnable Parameters

Method	Venue	NTU120(%)		NTU60(%)
Method	Venue	X-Sub	X-Set	X-Sub	X-View
CTR-GCN	ICCV’21	88.90	90.60	92.40	96.80
InfoGCN	CVPR’22	89.80	91.20	93.00	97.10
STSA-Net	Neuro.’23	88.50	90.70	92.70	96.70
HD-GCN	ICCV’23	90.10	91.60	93.40	97.20
SkateFormer	ECCV’24	89.80	91.40	93.50	97.80

Table 10: Mixed Recognition of Single- & Multi-Entity Actions. * indicates this model is implemented and trained using the public code.

Test-Time	+ Noise (%)		+ Mask (%)
Test-Time	$\sigma=10^{-3}$	$\sigma=10^{-2}$	$p_{m}=10^{-2}$	$p_{m}=10^{-1}$
CTR-GCN	88.55	80.72	81.15	56.37
+ CHASE	91.24	82.53	88.57	60.65

Table 11: Test-Time Skeleton Noises and Masking

Explore similar work

SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition
Jun 2, 2026Yanan Liu, Anqi Zhu, Jingmin Zhu +6Skeleton-Based Action Recognition Cross-Modal Representation Learning
Yunnan University, Kunming, China · Monash University · Lancaster University +1
Towards Universal Skeleton-Based Action Recognition
Apr 18, 2026Jidong Kuang, Hongsong Wang, Jie GuiSkeleton-Based Action Recognition Human Activity Recognition
School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China
PoseBridge: Bridging the Skeletonization Gap for Zero-Shot Skeleton-Based Action Recognition
May 12, 2026Sanghyeon Lee, Jinwoo Kim, Jong Taek LeeSkeleton-Based Action Recognition Human Activity Recognition
School of Computer Science and Engineering Kyungpook National University, Daegu, South Korea

Jun 2, 2026J/K move · Enter open · S save

SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition

Yanan Liu, Anqi Zhu, Jingmin Zhu, Jun Liu+5

Yunnan University, Kunming, China · Monash University · Lancaster University+1

Skeleton-Based Action Recognition Cross-Modal Representation Learning Hierarchical Representation Learning Few-Shot Learning Hyperbolic Representation Learning

Open paper AskPDF arXiv

Skeleton-based action recognition aims to understand human behaviors from body joint sequences and is especially challenging in the one-shot setting, where only a single labeled exemplar is available for each novel action. A key challenge is learning representations that capture the hierarchical and compositional structure of human motion while aligning effectively with high-level action semantics under extreme data scarcity. Existing approaches, largely based on Euclidean embeddings and low-level motion cues, struggle to model the tree-like organization of skeleton data, limiting cross-modal alignment and generalization to unseen action categories. We propose SkelHCC, a unified skeleton hyperbolic CLIP-driven cache adaptation framework for one-shot skeleton-based action recognition. SkelHCC introduces an Explicitly Hierarchical Hyperbolic CLIP (EH-HCLIP) module that embeds skeleton sequences and action language into a shared hyperbolic space. By leveraging the negative curvature and exponential volume growth of hyperbolic geometry, EH-HCLIP naturally encodes the joint-part-body hierarchy of human anatomy and yields structurally consistent cross-modal representations. To support efficient one-shot adaptation, SkelHCC further integrates a training-free LLM-guided Multi-granularity Voting Cache (LMV-Cache) for context-aware inference. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD demonstrate that SkelHCC consistently outperforms state-of-the-art methods.

Abstract

Explore similar work

SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition

Towards Universal Skeleton-Based Action Recognition

PoseBridge: Bridging the Skeletonization Gap for Zero-Shot Skeleton-Based Action Recognition