M3GCLR: Multi-View Mini-Max Infinite Skeleton-Data Game Contrastive Learning For Skeleton-Based Action Recognition

Yanshan Li; Ke Ma; Miaomiao Wei; Linhui Dai

cs.CVMar 10, 2026

M3GCLR: Multi-View Mini-Max Infinite Skeleton-Data Game Contrastive Learning For Skeleton-Based Action Recognition

Authors: Yanshan Li, Ke Ma, Miaomiao Wei, Linhui Dai

Organizations: Shenzhen University, the Institute of Intelligence Information Processing, and the Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen 518000, China

Abstract

In recent years, contrastive learning has drawn significant attention as an effective approach to reducing reliance on labeled data. However, existing methods for self-supervised skeleton-based action recognition still face three major limitations: insufficient modeling of view discrepancies, lack of effective adversarial mechanisms, and uncontrollable augmentation perturbations. To tackle these issues, we propose the Multi-view Mini-Max infinite skeleton-data Game Contrastive Learning for skeleton-based action Recognition (M3GCLR), a game-theoretic contrastive framework. First, we establish the Infinite Skeleton-data Game (ISG) model and the ISG equilibrium theorem, and further provide a rigorous proof, enabling mini-max optimization based on multi-view mutual information. Then, we generate normal-extreme data pairs through multi-view rotation augmentation and adopt temporally averaged input as a neutral anchor to achieve structural alignment, thereby explicitly characterizing perturbation strength. Next, leveraging the proposed equilibrium theorem, we construct a strongly adversarial mini-max skeleton-data game to encourage the model to mine richer action-discriminative information. Finally, we introduce the dual-loss equilibrium optimizer to optimize the game equilibrium, allowing the learning process to maximize action-relevant information while minimizing encoding redundancy, and we prove the equivalence between the proposed optimizer and the ISG model. Extensive Experiments show that M3GCLR achieves three-stream 82.1%, 85.8% accuracy on NTU RGB+D 60 (X-Sub, X-View) and 72.3%, 75.0% accuracy on NTU RGB+D 120 (X-Sub, X-Set). On PKU-MMD Part I and II, it attains 89.1%, 45.2% in three-stream respectively, all results matching or outperforming state-of-the-art performance. Ablation studies confirm the effectiveness of each component.

Figures & tables

Symbol	Description
$\Delta(\Omega)$	The set of all probability distributions over the sample space $\Omega$ .
$\times$ (or $\times_{i\in N}$ )	The Cartesian product of two or more sets, representing all possible ordered tuples. Example: if $A=\{1\}$ and $B=\{2,3\}$ , then $A\times B=\{(1,2),(1,3)\}$ .
$\Gamma=(N,(C_{i})_{i\in N},(u_{i})_{i\in N})$	A strategic-form game $\Gamma$ , where $N$ is the set of players, $C_{i}$ is the strategy set of player $i$ , and $u_{i}$ is the utility (or payoff) function of player $i$ .
$[\cdot]$	Pure strategy in a strategic game.
$-j$	The remaining set after removing element $j$ from $N$ , i.e., $-j=N\setminus\{j\}$ . Example: if $N=\{1,2,3\}$ , then $-2=\{1,3\}$ .
$\sigma_{i}$	A randomized strategy of player $i$ . Example: if $C_{1}=\{[a_{1}],[b_{1}]\}$ , then $\Delta(C_{1})=\{p[a_{1}]+(1-p)[b_{1}]\mid 0\leq p\leq 1\}$ , and $\sigma_{1}=0.5[a_{1}]+0.5[b_{1}]$ .

TABLE I: Significant Mathematical Symbols in This Section

Fig. 1: Pipeline of the proposed M 3 GCLR. The input sequence $\mathbf{X}^{(i)}$ is first processed by the Multi-view Rotation-based Augmentation Module, where the normal-augmentation Rotation Matrix, extreme-augmentation Rotation Matrix, and batch averaging are applied to generate a normally augmented data, an extremely augmented data, and the average data, respectively. These three views are fed into query encoder 1, query encoder 2, and the key encoder to obtain the feature embeddings through an MLP projection head. In the Mutual-information-based Mini-Max Infinite Skeleton-data Game Module, the mean mutual information among feature embeddings is computed to construct the utility functions of the ISG. By updating the encoder parameters to maximize the ISG utilities, strong adversarial feature learning is achieved. Finally, in the Dual-Loss-based Equilibrium Optimizer, the optimization is performed using both the $\mathcal{L}_{\mathrm{Push}}$ loss and the KL-divergence-based $\mathcal{MI}$ objective, forming the final loss $\mathcal{L}$ . This process further optimizes the model parameters and ensures the convergence of the ISG.

Fig. 2: Block diagram of MRAM. Rotation Matrices (RMs) around the $x$ , $y$ , $z$ axes $\mathbf{R}_{x}(\theta)$ , $\mathbf{R}_{y}(\theta)$ , $\mathbf{R}_{z}(\theta)$ are combined through matrix multiplication to obtain a multi-axis RM $\mathbf{R}_{xyz}(\theta)$ . After the input skeleton sequence is processed by the Multi-view Rotation-based Augmentation Module, three transformed views are generated: (a) the average data $\bar{\mathbf{X}}^{(i)}$ obtained by batch averaging, (b) the normally augmented data $\hat{\mathbf{X}}^{(i)}$ produced by multiplying the input with the normal-augmented RM $\mathbf{R}_{xyz}(\theta_{normal})$ , and (c) the extremely augmented data $\tilde{\mathbf{X}}^{(i)}$ derived from the extreme-augmented RM $\mathbf{R}_{xyz}(\theta_{extreme})$ . These augmented sequences are then fed into the subsequent Mutual-information-based Mini-Max Infinite Skeleton-data Game Module to facilitate robust adversarial representation learning.

Fig. 3: Visualizations of mean-motion sequences (the another isolated sequence represents another skeleton instance).

Fig. 4: Block diagram of M 3 ISGM. The average data $\bar{\mathbf{X}}^{(i)}$ , the normally augmented data $\hat{\mathbf{X}}^{(i)}$ , and the extremely augmented data $\tilde{\mathbf{X}}^{(i)}$ produced by the Multi-view Rotation-based Augmentation Module are fed into the Mutual-information-based Mini-Max Infinite Skeleton-data Game Module (M 3 ISGM). After passing through the key encoder, query encoder 1 (encoder 1) and query encoder 2 (encoder 2), their respective feature representations $\bar{\mathbf{z}}^{(i)}$ , $\hat{\mathbf{z}}^{(i)}$ , and $\tilde{\mathbf{z}}^{(i)}$ are obtained. The mutual information between $\hat{\mathbf{z}}^{(i)}$ and $\bar{\mathbf{z}}^{(i)}$ is computed as $I_{1}$ , and the mutual information between $\tilde{\mathbf{z}}^{(i)}$ and $\bar{\mathbf{z}}^{(i)}$ is computed as $I_{2}$ . The squared difference between $I_{1}$ and $I_{2}$ is added to $I_{1}$ is obtained encoder 2’s utility function $u_{2}$ , while encoder 1’s utility function is the negation of encoder 2’s, i.e. $u_{1}=-u_{2}$ . Based on the obtained equilibrium solution, the parameters of query encoder 1 and query encoder 2 (i.e., $f_{q\_1}$ and $f_{q\_2}$ ) are updated accordingly, which further provides support for the equilibrium optimization based on dual-loss learning.

Fig. 5: Block diagram of DLEO. After obtaining encoded features from the Mutual-information-based Mini-Max Infinite Skeleton-data Game Module, we feed them into the Dual-Loss-based Equilibrium Optimizer (DLEO). In DLEO, we first compute the distributions among the negative feature $\mathbf{m}_{k}$ , the average feature $\bar{\mathbf{z}}^{(i)}$ , the normally augmented feature $\hat{\mathbf{z}}^{(i)}$ , and the extremely augmented feature $\tilde{\mathbf{z}}^{(i)}$ , formulated as $p(\hat{\mathbf{z}}^{(i)}|\bar{\mathbf{z}}^{(i)})$ , $p(\tilde{\mathbf{z}}^{(i)}|\bar{\mathbf{z}}^{(i)})$ , $p(\hat{\mathbf{z}}^{(i)}|\mathbf{m}_{k})$ , and $p(\tilde{\mathbf{z}}^{(i)}|\mathbf{m}_{k})$ . Based on these distributions, the InfoNCE loss $\mathcal{L}_{\mathrm{Push}}$ and the KL divergence $\mathcal{MI}$ are calculated. Finally, the overall loss function $\mathcal{L}$ is computed from $\mathcal{L}_{\mathrm{Push}}$ and $\mathcal{MI}$ , enabling multi-view Mini-Max game-driven contrastive learning to achieve more robust representation learning for skeleton-based action recognition.

System Configuration	Parameters
CUDA Version	Cuda release V11.8.89
Pytorch Version	Pytorch 1.13.0+cu117
Python Version	3.9.12
GPU Model	Nvidia GeForce RTX3090 (24G)

TABLE II: System Configuration

Fig. 6: Comparison results of the 3s-baseline methods and 3s-M 3 GCLR on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD Part I datasets under the linear evaluation protocol. 3s-M 3 GCLR achieves consistently superior performance across all methods.

Fig. 7: Feature distribution visualization of nine representative actions on the NTU RGB+D 60 dataset. The scatter maps show the encoded features after linear evaluation for AimCLR and M 3 GCLR, where M 3 GCLR exhibits more compact intra-class clustering and clearer inter-class separation.

Fig. 8: Confusion matrices of AimCLR and M 3 GCLR under the linear evaluation protocol on the NTU RGB+D 60 dataset, visualized for the Joint, Motion, and Bone streams. Each matrix is computed from 100 samples per class across 9 actions. M 3 GCLR consistently demonstrates higher recognition accuracy and reduced misclassification.

Method	Stream	NTU RGB+D 60 (%)		NTU RGB+D 120 (%)		PKU-MMD (%)
Method	Stream	X-Sub Acc. $\uparrow$	X-View Acc. $\uparrow$	X-Sub Acc. $\uparrow$	X-Set Acc. $\uparrow$	Part I Acc. $\uparrow$
SkeletonCLR [ 14 ]	Joint	68.3	76.4	56.8	55.9	80.9
AimCLR [ 8 ]	Joint	74.3	79.6	63.4	63.4	83.0
M 3 GCLR (Ours)	Joint	77.5+3.2	81.9+2.3	68.3+4.9	70.1+6.7	83.2+0.2
SkeletonCLR [ 14 ]	Motion	53.3	50.8	39.6	40.2	63.4
AimCLR [ 8 ]	Motion	66.8	70.6	57.3	54.4	72.0

TABLE III: Comparison Results of the Proposed M 3 GCLR and Baseline Methods on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD Datasets under the Linear Evaluation Protocol

Method	Backbone	NTU RGB+D 60 (%)		NTU RGB+D 120 (%)
Method	Backbone	X-Sub Acc. $\uparrow$	X-View Acc. $\uparrow$	X-Sub Acc. $\uparrow$	X-Set Acc. $\uparrow$
CorsSCLR (2021) [ 14 ]	ST-GCN	72.9	79.9	67.9	66.7
MCAE (2021) [ 33 ]	MCAE	65.6	74.7	52.8	54.7
CP-STN (2021) [ 34 ]	ST-GCN	69.4	76.6	55.7	54.7
ST-CL (2022) [ 35 ]	GCN	68.1	69.4	54.2	55.6
AimCLR (2022) [ 8 ]	ST-GCN	78.2	83.8	68.2	68.8

TABLE IV: Comparison Results of the Proposed M 3 GCLR and Other Approaches on the NTU RGB+D 60 and NTU RGB+D 120 Datasets under the Linear Evaluation Protocol

Method	Backbone	PKU-MMD (%)
Method	Backbone	Part I Acc. $\uparrow$	Part II Acc. $\uparrow$
CorsSCLR (2021) [ 14 ]	ST-GCN	84.9	21.2
AimCLR (2022) [ 8 ]	ST-GCN	87.8	38.5
SkeAttnCLR (2023) [ 22 ]	ST-GCN	83.8	—
ASAR (2023) [ 21 ]	ST-GCN	83.5	38.8
CMCS (2024) [ 38 ]	ST-GCN	88.1	39.6

TABLE V: Comparison Results of the Proposed M 3 GCLR and Other Approaches on the PKU-MMD Part I and Part II Dataset under the Linear Evaluation Protocol

Normal Augmentation	MRAM	M 3 ISGM & DLEO	NTU RGB+D 60 (%)
Normal Augmentation	MRAM	M 3 ISGM & DLEO	X-Sub Acc. $\uparrow$	X-View Acc. $\uparrow$
✓			74.9	79.5
	✓		71.2	77.1
✓	✓		77.4	82.5
✓	✓	✓	82.1	85.5

TABLE VI: Model-Wise Ablation Results of M 3 GCLR on the NTU RGB+D 60 Dataset (Linear Evaluation)

Rotation Angle Range (°)	[–15, 15]	[–30, 30]	[–60, 60]	[–120, 120]
Normal	82.1	80.2	75.4	75.3
Extreme	70.1	71.3	82.1	78.1

TABLE VII: Ablation Results of the Three Streams under Different Rotation Angle Ranges on the NTU RGB+D 60 X-Sub Dataset (Linear Evaluation)

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$\Delta(\Omega)$	The set of all probability distributions over the sample space $\Omega$ .
$\times$ (or $\times_{i\in N}$ )	The Cartesian product of two or more sets, representing all possible ordered tuples. Example: if $A=\{1\}$ and $B=\{2,3\}$ , then $A\times B=\{(1,2),(1,3)\}$ .
$\Gamma=(N,(C_{i})_{i\in N},(u_{i})_{i\in N})$	A strategic-form game $\Gamma$ , where $N$ is the set of players, $C_{i}$ is the strategy set of player $i$ , and $u_{i}$ is the utility (or payoff) function of player $i$ .
$[\cdot]$	Pure strategy in a strategic game.
$-j$	The remaining set after removing element $j$ from $N$ , i.e., $-j=N\setminus\{j\}$ . Example: if $N=\{1,2,3\}$ , then $-2=\{1,3\}$ .
$\sigma_{i}$	A randomized strategy of player $i$ . Example: if $C_{1}=\{[a_{1}],[b_{1}]\}$ , then $\Delta(C_{1})=\{p[a_{1}]+(1-p)[b_{1}]\mid 0\leq p\leq 1\}$ , and $\sigma_{1}=0.5[a_{1}]+0.5[b_{1}]$ .

Appendix

TABLE A-I: Significant Mathematical Symbols in Appendix

Fig. A-1: Illustration of the convergence process of the mini-max ISG toward equilibrium.

Fig. A-2: Illustration of the optimization process of DLEO.

Explore similar work

Apr 20, 2026

Beyond Binary Contrast: Modeling Continuous Skeleton Action Spaces with Transitional Anchors

Self-supervised contrastive learning has emerged as a powerful paradigm for skeleton-based action recognition by enforcing consistency in the embedding space. However, existing methods rely on binary contrastive objectives that overlook the intrinsic continuity of human motion, resulting in fragmented feature clusters and rigid class boundaries. To address these limitations, we propose TranCLR, a Transitional anchor-based Contrastive Learning framework that captures the continuous geometry of the action space. Specifically, the proposed Action Transitional Anchor Construction (ATAC) explicitly models the geometric structure of transitional states to enhance the model's perception of motion continuity. Building upon these anchors, a Multi-Level Geometric Manifold Calibration (MGMC) mechanism is introduced to adaptively calibrate the action manifold across multiple levels of continuity, yielding a smoother and more discriminative representation space. Extensive experiments on the NTU RGB+D, NTU RGB+D 120 and PKU-MMD datasets demonstrate that TranCLR achieves superior accuracy and calibration performance, effectively learning continuous and uncertainty-aware skeleton representations. The code is available at https://github.com/Philchieh/TranCLR.

Yingjie Feng, Yi Wang, Jiaze Wang +2

Harbin Institute of Technology, Shenzhen · Central South University · FitX Technology (Hong Kong) Limited

Skeleton-Based Action Recognition Contrastive Learning Representation Learning Human Activity Recognition Self-Supervised Learning

Ask PDF

Oct 9, 2026

QCLR: A Plug-in Robustification Module via Quantization for Contrastive Learning in Self-Supervised Skeleton-Based Human Action Recognition

Skeleton-based human action recognition has gained increasing attention due to the stability of skeleton data across diverse environments and advances in motion capture technologies. Following the success of supervised learning approaches, self-supervised learning approaches, particularly contrastive learning ones, have achieved impressive performance. However, they remain vulnerable to adversarial attacks. In this work, we propose a plug-in robustification module via quantization for contrastive learning in self-supervised skeleton-based human action recognition (QCLR), which enhances the robustness of representations against adversarial attacks while preserving their representational capability. Specifically, we employ a dual-level quantization scheme across the encoding pipeline, which breaks the gradient propagation and absorbs small adversarial perturbations within the quantization bins. To mitigate the impact of inevitable quantization error, we further design an attention-guided multi-faceted quantization strategy to maintain the representation quality. Experiments on several benchmark datasets demonstrate that integrating QCLR into a progressive lineage of mainstream contrastive learning models substantially improves robustness against both gradient-based white-box and gradient-free black-box adversarial attacks, while preserving highly competitive classification accuracy.

Nanzhu Chen, Feixiang He, Yunfeng Diao +2

University of Leeds, Leeds, United Kingdom · Central South University, Changsha, China · Hefei University of Technology, Hefei, China +1

Adversarial Robustness Skeleton-Based Action Recognition Contrastive Learning

Ask PDF

Feb 26, 2026

Skarimva: Skeleton-based Action Recognition is a Multi-view Application

Human action recognition plays an important role when developing intelligent interactions between humans and machines. While there is a lot of active research on improving the machine learning algorithms for skeleton-based action recognition, not much attention has been given to the quality of the input skeleton data itself. This work demonstrates that by making use of multiple camera views to triangulate more accurate 3D~skeletons, the performance of state-of-the-art action recognition models can be improved significantly. This suggests that the quality of the input data is currently a limiting factor for the performance of these models. Based on these results, it is argued that the cost-benefit ratio of using multiple cameras is very favorable in most practical use-cases, therefore future research in skeleton-based action recognition should consider multi-view applications as the standard setup.

Daniel Bermuth, Alexander Poeppel, Wolfgang Reif

Institute for Software and Systems Engineering, University of Augsburg, Germany

Skeleton-Based Action Recognition

Ask PDF