Universal Skeleton Understanding via Differentiable Rendering and MLLMs

Ziyi Wang; Peiming Li; Xinshun Wang; Yang Tang; Kai-Kuang Ma; Mengyuan Liu

cs.CVMar 18, 2026

Universal Skeleton Understanding via Differentiable Rendering and MLLMs

Authors: Ziyi Wang, Peiming Li, Xinshun Wang, Yang Tang, Kai-Kuang Ma, Mengyuan Liu

Organizations: State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China · Tencent · Nanjing University of Aeronautics and Astronautics, Nanjing, Jiangsu, China

Abstract

Multimodal large language models (MLLMs) exhibit strong visual-language reasoning, yet cannot process structured, non-visual data such as human skeletons. Existing methods either compress skeleton dynamics into lossy feature vectors for text alignment, or quantize motion into discrete tokens that generalize poorly across heterogeneous skeleton formats. We present SkeletonLLM, which achieves universal skeleton understanding by translating arbitrary skeleton sequences into the MLLM's native visual modality. At its core is DrAction, a differentiable, format-agnostic renderer that converts skeletal kinematics into compact image sequences. Because the pipeline is end-to-end differentiable, MLLM gradients can directly guide the rendering to produce task-informative visual tokens. To further enhance reasoning capabilities, we introduce a cooperative training strategy: Causal Reasoning Distillation transfers structured, step-by-step reasoning from a teacher model, while Discriminative Finetuning sharpens decision boundaries between confusable actions. SkeletonLLM demonstrates strong generalization \revise{in open-vocabulary action recognition, while its learned reasoning capabilities naturally extend to motion captioning and question answering across heterogeneous skeleton formats} -- suggesting a viable path for applying MLLMs to non-native modalities. Code: https://github.com/wangzy01/SkeletonLLM.

Figures & tables

Panel of Figure 1 — Figure 1: Breaking Format Silos and the Modality Gap. (Top) MLLMs possess strong reasoning capabilities but cannot natively process structured skeleton data. (Middle) Traditional alignment methods are tied to specific skeleton topologies, compressing motion into a single vector for matching against text embeddings, which creates representation bottlenecks and brittle semantics. (Bottom) Our SkeletonLLM uses DrAction, a differentiable renderer that translates a single skeleton sequence of any format into the MLLM’s native visual language, enabling end-to-end optimization and unlocking powerful visual reasoning for diverse tasks.

Panel of Figure 2 — Figure 2: Overview of SkeletonLLM. The pipeline follows a Render-Reason-Respond process for universal understanding. Given a skeleton sequence, DrAction lifts joint trajectories into deformable 3D Gaussian primitives and renders motion-aware images. Joint transforms are computed via Linear Blend Skinning, and kinematic cues (depth, velocity) are fused through a Neural Feature Modulator. All parameters are optimized end-to-end by gradients from the MLLM. The rendered frames are processed by the MLLM’s vision encoder and a projector to yield visual tokens. During training, CR-Distill supervises with teacher-generated causal chains describing body-part dynamics, while Disc-FT sharpens decision boundaries via binary queries over confusing action pairs.

Methods	Type	Venue	NTU-60 (Acc, %)				NTU-120 (Acc, %)
Methods	Type	Venue	55/5	48/12	40/20	30/30	110/10	96/24	80/40	60/60
ReViSE ( Hubert Tsai et al., 2017 )	Align	ICCV’17	53.91	17.49	24.26	14.81	55.04	32.38	19.47	8.27
JPoSE ( Wray et al., 2019 )	Align	ICCV’19	64.82	28.75	20.05	12.39	51.93	32.44	13.71	7.65
CADA-VAE ( Schonfeld et al., 2019 )	Align	CVPRW’19	76.84	28.96	16.21	11.51	59.53	35.77	10.55	5.67
SynSE ( Gupta et al., 2021 )	Align	ICIP’21	75.81	33.30	19.85	12.00	62.69	38.70	13.64	7.73
PURLS ( Zhu et al., 2024 )	Align	CVPR’24	79.23	40.99	31.05	23.52	71.95	52.01	28.38	19.63

Table 1: Top-1 accuracy (%) of various methods on NTU-60 and NTU-120. Each split is denoted as X/Y, where X is the number of seen classes and Y is the number of unseen classes. The best results are in red , and the second-best are blue . † Results for Qwen2.5-VL-7B and InternVL3-8B were obtained by rendering skeletons with the non-learnable 3D+Velocity renderer (same as in Table 7 ;

448{\times}448

) and finetuning on MQA for 6 epochs.

Panel of Figure 3 — Figure 3: Cross-format rendering by DrAction. Top row: DrAction renders skeletons from four different formats into visually consistent image sequences. Bottom row: the underlying skeleton topologies vary significantly in joint count and connectivity—NW-UCLA (Kinect v1, 20 joints), NTU (Kinect v2, 25 joints), NTU-2D (pose estimation, 17 joints), and HumanML3D (MoCap, 22 joints). Despite these differences, DrAction produces a unified visual language, enabling seamless cross-format transfer.

Training	Testing	Skeleton Format	Method	Acc
NTU-60	NW-UCLA	Kinect v2 $\to$ Kinect v1	TDSM	43.19
NTU-60	NW-UCLA	Kinect v2 $\to$ Kinect v1	MotionGPT	10.35
NTU-60	NW-UCLA	Kinect v2 $\to$ Kinect v1	SKI-LVLM	31.87
NTU-60	NW-UCLA	Kinect v2 $\to$ Kinect v1	SkeletonLLM	60.38
HumanML3D	NW-UCLA	MoCap $\to$ Kinect v1	MotionGPT	9.11
HumanML3D	NW-UCLA	MoCap $\to$ Kinect v1	SKI-LVLM	28.74

Table 2: Cross-format transfer accuracy (%) for action recognition. Models are trained on the source dataset and evaluated directly on the target without finetuning.

Method	R@1 $\uparrow$	R@3 $\uparrow$	MM-Dist $\downarrow$	BLEU@4 $\uparrow$	ROUGE-L $\uparrow$	CIDEr $\uparrow$	BertScore $\uparrow$
MotionGPT	2.83	9.86	12.27	3.50	13.83	4.16	14.03
MotionLLM	4.90	14.34	9.67	5.57	19.72	5.08	23.44
InternVL3-8B	6.25	20.08	6.36	7.15	27.01	9.49	29.45
Ours	11.60	28.56	5.94	9.63	39.84	18.25	37.28

Table 3: Cross-format motion captioning on HumanML3D. Models are trained on NTU-60 (Kinect v2, 25 joints) with recognition supervision and evaluated on HumanML3D (SMPL, 22 joints) captioning without finetuning.

Task	Test Set	Format ( $J$ )	Superv.?	Metric
Open-vocab recog.	NTU-60 (55/5)	Kinect v2 (25)	✓	87.37% Acc
Cross-format recog.	NW-UCLA	Kinect v1 (20)	✗	60.38% Acc
Cross-format caption.	HumanML3D	SMPL (22)	✗	37.28 BertScore
Motion QA	Skeleton-QA	Kinect v2 (25)	✗	68 / 65% Acc

Table 4: Single-model multi-task evaluation. All results use the same NTU-60 (55/5) checkpoint with no task-specific retraining.

Training	NTU-60 55/5	NTU $\to$ NW-UCLA	Skeleton-QA (Temp / Causal)
NTU-60 only	87.37	60.38	68 / 65
Joint training	88.52	65.04	73.2 / 70.9

Table 5: Cross-dataset joint training. NTU-60 (25J) + HumanML3D (22J) with shared DrAction and MLLM.

Method	48/12	30/30
GPT-4o + 3D+Velocity	35.74	19.27
GPT-4o + DrAction	43.52	25.68
GPT-5.4 + DrAction	53.18	33.11
SkeletonLLM (joint-trained)	64.72	37.84

Table 6: DrAction paired with external closed-source MLLMs on NTU-60 zero-shot recognition. Joint training remains essential.

Panel of Figure 4 — Figure 4: Qualitative comparison of rendering methods. Fixed renderers (3D+Velocity, 2D, JTM ( Wang et al., 2016b ) ) produce visualizations that are either generic, information-poor, or perceptually complex. DrAction learns an abstract representation. With the NFM, it dynamically highlights kinematically salient regions (e.g., the kicking leg), producing a more informative visual language for the MLLM.

Renderer	NTU-60 (Acc, %)				NTU-120 (Acc, %)
Renderer	55/5	48/12	40/20	30/30	110/10	96/24	80/40	60/60
3D+Velocity	81.32	58.77	37.83	31.45	70.48	60.60	37.74	28.59
2D Projection	81.51	58.16	38.81	31.80	70.94	61.09	38.30	27.69
JTM	55.86	26.37	23.45	16.83	45.51	34.03	15.09	8.75
Ours (w/o NFM)	83.30	61.09	40.93	33.87	71.29	62.62	40.69	30.67
Ours (Full)	87.37	64.72	46.15	37.84	76.05	67.20	44.37	34.94

Table 7: Ablation on rendering methods on NTU-60 & NTU-120. Our differentiable DrAction outperforms non-learnable renderers.

Configuration	NTU-60 (Acc, %)				NTU-120 (Acc, %)
Configuration	55/5	48/12	40/20	30/30	110/10	96/24	80/40	60/60
w/o CR-Distill	85.27	63.37	43.06	35.90	74.68	64.97	43.96	33.79
w/o Disc-FT	85.80	63.66	45.58	36.29	75.25	65.90	43.05	33.40
w/o Both	84.43	62.09	40.78	35.05	72.95	64.05	41.13	32.77
Joint Training	82.68	61.46	40.30	34.10	72.42	62.03	40.18	30.94
Full Pipeline	87.37	64.72	46.15	37.84	76.05	67.20	44.37	34.94

Table 8: Ablation on progressive training strategy on NTU-60 & NTU-120.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Our Progressive Training Pipeline. To address the joint optimization challenge, we progressively activate and fine-tune model components. The training curriculum begins with (a) warming up the renderer to generate intelligible visuals and concludes with (d) refining recognition, both utilizing a multiple-choice question & answer (MQA) task. In between, the strategy incorporates (b) learning discriminative features via a binary judgment task and (c) instilling causal reasoning through knowledge distillation from a teacher model.

Component	Addresses	Key Mechanism
DrAction	Modality gap, format heterogeneity	Differentiable rendering with LBS
NFM	Information preservation	Pose-conditioned appearance
Cooperative Training	Optimization stability, reasoning depth	Progressive curriculum + distillation

Appendix

Table 9: Summary of SkeletonLLM components and their contributions.

Configuration	NTU-60 (Acc, %)				NTU-120 (Acc, %)
Configuration	55/5	48/12	40/20	30/30	110/10	96/24	80/40	60/60
w/o NFM	83.30	61.09	40.93	33.87	71.29	62.62	40.69	30.67
w/o Temporal	86.35	63.56	45.62	36.94	74.80	65.98	43.01	33.45
w/ RNN	86.33	64.45	45.99	37.18	75.97	66.89	43.17	34.36
w/ LSTM	86.65	64.95	45.41	37.62	75.72	67.43	43.54	34.27
w/ GRU (Full)	87.37	64.72	46.15	37.84	76.05	67.20	44.37	34.94

Appendix

Table 10: Ablation on NFM components and temporal modeling strategies on NTU-60 & NTU-120. We compare different temporal fusion methods (GRU, LSTM, RNN) within the NFM. The full NFM with GRU achieves the best performance across nearly all splits.

Panel of Figure 6 — Figure 6: Qualitative comparison of different rendering methods. Each column shows a representative action instance. From top to bottom, we compare three fixed renderers (3D+Velocity, 2D projection, and JTM ( Wang et al., 2016b ) ) with our learnable DrAction variants (w/o NFM and full DrAction). Static methods often miss or clutter fine-grained motion cues. In contrast, DrAction, particularly when enhanced by the Neural Feature Modulator (NFM), highlights motion-critical joints using adaptive color and opacity. These visualizations substantiate our claim that a task-optimized, differentiable renderer yields a more effective visual language for MLLMs than static, hand-designed representations. Additional rendered videos are available in the code repository: https://github.com/wangzy01/SkeletonLLM .

Methods	Type	Venue	Xsub (%)		Xview (%)
Methods	Type	Venue	46/5	39/12	46/5	39/12
ReViSE ( Hubert Tsai et al., 2017 )	Align	ICCV’17	54.2	19.3	54.1	12.7
JPoSE ( Wray et al., 2019 )	Align	ICCV’19	57.4	27.0	53.1	22.8
CADA-VAE ( Schonfeld et al., 2019 )	Align	CVPRW’19	73.9	33.7	74.5	29.5
SynSE ( Gupta et al., 2021 )	Align	ICIP’21	69.5	36.5	71.7	25.4
SMIE ( Zhou et al., 2023 )	Align	ACM MM’23	72.9	44.2	71.6	40.7

Appendix

Table 11: Top-1 accuracy (%) on the PKU-MMD dataset under cross-subject (Xsub) and cross-view (Xview) settings. We follow the splits from Chen et al. (2024a) . The best results are highlighted in red. † Results for Qwen2.5-VL-7B and InternVL3-8B were obtained by rendering skeletons with fixed visualization and finetuning on MQA for 6 epochs.

Panel of Figure 7 — Figure 7: Impact of rendered frame count. Accuracy on NTU-60 increases sharply from 4 to 10 frames and saturates at 12 frames (59.02%). Increasing to 16 frames yields minimal gains (59.45%), making 12 frames the optimal trade-off.

Resolution	$224\times 224$	$448\times 448$	$672\times 672$
Top-1 (%)	54.24	59.02	59.98

Appendix

Table 12: Impact of input resolution. Average Top-1 accuracy on NTU-60 increases with resolution, with

448\times 448

offering the best trade-off.

Panel of Figure 8 — Figure 8: Confusion matrix comparison on the NTU-60 (48/12 split). Left: InternVL3 baseline. Right: SkeletonLLM (Ours). Our method significantly reduces confusion between visually similar actions (highlighted in yellow) and improves accuracy on fine-grained classes (red boxes), demonstrating the effectiveness of Discriminative Finetuning.

Panel of Figure 9 — Figure 9: Comparison of reasoning processes on the NTU-60 (48/12 split). We visualize the chain-of-thought generated by SkeletonLLM w/o Disc-FT & CR-Distill (top) and the full SkeletonLLM (bottom) for a “Put on shoe” sequence. While the ablated variant hallucinates a “Pick up” action based on a coarse bending posture, the full SkeletonLLM accurately identifies fine-grained hand-foot interactions (red text) and employs causal logic (purple text) to rule out distractors.

Configuration	Per-frame (ms)	12 frames (ms)	Memory (GB)
$448\times 448$ , FP32	1.7	20.4	0.5
$224\times 224$ , FP32	1.6	19.3	0.5

Appendix

Table 13: DrAction rendering efficiency at different resolutions.

Stage	Latency (ms)	Proportion
DrAction rendering (12 frames)	20.4	10.6%
Vision Encoder	46.2	24.1%
LLM generation (avg. 16 tokens)	125.0	65.3%
End-to-end total	191.6	100%

Appendix

Table 14: End-to-end inference latency breakdown.

Component	Memory (GB)	Note
InternVL3-8B (BF16)	13.5	Model weights
LoRA Adapters	0.4	Finetuning params
DrAction (Gaussians + NFM)	0.5	Renderer params
Intermediate activations (12 frames)	1.1	KV Cache + activations
Peak inference total	15.5	—

Appendix

Table 15: GPU memory breakdown during inference.

Panel of Figure 10 — Figure 10: t-SNE visualization of feature representations on NTU-60 (48/12 split). The visualization is derived from the 48/12 open-vocabulary split (Xsub benchmark). Points in faded, lighter colors represent the 48 seen classes, while the darker, highlighted clusters correspond to the 12 unseen classes. Even though the unseen actions were never part of the training set, their samples spontaneously organize into compact, well-separated clusters.

Dataset	Joints	Acquisition	Topology
NTU-60/NTU-120	25	Kinect v2	Kinect skeleton
NTU-60 (2D)	17	2D Pose Est.	COCO keypoints
NW-UCLA	20	Kinect v1	Kinect skeleton
HumanML3D	22	MoCap	SMPL model
PKU-MMD	25	Kinect v2	Kinect skeleton

Appendix

Table 16: Comparison of skeleton formats used in cross-format experiments.

Figure 11: NFM color modulation transfers across skeleton formats. We use the same NTU-60-trained DrAction renderer to process examples from three heterogeneous skeleton formats: NTU-60/Kinect v2 (25 joints), HumanML3D/SMPL (22 joints), and NW-UCLA/Kinect v1 (20 joints). Each pair shows the rendered skeleton and the corresponding heatmap of the NFM-induced RGB modification magnitude, where warmer colors indicate stronger appearance updates. Although the joint sets and graph topologies differ, NFM consistently emphasizes semantically corresponding motion-functional regions such as the arms, head-neck area, pelvis, and feet. This suggests that NFM learns local body-part saliency patterns rather than memorizing format-specific joint indices, supporting DrAction’s cross-format generalization.

Method	Temporal	Causal	Fine-grained	Contrastive	ROUGE-L	BertScore
MotionGPT	31	26	33	40	22	34
MotionLLM	37	33	39	43	27	39
SKI-LVLM	29	25	32	38	21	35
InternVL3-8B	53	48	55	59	39	49
Ours (w/o CR)	55	52	59	62	43	51
Ours (Full)	68	65	73	76	52	57

Appendix

Table 17: Motion QA performance on the Skeleton-QA benchmark. Temporal, Causal, Fine-grained, and Contrastive columns report MQA accuracy (%).

Type	Example Templates
MQA	“Analyze the action sequence. Select the best match from options below. Describe your reasoning, then output the final result. A. [action1] B. [action2] …”
Temporal	“What is the first/last step of this action?”
	“In what order do the body parts move?”
Causal	“Why might this person be performing this action?”
	“What could be the goal of this movement?”
Fine-grained	“Which body parts are primarily involved?”

Appendix

Table 18: Example question templates for each reasoning type in Skeleton-QA. Templates shown are simplified illustrations; actual prompts include detailed instructions.

Panel of Figure 12 — Figure 12: Qualitative comparison of reasoning processes. Given a “headache” sequence rendered by DrAction, we compare outputs from different models on a multiple-choice QA task. MotionGPT captures only coarse patterns and selects “neck pain.” InternVL3 (fixed render) and SkeletonLLM w/o CR-Distill both misinterpret the gesture as a “salute,” focusing on superficial posture cues (“upright and stable,” “hand at brow level”) while missing subtle pressing and head-drooping dynamics. Only the full SkeletonLLM correctly identifies “headache” by recognizing the pressing posture at the temples and head drooping, while systematically ruling out alternatives with causal reasoning.

Figure 13: Prompt templates for MQA, Disc-FT, and CR-Distill. (Top-left) Prompt template for the MQA task. (Bottom-left) Prompt template for Disc-FT. (Top-right) Teacher prompt for CR-Distill. (Bottom-right) Student prompt for CR-Distill.

Figure 14: Top-5 MLLM-mined semantically similar actions for each class on NTU-60, used as hard-negative candidates in Disc-FT. Actions sharing similar body-part involvement form natural clusters (e.g., “clapping,” “rub two hands,” “pray with hands together,” “high-five”); confusing pairs often differ only in subtle temporal or spatial cues (e.g., “put on shoe” vs. “take off shoe,” “sit down” vs. “stand up”).

Figure 15: Top-5 MLLM-mined semantically similar actions for each class on NTU-120, used as hard-negative candidates in Disc-FT.

Figure 16: Examples of teacher-generated causal reasoning chains used in CR-Distill.

Explore similar work

Towards Universal Skeleton-Based Action Recognition
Apr 18, 2026Jidong Kuang, Hongsong Wang, Jie GuiSkeleton-Based Action Recognition Human Activity Recognition
School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China
LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model
Mar 28, 2026Ruosi Wang, Fangwei Zuo, Lei Li +1Skeleton-Based Action Recognition Cross-Modal Alignment
Northwestern Polytechnical University, No.1 Dongxiang Road, Chang’an District , Xi’an, 710129, Shaanxi, China · Shandong University of Finance and Economics, No.40 Shungeng Road, Licheng District, Jinan, Jinan, Shandong, China
T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition
Jun 19, 2026Di Yang, Mahmoud Ali, Quan Kong +2Skeleton-Based Action Recognition Representation Learning
Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China · Inria Center at Universit´e Cˆote d’Azur, Valbonne, France · Woven by Toyota, Tokyo, Japan +1

Apr 18, 2026J/K move · Enter open · S save

Towards Universal Skeleton-Based Action Recognition

Jidong Kuang, Hongsong Wang, Jie Gui

School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China

Skeleton-Based Action Recognition Human Activity Recognition

Open paper AskPDF arXiv

With the development of robotics, skeleton-based action recognition has become increasingly important, as human-robot interaction requires understanding the actions of humans and humanoid robots. Due to different sources of human skeletons and structures of humanoid robots, skeleton data naturally exhibit heterogeneity. However, previous works overlook the data heterogeneity of skeletons and solely construct models using homogeneous skeletons. Moreover, open-vocabulary action recognition is also essential for real-world applications. To this end, this work studies the challenging problem of heterogeneous skeleton-based action recognition with open vocabularies. We construct a large-scale Heterogeneous Open-Vocabulary (HOV) Skeleton dataset by integrating and refining multiple representative large-scale skeleton-based action datasets. To address universal skeleton-based action recognition, we propose a Transformer-based model that comprises three key components: unified skeleton representation, motion encoder for skeletons, and multi-grained motion-text alignment. The motion encoder feeds multi-modal skeleton embeddings into a two-stream Transformer-based encoder to learn spatio-temporal action representations, which are then mapped to a semantic space to align with text embeddings. Multi-grained motion-text alignment incorporates contrastive learning at three levels: global instance alignment, stream-specific alignment, and fine-grained alignment. Extensive experiments on popular benchmarks with heterogeneous skeleton data demonstrate both the effectiveness and the generalization ability of the proposed method. Code is available at https://github.com/jidongkuang/Universal-Skeleton.

Abstract

Explore similar work

Towards Universal Skeleton-Based Action Recognition

LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition