SP-HLM: Sparse-Pose-Driven Motion Generation and Whole-Body Control for Humanoid Loco-Manipulation

Zhaohui Yang; Quanbao Lin; Chengqi Shi; Zhenglang Weng; Jicheng Lin; Boyu Chen; Tengjie Zhu; Guanzhu Ren

cs.ROOct 21, 2026

SP-HLM: Sparse-Pose-Driven Motion Generation and Whole-Body Control for Humanoid Loco-Manipulation

Authors: Zhaohui Yang, Quanbao Lin, Chengqi Shi, Zhenglang Weng, Jicheng Lin, Boyu Chen, Tengjie Zhu, Guanzhu Ren,

Organizations: Shanghai Jiao Tong University · The University of Hong Kong · Moore Threads

Abstract

Humanoid loco-manipulation requires an effective interface between high-level task policies and whole-body control. Existing interfaces typically rely on a single action representation for both task intent and whole-body coordination. This may burden the high-level policy or leave one aspect underspecified. We present SP-HLM, a framework built on a sparse-pose-driven interface. The interface combines five task-critical body poses with a learned latent motion prior to represent spatial intent and whole-body coordination in complementary forms. SP-VLA, a vision-language-action model, first predicts task-relevant sparse poses and finger joint positions from multimodal observations and language instructions using asynchronous action-chunk inference. Given these sparse targets, SP-MGM, a motion generation model trained through two-stage distillation, reconstructs coordinated full-body motion in real time. A pretrained motion encoder maps this motion to a compact latent representation. SP-GMT, a general motion tracker, then integrates the explicit sparse targets with this implicit motion prior for whole-body control. Experiments in simulation and on a real humanoid robot demonstrate improved motion tracking, whole-body coordination, and loco-manipulation success over existing approaches. Additional results and videos are available on our blue{project page}.

Figures & tables

Fig. 2: Overview of the proposed SP-HLM framework based on the sparse-pose-driven interface. Top: the deployment pipeline integrating sparse-pose prediction, full-body motion generation, and explicit–implicit whole-body control. Bottom: the training procedures for SP-VLA, SP-MGM, and SP-GMT.

Panel of Figure 2 — Fig. 3: Qualitative rollouts of SP-HLM on eight simulated loco-manipulation tasks. Each sequence shows three temporally ordered keyframes illustrating task progression, with the corresponding language instruction shown above.

Filtered CLOT-500
Method	Progress $\uparrow$	Local MPJPE $\downarrow$	Global MPJPE $\downarrow$	Joint error $\downarrow$	EE global error $\downarrow$	EE local error $\downarrow$
	(%)	(m)	(m)	(rad)	(m)	(m)
SP-GMT w/ Joint-Space Reference	$99.35\pm 0.14$	$0.0305\pm 0.0000$	${0.0812}\pm 0.0004$	$0.0975\pm 0.0001$	${0.0879}\pm 0.0004$	$0.0571\pm 0.0001$
SP-GMT w/o Sparse Reference	$99.28\pm 0.19$	$0.0390\pm 0.0000$	$0.4436\pm 0.0036$	$0.1538\pm 0.0000$	$0.4506\pm 0.0036$	$0.0733\pm 0.0000$
SP-GMT w/o Implicit Reference	$98.93\pm 0.24$	$0.0659\pm 0.0000$	$0.1303\pm 0.0003$	$0.2407\pm 0.0001$	$0.1470\pm 0.0003$	$0.1239\pm 0.0001$
SP-GMT w/o Robot Pelvis Pose	$99.33\pm 0.27$	$0.0356\pm 0.0000$	$\underline{0.0753}\pm 0.0001$	$0.1330\pm 0.0001$	$\underline{0.0836}\pm 0.0002$	$0.0663\pm 0.0001$

TABLE II: Motion tracking under trajectory observation noise over ten seeded runs. All methods use the same evaluation configuration. Values report the population mean

\pm

standard deviation. Bold and underlined means indicate the best and second-best results, respectively.

Model	Network Depth	EE Error Mean	EE Error P95	EE Jerk P95	Body Jerk P95	Generation Rate
Model	$\downarrow$ (layers)	$\downarrow$ (cm)	$\downarrow$ (cm)	$\downarrow$ ( $\mathrm{m/s^{3}}$ )	$\downarrow$ ( $\mathrm{m/s^{3}}$ )	$\uparrow$ (Hz)
Kimodo (20 steps)	16	$1.941\pm 1.398$	$3.521\pm 2.832$	$\mathbf{450.768}\pm 224.996$	$\mathbf{370.225}\pm 178.458$	$5.405$
SP-MGM (w/o Stage II)	8	$8.441\pm 1.680$	$12.772\pm 3.080$	$1068.626\pm 506.355$	$916.724\pm 476.011$	$7.634$
SP-MGM (w/o $L_{\mathrm{temp}}$ )	8	$\underline{1.915}\pm 0.730$	$\underline{3.420}\pm 1.445$	$657.125\pm 311.392$	$570.246\pm 287.323$	$7.322$
SP-MGM (Ours)	8	$\mathbf{1.812}\pm 0.709$	$\mathbf{3.212}\pm 1.289$	$\underline{514.337}\pm 236.943$	$\underline{416.945}\pm 200.507$	$\underline{7.813}/\mathbf{11.494}$

TABLE III: Quantitative comparison of motion generation quality and efficiency. P95 denotes the 95th percentile of the corresponding error or jerk distribution. Reporting and highlighting conventions follow Table II .

Panel of Figure 5 — Fig. 4: Representative rollouts of SP-HLM on four real-world humanoid loco-manipulation tasks.

Real-world Task	T1: OP	T2: ST	T3: CO	T4: TO
Success Rate	13/20	11/20	5/20	8/20

TABLE IV: Real-world success rates of SP-HLM.

Reward term	Weight	$\sigma$
Global pelvis position	$0.5$	$0.3$ m
Global pelvis orientation	$0.5$	$0.4$ rad
Relative body position	$1.0$	$0.3$ m
Relative body orientation	$1.0$	$0.4$ rad
Global body linear velocity	$1.0$	$1.0$ m/s
Global body angular velocity	$1.0$	$3.14$ rad/s

TABLE V: Reward terms and default weights.

Humanoid Loco-Manipulation

Humanoid loco-manipulation studies how robots coordinate walking, balance, and whole-body interaction to move and handle objects.

See all 93 papers

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation
Jun 20, 2026Yingdong Hu, Haodong Zhu, Boyuan Zheng +6Humanoid Loco-Manipulation Whole-Body Robotic Manipulation
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI
KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation
Sep 16, 2026Sitong Chen, Fatemeh Zargarbashi, Jin Cheng +2Robotic Control RL for Robotics
ETH Zürich, Switzerland
Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining
Sep 30, 2026Chongyang Xu, Zhao Wu, Jin Chen +9Humanoid Loco-Manipulation Cross-Embodiment Robot Learning
Alibaba Group · Sichuan University · Shanghai Innovation Institute +2

Jun 20, 2026J/K move · Enter open · S save

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation

Yingdong Hu, Haodong Zhu, Boyuan Zheng, Yihang Hu+5

Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI

Humanoid Loco-Manipulation Whole-Body Robotic Manipulation Vision-Language-Action Models Long-Horizon Robotic Manipulation Robot Teleoperation Cross-Embodiment Robot Learning+2

Open paper AskPDF arXiv

Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and

Ψ_0

) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].

Abstract

Humanoid Loco-Manipulation

Explore similar work

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation

KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation

Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining