Exploiting Hierarchical Controller Structure in Contextual Parameter Learning for Humanoid Loco-Manipulation

Sebastian Hirt; Lukas Theiner; Jan Peters; Rolf Findeisen

cs.ROOct 3, 2026

Exploiting Hierarchical Controller Structure in Contextual Parameter Learning for Humanoid Loco-Manipulation

Authors: Sebastian Hirt, Lukas Theiner, Jan Peters, Rolf Findeisen

Organizations: Control and Cyber-Physical Systems Laboratory (CCPS), TU Darmstadt · Intelligent Autonomous Systems Laboratory, TU Darmstadt · German Research Center for AI (DFKI), Research Department SAIROL · Hessian Centre for Artificial Intelligence

Abstract

Hierarchical control architectures are widely used to decompose complex control problems into interacting control levels and are particularly important in robotics, where planning, whole-body motion, and lower-level control must be coordinated across different levels of abstraction and time scales. Their overall closed-loop performance, however, depends strongly on parameters distributed across the hierarchy, such that tuning controllers on different levels independently may neglect relevant cross-layer interactions. We propose a contextual Bayesian optimization framework for joint parameter learning in hierarchical control systems. Rather than modeling closed-loop performance only as a scalar black-box function, we retain separate observations of task performance, realization quality, and control effort. A correlated multi-output Gaussian process models these performance components, while their known aggregation into the overall closed-loop objective is evaluated analytically. The formulation exploits three complementary consequences of hierarchical control: informative performance quantities exposed by the hierarchy, coupling between parameters of different controller levels, and variations of these relations with operating conditions. We evaluate the approach for humanoid loco-manipulation, jointly tuning a centroidal predictive controller and a whole-body controller for physical box pushing under varying box mass. The proposed method achieves the lowest mean empirical regret during both training and adaptation among the considered baselines.

Figures & tables

Panel of Figure 1 — Fig. 1: Overview of the proposed framework. State-feedback paths within the controller hierarchy are omitted for clarity. Dotted guides associate task ( $J_{\mathrm{T}}$ ), realization ( $J_{\mathrm{R}}$ ), and effort ( $J_{\mathrm{E}}$ ) information with quantities exposed by the control hierarchy, while all components may depend on the full parameter vector. The hierarchy-informed surrogate provides the aggregated objective used by the joint acquisition $\alpha^{(n)}$ , while context dependence $\chi$ enables transfer across operating conditions.

Fig. 2: The simulated humanoid box-pushing task.

Group	Optimized quantities	Dim.
$\theta_{\mathrm{MPPI}}$	MPPI temperature; motion, momentum, effort,	6
	and wrench-variation weights
$\theta_{\mathrm{WBC}}$	CoM, torso, foot, posture, momentum, and	11
	wrench-tracking gains and task weights
$\theta_{\mathrm{ref}}$	Torso pitch, CoM offsets, step duration,	5
	and swing clearance

TABLE I: Optimized parameter groups.

Panel of Figure 4 — Fig. 3: Empirical regret during training (left) and adaptation at the unseen $4\,\mathrm{kg}$ mass (right). Legend labels indicate, in order, context use ( $c$ ), joint cross-layer tuning ( $j$ ), and component-wise performance observations ( $p$ ). $\mathrm{x}$ denotes an absent feature. Curves show the mean, and shaded regions indicate one standard deviation across ten independent runs.

Panel of Figure 5 — Fig. 4: Representative closed-loop trajectories at $2\,\mathrm{kg}$ . The top panel shows the identical box-progress task for both controllers under a shared external reference (dots). The middle and bottom panels show center-of-mass height and torso pitch, respectively, together with each controller’s tunable reference. Solid lines denote realized trajectories, dashed lines denote references.

Robotic Control

Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.

See all 584 papers

Oct 6, 2026

HULK: Learning Whole-Body Forceful Loco-Manipulation for Humanoids

Humanoid loco-manipulation of large, heavy objects demands forceful interaction across the entire body. However, such payloads shift a humanoid's center of mass and impose sustained loads across the upper body, challenging balance and command tracking. We present HULK, a whole-body control framework for forceful loco-manipulation. Using model predictive control (MPC) to guide reinforcement learning with predictions of the loaded dynamics, we train two teachers: one tracks arm motions under wrist forces, and the other locomotes while holding large objects against the body. A capture-point control barrier function augments the wrist-force teacher during training to improve balance under load. We distill both teachers into a single policy. Evaluation spans simulation and the Unitree G1. In simulation, the teacher with the barrier function achieves the lowest forward and lateral velocity tracking errors at 10 kg per arm among evaluated controllers and reduces aggregate divergent component of motion (DCM) excursion magnitude by 35.7% relative to MPC-guided reinforcement learning alone. Our wrist-force teacher withstands torso push disturbances of up to 130 N.

An Dang, Arturo Flores Alvarez, Yu-Ming Chen +5

Amazon · University of Michigan · University of California, Los Angeles +1

Humanoid Loco-Manipulation Whole-Body Control Force-Controlled Robotic Manipulation RL Control

Ask PDF

Sep 16, 2026

Learning Holistic Whole-Body Loco-Manipulation with a Bipedal Mobile Manipulator

Bipedal loco-manipulation enables robots to interact with objects beyond the nominal workspace of their arms by coordinating locomotion and manipulation. Realizing this capability requires a low-level whole-body controller that translates task-level manipulation goals into coordinated arm and leg motions while maintaining balance. We present a unified whole-body controller trained with reinforcement learning that directly maps 6-DoF end-effector targets to coordinated actions for the bipedal base and robotic arm. Given only an end-effector target, the learned controller autonomously coordinates reaching, postural adaptation, and stepping without explicit base-velocity or footstep commands. A reward-gating strategy regulates the trade-offs among end-effector tracking, locomotion, and balance during training, while a temporal context estimator combines windowed Transformer encoding, recurrent GRU memory, and auxiliary dynamics prediction to extract dynamics-relevant information from observation history. Real-robot experiments demonstrate that the same controller supports reaching, postural adaptation, and stepping under commands from VR teleoperation, a learned diffusion policy, and scripted trajectories, providing a common end-effector interface for diverse manipulation tasks.

Zhongyu Chen, Yuxuan Nai, Qian Chen +8

ZJU-UIUC Institute. · LimX Dynamics.

Whole-Body Robotic Manipulation Humanoid Loco-Manipulation Robot Locomotion RL for Robotics Whole-Body Control Robotic Manipulation+1

Ask PDF

Oct 26, 2026

Learning Multi-Phase Robot Loco-Manipulation via Phase-Conditioned Policy Optimization

Complex whole-body loco-manipulation requires executing diverse maneuvers, such as crouching, grasping, and locomotion, within a single long-horizon task. While performing simple pick-and-place behaviors is well-established, scaling to complex, long-horizon tasks remains difficult. Currently, this challenge is addressed by relying on human motion data and correspondingly humanoid platforms, which limits platform independence and introduces substantial data processing overhead. Instead, we extend proximal policy optimization (PPO) to train a single policy for multi-phase tasks across diverse robotic hardware, leveraging prior task structure to shape behavior. Our method adapts PPO using a phase-conditioned critic, phase-gated reward functions, and a multi-phase reset curriculum to guide exploration. To support varied hardware, we employ a decoupled hierarchical control architecture that separates lower-level balancing dynamics from high-level phase execution. We demonstrate multi-stage loco-manipulation skills on hardware on both the Unitree G1 and a Boston Dynamics Spot equipped with a manipulator arm, and validate our design choices through ablations. Our approach enables the acquisition of complex loco-manipulation behaviors purely through environment interaction while directly incorporating high-level task structure into policy learning.

Simone Manni, Annika Wollschläger, Dario Bellicoso +4

Robotic Systems Lab (RSL), ETH Zurich, Zurich, Switzerland. · Robotics and AI Institute (RAI), Zurich, Switzerland.

Robotic Loco-Manipulation Long-Horizon Robotic Manipulation Reinforcement Learning Proximal Policy Optimization Cross-Embodiment Robot Learning

Ask PDF

Abstract

Robotic Control

Explore similar work

HULK: Learning Whole-Body Forceful Loco-Manipulation for Humanoids

Learning Holistic Whole-Body Loco-Manipulation with a Bipedal Mobile Manipulator

Learning Multi-Phase Robot Loco-Manipulation via Phase-Conditioned Policy Optimization