cs.ROOct 27, 2026

METATRON: A Promptable Whole-Body Execution Layer for Humanoids

Authors: Chen Tessler, Yifeng Jiang, Alec Otto, Yi Shi, Umar Iqbal, Gal Chechik, Xue Bin Peng

Organizations: NVIDIA. · Dollhouse Robotics.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Humanoid controllers often require task-specific training despite sharing the need for balance and whole-body coordination. We present Metatron, a promptable whole-body execution layer combining a generative base policy with a programmable skill layer. The base policy receives partial goals, collectively called a prompt, that can combine timed body-part targets, velocity commands, and style cues. It produces diverse whole-body motions consistent with these goals. Training combines motion capture, synthetic recovery trajectories, and retargeted scene-interaction motions. After training, the policy is frozen. Stateful programs called skills express task intent as prompts at each control step, drawing rough geometric sketches that the policy turns into coordinated movement. Skills are ordinary code, refined by coding agents that analyze motion data and fit compact rules without task-specific policy training. On the physical Unitree G1, one frozen checkpoint supports walking, recovery, teleoperated object pickup, and sitting on furniture. Comparative evaluations show higher reaching success and push robustness than the tested reference-tracking pipelines. The same recipe produces climbing and acrobatics on a simulated character. Metatron enables coding agents to turn learned motor capabilities into new skills through code, without further policy training.

Figures & tables

Fig. 1 : One frozen controller per embodiment, diverse behaviors. Partial prompts specify goals; each controller provides whole-body coordination across tasks without task-specific training. Teal and gray blocks indicate specified and unspecified goal components, respectively.
Fig. 2 : Model and training. The prior maps partial prompts and observation history to a deterministic condition and a latent distribution. The decoder combines the condition with a sphere-projected latent to produce joint targets. Training adds a posterior conditioned on the full reference and supervision from a frozen tracker. Only the solid path runs at deployment. Pose renders illustrate state, not camera inputs.
Fig. 3 : Agent-assisted skill development. Given a task specification and optional motion examples, a human and LLM agent define evaluation scenarios and criteria. The LLM agent then iteratively refines the skill code using simulation feedback. Skills specify goals for the frozen Metatron policy, enabling the model to perform new tasks without additional training.
Fig. 4 : Change the prompt, retain the controller. In this simulation skill, a detected fall switches steering goals to a whole-body upright pose; steering resumes once upright. G1 deployment instead requests torso height alone.
Fig. 5 : Multiple control modes through one G1 policy. Each mode specifies different goals for the same frozen controller.
Fig. 6 : SOMA: On a simulated SOMA character, with the same recipe, Metatron runs, slides off a ledge, recovers from the fall, then runs and climbs up a wall.
Fig. 7 : Recovery: Metatron recovers from a range of initial configurations.
MethodReached (%)Error (cm)Upright (%)
Metatron835.9100
ARDY + SONIC396.482
GRU + IK + SONIC285.681
GRU + SONIC0—87
TABLE I : Whole-body wrist reaching on a 54-target grid. Metatron is trained without synthetic recovery data for this comparison. A target is reached if the wrist is held within 10 cm of it while the robot remains upright. Error is measured only on reached targets .
 Perturbation force (N)
Method2004006008001000
Metatron100100627575
ARDY+SONIC1001001700
GRU+SONIC100752500
TABLE II : Push robustness during steering (100 ms force pulse at torso, 8 directions). Success: upright after 8 s. Metatron trained without recovery data. Results: success (%).
 Perturbation force (N)
Method0200400600800
Metatron97.4−1.8+1.1\boldsymbol{97.4^{+1.1}_{-1.8}}94.8−2.3+1.6\boldsymbol{94.8^{+1.6}_{-2.3}}33.0−4.0+4.2\boldsymbol{33.0^{+4.2}_{-4.0}}7.4−2.0+2.6\boldsymbol{7.4^{+2.6}_{-2.0}}6.2−1.8+2.5\boldsymbol{6.2^{+2.5}_{-1.8}}
Global only87.6−3.2+2.687.6^{+2.6}_{-3.2}87.8−3.2+2.687.8^{+2.6}_{-3.2}38.2−4.2+4.3\boldsymbol{38.2^{+4.3}_{-4.2}}9.6−2.3+2.9\boldsymbol{9.6^{+2.9}_{-2.3}}6.2−1.8+2.5\boldsymbol{6.2^{+2.5}_{-1.8}}
Mocap only92.0−2.7+2.192.0^{+2.1}_{-2.7}84.8−3.4+2.984.8^{+2.9}_{-3.4}31.8−3.9+4.2\boldsymbol{31.8^{+4.2}_{-3.9}}6.4−1.8+2.5\boldsymbol{6.4^{+2.5}_{-1.8}}5.0−1.6+2.3\boldsymbol{5.0^{+2.3}_{-1.6}}
No aux.81.6−3.6+3.281.6^{+3.2}_{-3.6}78.8−3.8+3.478.8^{+3.4}_{-3.8}29.2−3.8+4.129.2^{+4.1}_{-3.8}5.4−1.7+2.3\boldsymbol{5.4^{+2.3}_{-1.7}}4.6−1.5+2.2\boldsymbol{4.6^{+2.2}_{-1.5}}
MaskedMimic +98.2−1.6+0.9\boldsymbol{98.2^{+0.9}_{-1.6}}90.0−2.9+2.390.0^{+2.3}_{-2.9}27.8−3.7+4.127.8^{+4.1}_{-3.7}4.2−1.4+2.14.2^{+2.1}_{-1.4}4.6−1.5+2.2\boldsymbol{4.6^{+2.2}_{-1.5}}
TABLE III : Push robustness during XY waypoint reaching. Horizontal torso pushes last 100 ms, recurring every 1–2 s during the first 5 s until arrival. Success: reaching within 0.5 m of the goal within 10 s, with relative torso height ≥0.45\geq 0.45 m and tilt ≤60∘\leq 60^{\circ} at arrival. Entries report success (%) with upper/lower 95% Wilson CI deviations in percentage points over 500 episodes. Bold marks CI overlap with the highest rate at each force (unrounded bounds).
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

SettingG1 BONESSOMA
Environments per GPU81924096
Minibatch size per GPU163848192
Optimization passes per rollout21
Gradient-norm limit150
Entropy coefficient0.010.005
Reward normalizationNoYes
Appendix
TABLE IV: Expert PPO settings, distinct from student distillation.
Termwwσ\sigma
Torso, hands, feet: global position10.3
Torso, hands, feet: global rotation10.4
Torso and hands: global position20.1
Torso and hands: global rotation0.40.1
Anchor-relative body position10.3
Anchor-relative body rotation10.4
Appendix
TABLE V: G1 BONES expert rewards. Tracking terms are wexp⁡(−e/σ2)w\exp(-e/\sigma^{2}) . Position and velocity errors sum squared coordinate errors, then average over the selected bodies; rotation error is the mean squared relative rotation angle.
SkillInputPrompt fieldsUpdate
Dense trackingreference cliptimed body positions and orientationsreference clock
Steeringjoystick velocity and turnplanar velocity; five timed torso, heading, and alternating foot targetsevery step
Path followingwaypoint listtimed torso waypoints and planar velocityreceding horizon
Locationtarget position or orientationtimed body constraint; planar position, height, heading, or full anchor orientationon target change
Styleexample clipunordered motion-state cueon reset or override
Sparse goalpose from a trajectoryone timed future poseon target transition
Appendix
TABLE VI : Harness skills and the prompts they construct. Each embodiment uses one frozen policy across its supported skills. The recovery row describes the simulation state machine; hardware get-up demonstrations are reported in section VI-B .
More in this topic

Humanoid Robot Control

Humanoid robot control develops methods for coordinating bipedal balance, locomotion, and whole-body interaction with people and objects.

See all 206 papers

Explore similar work

CardsList
Oct 3, 2026

Continual Humanoid Motion Learning

Humanoid whole-body controllers can now track a diverse set of dynamic motions, but they are typically trained offline and then frozen, so teaching such a controller a new skill tends to erode the skills it already mastered. We study continual learning for humanoid whole-body motion, where a single controller must acquire skills from a sequential task stream without revisiting past data. We introduce Similarity-guided LoRA-PNN, a progressive neural network (PNN) policy that prevents catastrophic forgetting by construction while reusing knowledge across skills through lightweight low-rank adaptation. A two-level motion-similarity measure, built from dynamic time warping aggregated by optimal transport, decides which prior skill to build on and how much new capacity to allocate, yielding strong forward transfer and large efficiency gains. Across six sequentially learned skill categories, our similarity-guided LoRA policy attains the best forward transfer (0.125 vs. 0.079) and the highest average accuracy among all methods, while saving up to 94.5% of trainable parameters and 40.8% of training time. The resulting controller reaches 96.13% sim-to-sim transfer and is deployed on a physical Unitree G1. Our code is available at https://anonymous.4open.science/r/continual-humanoid-learning-35D3.

Zhewen He, Hao Huang, Geeta Chandra Raju Bethala +4

New York University Abu Dhabi, Abu Dhabi, United Arab Emirates. · Fudan University, Shanghai, China.

Continual Robot LearningHumanoid Motion TrackingLow-Rank Adaptation
AskPDF
Sep 11, 2026

CHOREO: Every Humanoid Skill as a Trajectory

Recent advances in humanoid robotics have produced diverse skills through reinforcement learning, motion imitation, and generative modeling. Yet these capabilities remain siloed because they are built around incompatible representations, interfaces, and controllers. We present CHOREO, a framework for training-free composition of heterogeneous humanoid skills. Our key observation is that, regardless of how a skill is learned, it can ultimately be expressed as an executable motion trajectory. Based on this observation, CHOREO converts each capability into SkillMotion, a unified representation that combines motion states, contacts, semantics, and boundary conditions. Skills are composed through direct continuation, cubic Hermite blending, or validated bridge motions, without retraining source models or updating models at test time. On Unitree G1 in MuJoCo, CHOREO organizes 2,950 admitted SkillMotion assets derived from heterogeneous sources and achieves 95.4% sequence success across 130 multi-action tasks, including 93.8% success on eight-action sequences. These results demonstrate that executable trajectories provide a scalable interface for accumulating and composing pretrained humanoid capabilities.

Ziyi Sun, Jingwen Chen, Yuxi Wang +4

College of Engineering, Ocean University of China · Faculty of Information Science and Engineering, Ocean University of China · School of Artificial Intelligence, University of Chinese Academy of Sciences +1

Humanoid RoboticsHumanoid Robot ControlRobot Motion GenerationRobot Skill Learning
AskPDF
Sep 21, 2026

MimicAgent: Quadruped Skills via Prompt-to-Trajectory Generation

We present MimicAgent, a prompt-to-trajectory generation framework for learning dynamic quadruped skills. Although reward shaping is extensively used when training quadruped policies, navigating the resulting reward landscape is notoriously difficult, requiring hours of "graduate student descent". Eureka attempts to automate reward design with LLMs, but we find that it struggles to generalize across diverse skills and morphologies. Our key observation is that it is far easier for a human - and by association, an LLM - to generate reference motions than to shape reward functions. Our hypothesis is motivated by the success of example-guided RL for humanoids, which exploits large-scale motion capture datasets as references for training locomotion policies. Unlike humanoids, quadrupeds lack such reference motion data. Towards this end, we propose MimicAgent, an agentic harness that, given a skill prompt, generates quadruped reference trajectories with coding agents. These coarse reference trajectories are then used to train example-guided RL policies that are deployable in simulation and in the real-world. Notably, we find that when prompting Claude Fable 5.1 within our agentic harness, 87% of prompts yield semantically aligned reference trajectories.

Lucky Kant Nayak, Narayanan Palghat Parameswaran, Neehar Peri +1

Carnegie Mellon University

Reinforcement LearningTrajectory GenerationLLM Agent Trajectory SynthesisRL for Legged LocomotionPolicy LearningQuadruped Locomotion
AskPDF