Recent advances in reinforcement learning (RL) have enabled impressive humanoid behaviors in simulation, yet transferring these results to new robots remains challenging. In many real deployments, the primary bottleneck is no longer simulation throughput or algorithm design, but the absence of systematic infrastructure that links environment verification, training, evaluation, and deployment in a coherent loop. To address this gap, we present AGILE, an end-to-end workflow for humanoid RL that standardizes the policy-development lifecycle to mitigate common sim-to-real failure modes. AGILE comprises four stages: (1) interactive environment verification, (2) reproducible training, (3) unified evaluation, and (4) descriptor-driven deployment via robot/task configuration descriptors. For evaluation stage, AGILE supports both scenario-based tests and randomized rollouts under a shared suite of motion-quality diagnostics, enabling automated regression testing and principled robustness assessment. AGILE also incorporates a set of training stabilizations and algorithmic enhancements in training stage to improve optimization stability and sim-to-real transfer. With this pipeline in place, we validate AGILE across five representative humanoid skills spanning locomotion, recovery, motion imitation, and loco-manipulation on two hardware platforms (Unitree G1 and Booster T1), achieving consistent sim-to-real transfer. Overall, AGILE shows that a standardized, end-to-end workflow can substantially improve the reliability and reproducibility of humanoid RL development.
Figures & tables
Figure 1: Overview of agile learning workflow. The workflow covers prepare-training, batch cloud training with reproducible logging and deployment-oriented features, evaluation using quantitative motion metrics and automated HTML reports, and deployment by exporting the learned policy and I/O descriptors for Sim2Sim and Sim2Real transfer. Example applications include locomotion, loco-manipulation, DeepMimic-style imitation, and synthetic data generation (SDG) for VLA training, supported by an algorithmic enhancements library (e.g., curricula, regularization, adaptive sampling, reward normalization, symmetry augmentation, and distillation).
Feature
agile
Holosoma
H.Verse
Proto.
Env. debugging
✓
–
–
–
Algo. enhancements
✓
∼
–
∼
Determ. evaluation
✓
∼
–
✓
Sim-to-sim pipeline
✓
✓
✓
✓
Descriptor-driven export
✓
–
–
–
Sim-to-real validation
✓
✓
✓
✓
Table 1: Feature comparison of humanoid RL frameworks. ✓= supported, ∼ = partial, – = not supported.
Figure 2: Deterministic Evaluation Pipeline. A unified framework for assessing humanoid policies across Isaac Lab (GPU) and MuJoCo (CPU) backends.
Figure 3: Decoupled whole-body control: upper- and lower-body policies are trained separately. This design allows greater flexibility to meet different application requirements. For example, an IK policy can be used for high-accuracy tasks while a VLA upper-body policy enables autonomous execution with language input.
Task
Robot
Actions
Observation (pol/crit)
Reward dim
Episode
Terrain
Training
Locomotion
G1
Leg pos (12)
noisy / + scans
15
30 s
rough
10h
Locomotion
T1
Leg pos (10)
noisy / + vel,ht
20
30 s
rough
10h
Loco + Height
G1
Leg pos (12)
noisy / + scans
22
30 s
rough
10h
Stand-up
G1
All joints (rel)
noisy / + forces
18
15 s
stand-up
25h
Stand-up
T1
All joints (rel)
noisy / + forces
18
15 s
stand-up
15h
Motion Imitation
G1
All joints (rel)
noisy / + vel
9
8 s
flat
6h
Table 2: MDP overview and training time (on a single L40) per task.
Figure 4: Velocity and height command tracking in MuJoCo. Each subplot shows a sweep of commanded values (dashed) versus actual robot response (solid) for forward velocity, lateral velocity, yaw rate, and standing height. The solid line is smoothed with the actual showing as the lighter shadow for easier visualization.
Teacher (Privileged)
Student (Distilled)
Det.
Random
Deterministic
Duration
50s
50s
200s
500s
RNN (50s)
Hist. (50s)
vx (m/s)
0.070
0.142± 0.019
0.136± 0.010
0.136± 0.003
0.116
0.097
vy (m/s)
0.083
0.118± 0.013
0.110± 0.006
0.110± 0.002
0.110
0.087
ωz (rad/s)
0.074
0.116± 0.013
0.113± 0.007
0.112± 0.003
0.117
0.079
h (m)
0.035
0.046± 0.007
0.039± 0.003
0.036± 0.001
0.037
0.037
Table 3: Sim-to-Sim evaluation of tracking on Unitree G1 velocity+height control. Left: deterministic sweep (50 s) vs. random commands at increasing durations (2 s resample interval) over 10 runs. Right: teacher vs. distilled student policies (deterministic sweep, 50 s).
Figure 5: GR00T post-training pipeline. (1) RL expert trained with physics randomization. (2) Demonstration collection via tiled parallel rendering with scene randomization. (3) VLA fine-tuning on the collected dataset. (4) Closed-loop evaluation in simulation.
Figure 6: Effect of L2C2 on motion smoothness under increasing observation noise. Subplots: RMS joint acceleration, RMS joint jerk, position limit violations, and high-frequency energy ratio (fraction of joint-trajectory energy above 10 Hz).
Figure 7: Ablation studies (5 seeds, shaded = ±1 std unless noted). (a) Reward normalization; dashed curves scale all rewards by 100 × before training and rescale for comparability. (b) Value-bootstrapped vs. tuned termination penalty (shaded = min/max). (c) Virtual harness on height-controlled locomotion. (d) Symmetry augmentation.
Figure 8: Sim-to-real transfer across five tasks on two robot platforms. (a) Velocity and height controlled locomotion on Unitree G1. (b) Velocity controlled locomotion on Booster T1. (c) Stand-up on Unitree G1. (d) Stand-up on Booster T1. (e) Motion imitation (dancing) on Unitree G1.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Actor network
[256,256,128] (pick&place: [256,128,64] )
Critic network
[512,256,128] (pick&place: [256,128,64] )
Activation
ELU
Learning rate
10−3
Discount γ
0.99 (stand-up: 0.995 )
GAE λ
0.95
Appendix
Table 4: PPO hyperparameters. Task-specific overrides in parentheses.
Learning generalizable humanoid loco-manipulation remains bottlenecked by costly real-world demonstrations. Simulation provides a scalable alternative, but requires visually faithful, physics-ready scenes and executable whole-body motions. We present Zero-HLM, a real-to-sim-to-real framework that transforms smartphone captures and simulated source demonstrations into scalable training data for humanoid loco-manipulation. Gaussian-guided agentic real-to-sim reconstruction aligns visual and physical scenes; motion-environment co-adaptation uses rollout feedback to correct whole-body references and interaction geometry. We generate 4,800 physically validated demonstrations for zero-shot transfer across three Unitree G1 tasks. Simulation-trained GR00T N1.7 achieves 66.7% mean full-task success, versus 46.7% with real-data training. On generalized configurations, expanded simulation training raises pooled success from 23% for base-trained real-data policies to 69%, without additional real demonstrations. Project page: https://young-bit.github.io/zero-hlm/
Dianyi Yang, Mengya Liu, Zhumei Wang +9
Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Tsinghua University +1
Reinforcement learning (RL) has become a powerful paradigm for robot learning, particularly in sim-to-real settings, but its broader adoption remains limited by the engineering pipeline surrounding the algorithms. Building tasks, shaping rewards, and tuning hyperparameters require substantial expert effort, making RL workflows costly and difficult to scale. We introduce HARBOR, an agentic framework that frames robot RL automation as a harness-engineering problem: given a simulator codebase and a task specification, it automates the workflow from environment setup to policy training in simulation. HARBOR decomposes such high-level objectives into bounded stages executed by specialized agents through standardized commands, persistent artifacts, executable gates, and reusable knowledge, and scales iteration via decentralized parallel trials and experience learning across runs. We evaluate HARBOR across 6 benchmarks and 16 tasks in total, spanning manipulation, locomotion, and bimanual dexterous control. We demonstrate that HARBOR automates the simulation RL workflow end-to-end, designs rewards, tunes algorithms to match or improve over default configurations, and reduces engineering effort at practical token and wall-clock cost; the resulting policies can also be transferred to real robots.
Zechu Li, Yufeng Jin, Xiaoyang Liu +4
TU Darmstadt 1 · Hessian.AI 7 · Honda Research Institute Europe 2 +4
Humanoid foundation models are advancing faster than we can evaluate them. While real-world testing is expensive and difficult to reproduce, existing simulation benchmarks focus primarily on table-top or wheeled robots. A scalable and reproducible benchmark for whole-body humanoid loco-manipulation remains an open problem. To this end, we present SIMPLE, a unified simulation testbed for humanoid policy learning and evaluation. SIMPLE couples the accurate contact-rich dynamics of MuJoCo with the photorealistic rendering of IsaacSim. It provides a large-scale environment comprising 16 diverse whole-body tasks, 50 indoor scenes, and over 1,000 object assets. To facilitate scalable data collection, the framework integrates two data generation pipelines: automated trajectory generation via motion planning and a low-latency VR teleoperation interface. We further integrate and benchmark mainstream humanoid policies at scale in SIMPLE, including lightweight imitation networks, large vision-language-action (VLA) models, and recent world action models (WAMs). Our experiments reveal a strong correlation between policy performance in simulation and the real world. Furthermore, we demonstrate that policies trained on data collected in SIMPLE can be transferred zero-shot to physical humanoid robots under similar settings, providing a robust and reproducible foundation for humanoid robotics research.