Recent advances in reinforcement learning (RL) have enabled impressive humanoid behaviors in simulation, yet transferring these results to new robots remains challenging. In many real deployments, the primary bottleneck is no longer simulation throughput or algorithm design, but the absence of systematic infrastructure that links environment verification, training, evaluation, and deployment in a coherent loop. To address this gap, we present AGILE, an end-to-end workflow for humanoid RL that standardizes the policy-development lifecycle to mitigate common sim-to-real failure modes. AGILE comprises four stages: (1) interactive environment verification, (2) reproducible training, (3) unified evaluation, and (4) descriptor-driven deployment via robot/task configuration descriptors. For evaluation stage, AGILE supports both scenario-based tests and randomized rollouts under a shared suite of motion-quality diagnostics, enabling automated regression testing and principled robustness assessment. AGILE also incorporates a set of training stabilizations and algorithmic enhancements in training stage to improve optimization stability and sim-to-real transfer. With this pipeline in place, we validate AGILE across five representative humanoid skills spanning locomotion, recovery, motion imitation, and loco-manipulation on two hardware platforms (Unitree G1 and Booster T1), achieving consistent sim-to-real transfer. Overall, AGILE shows that a standardized, end-to-end workflow can substantially improve the reliability and reproducibility of humanoid RL development.
Figures & tables
Figure 1: Overview of agile learning workflow. The workflow covers prepare-training, batch cloud training with reproducible logging and deployment-oriented features, evaluation using quantitative motion metrics and automated HTML reports, and deployment by exporting the learned policy and I/O descriptors for Sim2Sim and Sim2Real transfer. Example applications include locomotion, loco-manipulation, DeepMimic-style imitation, and synthetic data generation (SDG) for VLA training, supported by an algorithmic enhancements library (e.g., curricula, regularization, adaptive sampling, reward normalization, symmetry augmentation, and distillation).
Feature
agile
Holosoma
H.Verse
Proto.
Env. debugging
✓
–
–
–
Algo. enhancements
✓
∼
–
∼
Determ. evaluation
✓
∼
–
✓
Sim-to-sim pipeline
✓
✓
✓
✓
Descriptor-driven export
✓
–
–
–
Sim-to-real validation
✓
✓
✓
✓
Table 1: Feature comparison of humanoid RL frameworks. ✓= supported, ∼ = partial, – = not supported.
Figure 2: Deterministic Evaluation Pipeline. A unified framework for assessing humanoid policies across Isaac Lab (GPU) and MuJoCo (CPU) backends.
Figure 3: Decoupled whole-body control: upper- and lower-body policies are trained separately. This design allows greater flexibility to meet different application requirements. For example, an IK policy can be used for high-accuracy tasks while a VLA upper-body policy enables autonomous execution with language input.
Task
Robot
Actions
Observation (pol/crit)
Reward dim
Episode
Terrain
Training
Locomotion
G1
Leg pos (12)
noisy / + scans
15
30 s
rough
10h
Locomotion
T1
Leg pos (10)
noisy / + vel,ht
20
30 s
rough
10h
Loco + Height
G1
Leg pos (12)
noisy / + scans
22
30 s
rough
10h
Stand-up
G1
All joints (rel)
noisy / + forces
18
15 s
stand-up
25h
Stand-up
T1
All joints (rel)
noisy / + forces
18
15 s
stand-up
15h
Motion Imitation
G1
All joints (rel)
noisy / + vel
9
8 s
flat
6h
Table 2: MDP overview and training time (on a single L40) per task.
Figure 4: Velocity and height command tracking in MuJoCo. Each subplot shows a sweep of commanded values (dashed) versus actual robot response (solid) for forward velocity, lateral velocity, yaw rate, and standing height. The solid line is smoothed with the actual showing as the lighter shadow for easier visualization.
Teacher (Privileged)
Student (Distilled)
Det.
Random
Deterministic
Duration
50s
50s
200s
500s
RNN (50s)
Hist. (50s)
vx (m/s)
0.070
0.142± 0.019
0.136± 0.010
0.136± 0.003
0.116
0.097
vy (m/s)
0.083
0.118± 0.013
0.110± 0.006
0.110± 0.002
0.110
0.087
ωz (rad/s)
0.074
0.116± 0.013
0.113± 0.007
0.112± 0.003
0.117
0.079
h (m)
0.035
0.046± 0.007
0.039± 0.003
0.036± 0.001
0.037
0.037
Table 3: Sim-to-Sim evaluation of tracking on Unitree G1 velocity+height control. Left: deterministic sweep (50 s) vs. random commands at increasing durations (2 s resample interval) over 10 runs. Right: teacher vs. distilled student policies (deterministic sweep, 50 s).
Figure 5: GR00T post-training pipeline. (1) RL expert trained with physics randomization. (2) Demonstration collection via tiled parallel rendering with scene randomization. (3) VLA fine-tuning on the collected dataset. (4) Closed-loop evaluation in simulation.
Figure 6: Effect of L2C2 on motion smoothness under increasing observation noise. Subplots: RMS joint acceleration, RMS joint jerk, position limit violations, and high-frequency energy ratio (fraction of joint-trajectory energy above 10 Hz).
Figure 7: Ablation studies (5 seeds, shaded = ±1 std unless noted). (a) Reward normalization; dashed curves scale all rewards by 100 × before training and rescale for comparability. (b) Value-bootstrapped vs. tuned termination penalty (shaded = min/max). (c) Virtual harness on height-controlled locomotion. (d) Symmetry augmentation.
Figure 8: Sim-to-real transfer across five tasks on two robot platforms. (a) Velocity and height controlled locomotion on Unitree G1. (b) Velocity controlled locomotion on Booster T1. (c) Stand-up on Unitree G1. (d) Stand-up on Booster T1. (e) Motion imitation (dancing) on Unitree G1.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Actor network
[256,256,128] (pick&place: [256,128,64] )
Critic network
[512,256,128] (pick&place: [256,128,64] )
Activation
ELU
Learning rate
10−3
Discount γ
0.99 (stand-up: 0.995 )
GAE λ
0.95
Appendix
Table 4: PPO hyperparameters. Task-specific overrides in parentheses.