Agent-Driven Autonomous Reinforcement Learning Research: Iterative Policy Improvement for Quadruped Locomotion
Organizations: Department of Mechanical Engineering, Indian Institute of Technology Kanpur, Kanpur, India
Abstract
This paper documents a case study in agent-driven autonomous reinforcement learning research for quadruped locomotion. The setting was not a fully self-starting research system. A human provided high-level directives through an agentic coding environment, while an agent carried out most of the execution loop: reading code, diagnosing failures, editing reward and terrain configurations, launching and monitoring jobs, analyzing intermediate metrics, and proposing the next wave of experiments. Across more than 70 experiments organized into fourteen waves on a DHAV1 12-DoF quadruped in Isaac Lab, the agent progressed from early rough-terrain runs with mean reward around 7 to a best logged Wave 12 run, exp063, with velocity error 0.263 and 97% timeout over 2000 iterations, independently reproduced five times across different GPUs. The archive also records several concrete autonomous research decisions: isolating PhysX deadlocks to terrain sets containing boxes and stair-like primitives, porting four reward terms from openly available reference implementations \cite{deeprobotics, rlsar}, correcting Isaac Sim import and bootstrapping issues, reducing environment count for diagnosis, terminating hung runs, and pivoting effort away from HIM after repeated terrain=0.0 outcomes. Relative to the AutoResearch paradigm \cite{autoresearch}, this case study operates in a more failure-prone robotics RL setting with multi-GPU experiment management and simulator-specific engineering constraints. The contribution is empirical and documentary: it shows that an agent can materially execute the iterative RL research loop in this domain with limited human intervention, while also making clear where human direction still shaped the agenda.
Figures & tables
| Wave | Agent decision | Evidence used | Why it mattered |
|---|---|---|---|
| 3–5 | Continue testing Basic PPO and HIM separately instead of collapsing to one policy family too early | Basic PPO showed partial terrain progress while HIM often survived episodes without terrain advancement | Preserved a fair comparison long enough to show that HIM consistently stayed at terrain=0.0 in the later rough-terrain setup |
| 5 | Reduce environment count in diagnostic runs | Hangs might have been caused by large-scale parallelism or buffer pressure | Showed that fewer environments delayed failures but did not remove them, which narrowed the cause away from batch size alone |
| 6 | Isolate terrain families and switch to a reference-inspired smooth terrain mixture | Controlled runs implicated boxes and stair-like primitives in repeated PhysX deadlocks | Produced the first reliable long rough-terrain runs, making later reward comparisons interpretable |
| 6 | Kill hung experiments and replace them instead of waiting for eventual recovery | Repeated deadlocks showed no sign of self-recovery and blocked GPU capacity | Increased throughput and kept the search loop moving on a shared 4-GPU workstation |
| 7 | Read an openly available reference implementation [ 1 ] and port four reward functions | The reference included gait and symmetry incentives absent from the current setup | Added diagonal gait reward, air-time variance regularization, joint-mirror symmetry, and feet-contact-without-command shaping to the search space |
| 7 | Fix import and runtime issues caused by eager module imports during Isaac Sim startup | Omni and pxr bootstrapping problems blocked some experiment paths before training began | Converted some failures from infrastructure issues into trainable runs |
| Wave | Focus and main observation |
|---|---|
| 1–2 | Baselines and softer penalties, both Basic PPO and HIM struggled on rough terrain under the initial reward design. |
| 3 | Terminal-observation handling was fixed for HIM, but HIM still remained at terrain 0.0 while Basic PPO reached terrain 0.53 before hanging. |
| 4 | Architecture diagnostics showed stronger tracking improved HIM reward but did not produce terrain progression. |
| 5 | Curriculum, spawn, PhysX buffer, and environment-count tests did not eliminate hangs, fewer environments delayed but did not remove failures. |
| 6 | Terrain-style isolation identified the DR-style mixture as hang-free in the logged runs. |
| 7 | Four reference-inspired reward terms were ported, aggressive penalties caused dead policies. |
| Exp | Configuration | Reward | Terrain | Outcome |
|---|---|---|---|---|
| exp019 | Basic PPO, softened rewards, default rough terrain | 21.0 | 0.53 | Hung at about 154 iterations |
| exp028b | Basic PPO, DR-style terrain, softened rewards | – | 6.59 | Completed 500 iterations |
| exp029 | HIM, DR-style terrain, strong tracking | – | 0.0 | Completed 500 iterations |
| exp033 | Basic PPO, DR terrain, aggressive penalties | -0.85 | 0.0 | Dead policy |
| exp035 | Basic PPO, DR terrain, moderate penalties | 65.6 | 5.87 | Stable at 1363 iterations |
| exp039 | Basic PPO, DR terrain, moderate penalties, gait boost, stronger tracking | 116.3 | 6.18 | Best Wave 8 run at 991 iterations |
| Setting | Observed effect in the archive |
|---|---|
| Softer early penalties (for example flat_orientation_l2 : -5.0 to -1.0) | Improved exploration relative to the initial rough-terrain setting, but did not solve simulator hangs or HIM’s terrain=0.0 behavior by itself. |
| DR-style reward ports with moderate penalties | Produced the strongest Basic PPO runs, including exp035 and exp039. |
| Aggressive penalties (for example base_height_l2 =-10, flat_orientation_l2 =-10, contact_forces =-0.1) | Caused dead policies in exp033 and related runs, with terrain staying at 0.0 and base-contact termination at 100%. |
| Stronger gait reward and command tracking in the best Wave 8 configuration | Improved progress relative to exp035 and led to the best logged result in exp039. |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,717 papers