cs.ROMar 28, 2026

Agent-Driven Autonomous Reinforcement Learning Research: Iterative Policy Improvement for Quadruped Locomotion

Authors: Nimesh Khandelwal, Shakti S. Gupta

Organizations: Department of Mechanical Engineering, Indian Institute of Technology Kanpur, Kanpur, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper documents a case study in agent-driven autonomous reinforcement learning research for quadruped locomotion. The setting was not a fully self-starting research system. A human provided high-level directives through an agentic coding environment, while an agent carried out most of the execution loop: reading code, diagnosing failures, editing reward and terrain configurations, launching and monitoring jobs, analyzing intermediate metrics, and proposing the next wave of experiments. Across more than 70 experiments organized into fourteen waves on a DHAV1 12-DoF quadruped in Isaac Lab, the agent progressed from early rough-terrain runs with mean reward around 7 to a best logged Wave 12 run, exp063, with velocity error 0.263 and 97% timeout over 2000 iterations, independently reproduced five times across different GPUs. The archive also records several concrete autonomous research decisions: isolating PhysX deadlocks to terrain sets containing boxes and stair-like primitives, porting four reward terms from openly available reference implementations \cite{deeprobotics, rlsar}, correcting Isaac Sim import and bootstrapping issues, reducing environment count for diagnosis, terminating hung runs, and pivoting effort away from HIM after repeated terrain=0.0 outcomes. Relative to the AutoResearch paradigm \cite{autoresearch}, this case study operates in a more failure-prone robotics RL setting with multi-GPU experiment management and simulator-specific engineering constraints. The contribution is empirical and documentary: it shows that an agent can materially execute the iterative RL research loop in this domain with limited human intervention, while also making clear where human direction still shaped the agenda.

Figures & tables

WaveAgent decisionEvidence usedWhy it mattered
3–5Continue testing Basic PPO and HIM separately instead of collapsing to one policy family too earlyBasic PPO showed partial terrain progress while HIM often survived episodes without terrain advancementPreserved a fair comparison long enough to show that HIM consistently stayed at terrain=0.0 in the later rough-terrain setup
5Reduce environment count in diagnostic runsHangs might have been caused by large-scale parallelism or buffer pressureShowed that fewer environments delayed failures but did not remove them, which narrowed the cause away from batch size alone
6Isolate terrain families and switch to a reference-inspired smooth terrain mixtureControlled runs implicated boxes and stair-like primitives in repeated PhysX deadlocksProduced the first reliable long rough-terrain runs, making later reward comparisons interpretable
6Kill hung experiments and replace them instead of waiting for eventual recoveryRepeated deadlocks showed no sign of self-recovery and blocked GPU capacityIncreased throughput and kept the search loop moving on a shared 4-GPU workstation
7Read an openly available reference implementation [ 1 ] and port four reward functionsThe reference included gait and symmetry incentives absent from the current setupAdded diagonal gait reward, air-time variance regularization, joint-mirror symmetry, and feet-contact-without-command shaping to the search space
7Fix import and runtime issues caused by eager module imports during Isaac Sim startupOmni and pxr bootstrapping problems blocked some experiment paths before training beganConverted some failures from infrastructure issues into trainable runs
TABLE I: Autonomous decision points made by the agent during the study.
WaveFocus and main observation
1–2Baselines and softer penalties, both Basic PPO and HIM struggled on rough terrain under the initial reward design.
3Terminal-observation handling was fixed for HIM, but HIM still remained at terrain 0.0 while Basic PPO reached terrain 0.53 before hanging.
4Architecture diagnostics showed stronger tracking improved HIM reward but did not produce terrain progression.
5Curriculum, spawn, PhysX buffer, and environment-count tests did not eliminate hangs, fewer environments delayed but did not remove failures.
6Terrain-style isolation identified the DR-style mixture as hang-free in the logged runs.
7Four reference-inspired reward terms were ported, aggressive penalties caused dead policies.
TABLE II: Wave-by-wave progression of the checked-in record.
ExpConfigurationRewardTerrainOutcome
exp019Basic PPO, softened rewards, default rough terrain21.00.53Hung at about 154 iterations
exp028bBasic PPO, DR-style terrain, softened rewards–6.59Completed 500 iterations
exp029HIM, DR-style terrain, strong tracking–0.0Completed 500 iterations
exp033Basic PPO, DR terrain, aggressive penalties-0.850.0Dead policy
exp035Basic PPO, DR terrain, moderate penalties65.65.87Stable at 1363 iterations
exp039Basic PPO, DR terrain, moderate penalties, gait boost, stronger tracking116.36.18Best Wave 8 run at 991 iterations
TABLE III: Repository-backed runs that define the main conclusions.
SettingObserved effect in the archive
Softer early penalties (for example flat_orientation_l2 : -5.0 to -1.0)Improved exploration relative to the initial rough-terrain setting, but did not solve simulator hangs or HIM’s terrain=0.0 behavior by itself.
DR-style reward ports with moderate penaltiesProduced the strongest Basic PPO runs, including exp035 and exp039.
Aggressive penalties (for example base_height_l2 =-10, flat_orientation_l2 =-10, contact_forces =-0.1)Caused dead policies in exp033 and related runs, with terrain staying at 0.0 and base-contact termination at 100%.
Stronger gait reward and command tracking in the best Wave 8 configurationImproved progress relative to exp035 and led to the best logged result in exp039.
TABLE IV: Reward-weight pattern observed in the later waves.
Fig. 1: Mean reward vs. iteration for key experiments across waves. Later waves with refined reward weights and terrain configuration show consistently higher and faster-rising reward curves.
Fig. 2: Planar velocity tracking error across key experiments. Lower values indicate better command following. The best run (exp063) converges to error 0.263, reproduced five times.
Fig. 3: Experiment completion rate by wave. Red: << 50%, yellow: 50–80%, green: >> 80%. Fractions show completed/launched. The PhysX hang problem persisted throughout the study, with Wave 11 achieving the first 100% completion rate.
Fig. 4: HIM vs Basic PPO on DR terrain with identical reward configuration. (a) Mean reward, (b) velocity tracking error, (c) episode survival rate. Basic PPO dramatically outperforms HIM across all metrics in this specific project configuration.
Fig. 5: Best velocity error per wave. The optimization converges at Wave 12 (vel_err=0.263). Waves 13–14 explored the remaining parameter space without improvement. Vertical gray lines indicate waves without comparable velocity error data (early diagnostic or HIM-only waves).
Fig. 6: Parameter sensitivity analysis. Each subplot varies one parameter while holding others at their best values. Gold stars mark optima, red crosses mark hung experiments. The clear U-shapes and failure modes demonstrate that the agent’s systematic single-variable methodology was effective at identifying the optimal configuration.
Fig. 7: Peak mean reward per experiment. Green: strong ( >> 80), yellow: moderate ( >> 20), red: failed/collapsed. Later experiments consistently reach higher peaks, reflecting the cumulative effect of agent-driven reward and terrain tuning.
Fig. 8: Episode survival rate (timeout %) across key experiments. Higher values indicate fewer premature terminations due to falls. The best configurations sustain >> 95% timeout rates, indicating stable locomotion throughout most episodes.
Fig. 9: Reward decomposition for exp063 (best configuration). Top: positive components (tracking, gait). Bottom: penalty components. At convergence, tracking rewards dominate while stability penalties are near zero, indicating the policy learned stable locomotion.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Explore similar work

CardsList
Sep 21, 2026

MimicAgent: Quadruped Skills via Prompt-to-Trajectory Generation

We present MimicAgent, a prompt-to-trajectory generation framework for learning dynamic quadruped skills. Although reward shaping is extensively used when training quadruped policies, navigating the resulting reward landscape is notoriously difficult, requiring hours of "graduate student descent". Eureka attempts to automate reward design with LLMs, but we find that it struggles to generalize across diverse skills and morphologies. Our key observation is that it is far easier for a human - and by association, an LLM - to generate reference motions than to shape reward functions. Our hypothesis is motivated by the success of example-guided RL for humanoids, which exploits large-scale motion capture datasets as references for training locomotion policies. Unlike humanoids, quadrupeds lack such reference motion data. Towards this end, we propose MimicAgent, an agentic harness that, given a skill prompt, generates quadruped reference trajectories with coding agents. These coarse reference trajectories are then used to train example-guided RL policies that are deployable in simulation and in the real-world. Notably, we find that when prompting Claude Fable 5.1 within our agentic harness, 87% of prompts yield semantically aligned reference trajectories.

Lucky Kant Nayak, Narayanan Palghat Parameswaran, Neehar Peri +1

Carnegie Mellon University

Reinforcement LearningTrajectory GenerationLLM Agent Trajectory SynthesisRL for Legged LocomotionPolicy LearningQuadruped Locomotion
AskPDF
Jul 30, 2026

An AI Scientist that Doesn't Drift: Taste, Structure, and Falsifiable Findings in a Quadruped Navigation Research Loop

Autonomous research loops driven by large language models can run machine-learning experiments at scale but tend to drift toward local refinements of whichever metric they optimise rather than testing the hypotheses that motivate the experiments. We address this structurally and present an AI Scientist for studying generalisation in quadruped robot navigation policies in simulation. Building on the autoresearch paradigm of Karpathy, our loop adds three components: an immutable experiment card that pairs each iteration's prediction with its outcome under a fixed schema, so a falsified hypothesis cannot be retconned; specialised subagents restricted to mechanical roles; and kkanbu, a preference oracle that holds the user's research taste as a typed knowledge graph and is the only component permitted to make subjective judgements. To isolate the oracle we run the identical loop twice across eleven research streams, with and without kkanbu. Neither arm drifts: both falsify roughly three quarters of their own hypotheses, and the best trained policy comes from the oracle-less arm. What the oracle changes is direction, not score: it alone explores test-time adaptation, it authored the winning designs where its arm led, and it carried lessons across streams that the other arm repeatedly re-derived. The scaffold keeps the loop honest; kkanbu decides where it looks.

Yiwen Zhang, Eloise Zeng, Jaeha Lee +1

Division of Physics, Mathematics and Astronomy, California Institute of Technology, Pasadena, CA 91125, U.S.A. · Department of Mechanical and Civil Engineering, California Institute of Technology, Pasadena, CA 91125, U.S.A.

AI for ScienceHuman-in-the-Loop AIAutonomous ExperimentationAutonomous Research Agents
AskPDF
Jul 20, 2026

Towards Torque-Driven Reinforcement Learning for Quadruped Locomotion

Reinforcement learning (RL) for legged robots is advancing locomotion, demonstrating its ability to adapt to new and challenging terrain. Traditionally, these RL locomotion frameworks are position-based, making the policy less adaptable to terrain types and requiring state estimation techniques in the observation space, i.e., linear velocity. Moreover, these RL frameworks often use small, lightweight quadrupeds that are limited in their viability for high-complexity tasks due to hardware constraints. This work explores an RL torque control framework for heavyweight high-torque quadrupeds. The RL framework in this paper can traverse rough terrain and effectively track a desired linear velocity without requiring knowledge of the agent's current velocity. Using Nvidia's Isaac Sim and Isaac Lab, simulation results of the RL torque control policy are shown on the Unitree B1 quadruped, achieving speeds of 3.5 m/s and 1.5 rad/s. In addition, the quadruped can walk up and down stairs without the aid of an exteroceptive sensor.

Jordan Dowdy, Jean Chagas Vaz

Department of Electrical and Computer Engineering at the University of Louisville. · Faculty of Electrical and Computer Engineering, University of Louisville, Louisville, KY, 40208, USA.

Reinforcement LearningTerrain-Aware Robot LocomotionRL for Legged LocomotionQuadruped Locomotion
AskPDF