LLMs as MDP Designers: A Dependency-Aware Agentic Framework for Automated Robot Policy Generation
Organizations: Department of AI & Robotics, Sejong University, Seoul 05006 · Department of AI, Sejong University, Seoul 05006 · Department of AI & Robotics, AI Robotics Institute (AIRI), Sejong University, Seoul 05006
Abstract
We investigate the automated generation of executable robot reinforcement learning (RL) policies from natural-language task descriptions. Rather than using large language models (LLMs) as execution-time decision makers, we use them as designers that construct RL pipelines before deployment, including Markov decision process (MDP) formulation and simulator implementation. We identify a phase-wise structure in this problem: MDP formulation is dependency-structured and sequential, whereas simulator implementation becomes modular once the formulation is fixed. Based on this structure, we propose ARPG, a dependency-aware agentic framework that performs sequential MDP modeling, localized verification and correction, and simulator-structured implementation while externalizing intermediate artifacts. ARPG targets a key failure mode of automated policy generation: executable code may still yield non-learnable policies when state, action, reward, and termination artifacts are not semantically aligned. Experiments in Isaac Sim show that ARPG improves modeling correctness, environment executability, and policy-generation success compared with monolithic LLM baselines and off-the-shelf agentic systems. Additional demonstrations on custom robotics, standard RL, and engineering-domain MDPs further suggest that the same modeling pipeline can be reused beyond the main benchmark suite.
Figures & tables
| Component | Monolithic LLM | Off-the-shelf agents | ARPG (Ours) |
| Component-wise implementation | ✗ | ✓ | ✓ |
| Dependency-aware MDP artifactization | ✗ | ✗ | ✓ |
| Artifact-localized correction | ✗ | ✗ | ✓ |
| Task | GPT-5.2 | Gemini-3 | Sonnet 4.6 | |||
| ARPG | One-shot | Iterative | Codex | One-shot | Claude | |
| CartPole | 1.0 / 1.0 / 1.0 | 0.4 / 0.0 / 0.0 | 1.0 / 1.0 / 1.0 | 1.0 / 0.6 / 0.6 | 0.6 / 0.0 / 0.0 | 1.0 / 1.0 / 1.0 |
| Ant | 1.0 / 0.9 / 0.9 | 0.2 / 0.0 / 0.0 | 0.2 / 0.3 / 0.1 | 0.4 / 0.8 / 0.4 | 0.1 / 0.0 / 0.0 | 0.0 / 0.9 / 0.0 |
| Humanoid | 0.9 / 0.7 / 0.4 | 0.1 / 0.0 / 0.0 | 0.1 / 0.3 / 0.0 | 0.1 / 0.5 / 0.1 | 0.3 / 0.0 / 0.0 | 0.0 / 0.9 / 0.0 |
| Reach | 0.9 / 0.9 / 0.9 | 0.0 / 0.0 / 0.0 | 0.0 / 0.6 / 0.0 | 0.4 / 0.3 / 0.2 | 0.0 / 0.0 / 0.0 | 0.5 / 1.0 / 0.3 |
| Lift | 1.0 / 0.9 / 0.4 | 0.0 / 0.0 / 0.0 | 0.0 / 0.9 / 0.0 | 0.0 / 0.6 / 0.0 | 0.0 / 0.0 / 0.0 | 0.0 / 0.9 / 0.0 |
| Task | ARPG | w/o EC | w/o sequential |
| CartPole | 1.0 / 1.0 / 1.0 | 1.0 / 1.0 / 1.0 | 0.9 / 0.9 / 0.9 |
| Ant | 1.0 / 0.9 / 0.9 | 0.9 / 0.9 / 0.6 | 0.3 / 0.9 / 0.1 |
| Humanoid | 0.9 / 0.7 / 0.4 | 0.3 / 1.0 / 0.3 | 0.1 / 1.0 / 0.1 |
| Reach | 0.9 / 0.9 / 0.9 | 0.5 / 0.8 / 0.1 | 0.0 / 1.0 / 0.0 |
| Lift | 1.0 / 0.9 / 0.4 | 0.2 / 1.0 / 0.2 | 0.0 / 0.9 / 0.0 |
| Setting | Examples | Backend |
| Robotics variants | Jumping Ant, Jumping Humanoid, Pick-and-place | Isaac Sim |
| Classic control | MountainCar, Drone delivery | Gymnasium, Grid env. |
| Engineering MDPs | Wireless, Energy | Custom Env. |
| Task | Tokens (M) | Runtime (min) | ||||
| ARPG | Claude | Codex | ARPG | Claude | Codex | |
| CartPole | 0.160 (0.140, 0.020) | 2.586 (0.036, 2.550) | 2.665 (0.199, 2.466) | 2.9 | 8.2 | 7.4 |
| Ant | 0.149 (0.129, 0.020) | 5.711 (0.061, 5.650) | 2.671 (0.177, 2.494) | 3.9 | 17.2 | 7.7 |
| Humanoid | 0.248 (0.208, 0.040) | 7.882 (0.092, 7.790) | 4.201 (0.244, 3.957) | 4.5 | 24.1 | 11.8 |
| Reach | 0.195 (0.165, 0.030) | 1.818 (0.178, 1.640) | 2.433 (0.173, 2.260) | 4.6 | 11.8 | 9.3 |
| Lift | 0.232 (0.192, 0.040) | 1.769 (0.119, 1.650) | 2.315 (0.165, 2.150) | 6.2 | 15.0 | 8.2 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| ARPG | Gemini-3 | GPT-5.2 | Claude | Codex |
| 0.8 | 0.3 | 0.0 | 0.0 | 0.7 |
| Checklist outcome | Count | Percentage |
| Violation corrected ( ) | 130 | 30.95% |
| Violation unresolved ( ) | 74 | 17.62% |
| Satisfied item preserved ( ) | 186 | 44.29% |
| Violation introduced ( ) | 30 | 7.14% |
| Component | Pass criterion | Failure criterion |
|---|---|---|
| Structure | Observation/state, action, reward, and failure/termination are identifiable and mutually consistent. | A field is missing or contradictory. |
| Cart position | Continuous cart displacement is observed or explicitly derivable from observations. | Position is missing or only a Boolean is given. |
| Cart velocity | Continuous cart velocity is observed or explicitly derivable from observations. | Velocity is unavailable. |
| Pole angle | Pole angle relative to upright is observed or equivalently represented. | Pole angle is unavailable or only an upright Boolean is given. |
| Pole angular velocity | Continuous pole angular velocity is observed or explicitly derivable. | Angular velocity is unavailable. |
| Action type | Continuous force/effort control has explicit numerical or symbolic bounds. | Control is discrete or its type/bounds are unresolved. |
| Component | Pass criterion | Failure criterion |
|---|---|---|
| Structure | Observation/state, action, reward, collapse, and timeout are identifiable. | A required field is missing. |
| Base state | Orientation/uprightness, height, and 3D linear/angular velocity have explicit raw or derived sources. | Essential base-state information is unavailable. |
| Joint state | Positions and velocities cover all eight actuated joints. | A required joint or its velocity is missing. |
| Action dimension | One continuous 8D action vector is fully specified. | The action is noncontinuous, or its dimension is unresolved or differs from eight. |
| Action semantics | Joint effort/torque or normalized commands map explicitly to the effort actuators. | Commands conflict with the declared actuator interface. |
| Forward signal | Global progress or speed is rewarded as a locomotion objective. | Forward progress is absent, direction is unresolved, or progress is only indirectly gated. |
| Component | Pass criterion | Failure criterion |
|---|---|---|
| Structure | Observation/state, action, reward, collapse, and timeout are identifiable and consistent. | A field is missing or the schema is contradictory. |
| Base state | Base position/height, orientation, and linear/angular velocities have explicit raw or derived sources. | Required pose or velocity information is unavailable. |
| Joint state | Positions and velocities cover every controlled joint. | A controlled joint or its velocity is missing, or the state and action joint sets disagree. |
| Stability cue | At least one uprightness, heading, height, or foot-contact cue has a declared raw or derived source. | No stability cue is available. |
| Action schema | One continuous per-joint vector has a uniquely determined dimension. | The action is noncontinuous, its dimension is unresolved, or conflicting schemas remain. |
| Action semantics | Position targets, deltas, or normalized targets map to the implicit actuators. | Raw effort-only commands or other semantics conflict with the actuators. |
| Component | Pass criterion | Failure criterion |
|---|---|---|
| Structure | Observation/state, action, reward, and timeout are identifiable. | A required field is missing. |
| Joint state | Positions and velocities cover all seven arm joints. | A required joint or its velocity is missing. |
| Target command | The commanded 3D end-effector position is in, or explicitly derivable from, realized observations. | The commanded position is unavailable from the realized observations. |
| Orientation | If orientation is rewarded, actual and commanded orientations have declared observable sources or equivalents. | One orientation source is unavailable. |
| Action history | If action rate is rewarded, the previous action has an observation or declared manager-memory source. | The previous action is only optional or has no available temporal source. |
| Action dimension | A continuous 7D arm joint-space command is fully specified. | Cartesian/effort-only control conflicts with the interface. |
| Component | Pass criterion | Failure criterion |
|---|---|---|
| Structure | Observation/state, action, staged reward, drop, and timeout are identifiable. | A required field is missing. |
| Robot state | Positions/velocities cover seven arm and two gripper joints. | A required arm or gripper position or velocity is missing. |
| Object position | The object 3D position is observed or explicitly derivable from observations. | It is absent or available only as latent simulator state. |
| Object orientation | Object orientation is observed or equivalently represented. | It is optional only, absent, or latent only. |
| Object linear velocity | The object 3D linear velocity is observed or explicitly derivable. | It is absent or latent only. |
| Object angular velocity | The object 3D angular velocity is observed or explicitly derivable. | It is absent or latent only. |
| Successes | Rate | 95% CI |
| 0/10 | 0.0 | [0.00, 0.28] |
| 1/10 | 0.1 | [0.02, 0.40] |
| 2/10 | 0.2 | [0.06, 0.51] |
| 3/10 | 0.3 | [0.11, 0.60] |
| 4/10 | 0.4 | [0.17, 0.69] |
| 5/10 | 0.5 | [0.24, 0.76] |
| Task | Task type | Backend | Objective | Reported evidence |
| Diagonal CartPole | Robotics variant | Isaac Sim | Balance a cart-pole system with a diagonal initial pole. | Rollout video |
| Jumping Ant | Robotics variant | Isaac Sim | Perform repeated vertical jumps in place. | Rollout video |
| In-place rotation Ant | Robotics variant | Isaac Sim | Rotate the Ant in place without translational drift. | Rollout video |
| Jumping Humanoid | Robotics variant | Isaac Sim | Generate vertical jumping behavior for a humanoid robot. | Rollout video |
| Pick-and-place | Robotics variant | Isaac Sim | Transfer a cube between two tables. | Rollout video |
| CartPole | Classic control | Gymnasium | Balance the pole with discrete cart forces. | Evaluation curve |
| Policy | Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec | Total |
| Expert | 13.65 | 11.91 | 9.50 | 9.27 | 7.99 | 10.27 | 19.27 | 20.69 | 11.35 | 9.27 | 10.59 | 13.62 | 147.40 |
| ARPG | 13.63 | 11.86 | 9.52 | 9.15 | 8.00 | 10.31 | 20.05 | 21.11 | 11.39 | 9.27 | 10.64 | 13.75 | 148.72 |
LLM-Guided RL
Large language model-guided reinforcement learning uses language models to shape rewards, exploration, planning, and policy learning.
See all 156 papers