LLMs as MDP Designers: A Dependency-Aware Agentic Framework for Automated Robot Policy Generation
Authors: Hong Je-Gal, Jinwoo Kim, Chanbin Yi, Hyun-Suk Lee
Organizations: Department of AI & Robotics, Sejong University, Seoul 05006 · Department of AI, Sejong University, Seoul 05006 · Department of AI & Robotics, AI Robotics Institute (AIRI), Sejong University, Seoul 05006
We investigate the automated generation of executable robot reinforcement learning (RL) policies from natural-language task descriptions. Rather than using large language models (LLMs) as execution-time decision makers, we use them as designers that construct RL pipelines before deployment, including Markov decision process (MDP) formulation and simulator implementation. We identify a phase-wise structure in this problem: MDP formulation is dependency-structured and sequential, whereas simulator implementation becomes modular once the formulation is fixed. Based on this structure, we propose ARPG, a dependency-aware agentic framework that performs sequential MDP modeling, localized verification and correction, and simulator-structured implementation while externalizing intermediate artifacts. ARPG targets a key failure mode of automated policy generation: executable code may still yield non-learnable policies when state, action, reward, and termination artifacts are not semantically aligned. Experiments in Isaac Sim show that ARPG improves modeling correctness, environment executability, and policy-generation success compared with monolithic LLM baselines and off-the-shelf agentic systems. Additional demonstrations on custom robotics, standard RL, and engineering-domain MDPs further suggest that the same modeling pipeline can be reused beyond the main benchmark suite.
Figures & tables
Figure 1: Human-driven RL policy generation process. The arrows illustrate iterative dependencies and back-and-forth adjustments across stages.
Figure 2: Overview of the ARPG framework for structured formulation and policy generation.
Component
Monolithic LLM
Off-the-shelf agents
ARPG (Ours)
Component-wise implementation
✗
✓
✓
Dependency-aware MDP artifactization
✗
✗
✓
Artifact-localized correction
✗
✗
✓
Table 1: Comparison of structural components.
Task
GPT-5.2
Gemini-3
Sonnet 4.6
ARPG
One-shot
Iterative
Codex
One-shot
Claude
CartPole
1.0 / 1.0 / 1.0
0.4 / 0.0 / 0.0
1.0 / 1.0 / 1.0
1.0 / 0.6 / 0.6
0.6 / 0.0 / 0.0
1.0 / 1.0 / 1.0
Ant
1.0 / 0.9 / 0.9
0.2 / 0.0 / 0.0
0.2 / 0.3 / 0.1
0.4 / 0.8 / 0.4
0.1 / 0.0 / 0.0
0.0 / 0.9 / 0.0
Humanoid
0.9 / 0.7 / 0.4
0.1 / 0.0 / 0.0
0.1 / 0.3 / 0.0
0.1 / 0.5 / 0.1
0.3 / 0.0 / 0.0
0.0 / 0.9 / 0.0
Reach
0.9 / 0.9 / 0.9
0.0 / 0.0 / 0.0
0.0 / 0.6 / 0.0
0.4 / 0.3 / 0.2
0.0 / 0.0 / 0.0
0.5 / 1.0 / 0.3
Lift
1.0 / 0.9 / 0.4
0.0 / 0.0 / 0.0
0.0 / 0.9 / 0.0
0.0 / 0.6 / 0.0
0.0 / 0.0 / 0.0
0.0 / 0.9 / 0.0
Table 2: Success rates across benchmark tasks. All results are reported as triplets in the order of MSR, EISR, and PGSR. Bold indicates the best value for each metric within a task. Corresponding 95% Wilson confidence intervals are provided in Appendix F.3 .
Task
ARPG
w/o EC
w/o sequential
CartPole
1.0 / 1.0 / 1.0
1.0 / 1.0 / 1.0
0.9 / 0.9 / 0.9
Ant
1.0 / 0.9 / 0.9
0.9 / 0.9 / 0.6
0.3 / 0.9 / 0.1
Humanoid
0.9 / 0.7 / 0.4
0.3 / 1.0 / 0.3
0.1 / 1.0 / 0.1
Reach
0.9 / 0.9 / 0.9
0.5 / 0.8 / 0.1
0.0 / 1.0 / 0.0
Lift
1.0 / 0.9 / 0.4
0.2 / 1.0 / 0.2
0.0 / 0.9 / 0.0
Table 3: Ablation results.
Setting
Examples
Backend
Robotics variants
Jumping Ant, Jumping Humanoid, Pick-and-place
Isaac Sim
Classic control
MountainCar, Drone delivery
Gymnasium, Grid env.
Engineering MDPs
Wireless, Energy
Custom Env.
Table 4: Examples of demonstrations.
Task
Tokens (M)
Runtime (min)
ARPG
Claude
Codex
ARPG
Claude
Codex
CartPole
0.160 (0.140, 0.020)
2.586 (0.036, 2.550)
2.665 (0.199, 2.466)
2.9
8.2
7.4
Ant
0.149 (0.129, 0.020)
5.711 (0.061, 5.650)
2.671 (0.177, 2.494)
3.9
17.2
7.7
Humanoid
0.248 (0.208, 0.040)
7.882 (0.092, 7.790)
4.201 (0.244, 3.957)
4.5
24.1
11.8
Reach
0.195 (0.165, 0.030)
1.818 (0.178, 1.640)
2.433 (0.173, 2.260)
4.6
11.8
9.3
Lift
0.232 (0.192, 0.040)
1.769 (0.119, 1.650)
2.315 (0.165, 2.150)
6.2
15.0
8.2
Table 5: Computational cost comparison. Token usage is reported as total tokens, with direct and cached tokens shown in parentheses. Bold indicates the lowest value within each task.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
ARPG
Gemini-3
GPT-5.2
Claude
Codex
0.8
0.3
0.0
0.0
0.7
Appendix
Table 6: MSR under underspecified Humanoid descriptions.
Figure 3: Error-correction agent invocation counts by modeling component with underspecified Humanoid task descriptions.
Checklist outcome
Count
Percentage
Violation corrected ( X→O )
130
30.95%
Violation unresolved ( X→X )
74
17.62%
Satisfied item preserved ( O→O )
186
44.29%
Violation introduced ( O→X )
30
7.14%
Appendix
Table 7: Pre/post checklist outcomes for triggered Humanoid EC events.
Figure 4: Implementation difficulty statistics from coding-agent debug invocations.
Component
Pass criterion
Failure criterion
Structure
Observation/state, action, reward, and failure/termination are identifiable and mutually consistent.
A field is missing or contradictory.
Cart position
Continuous cart displacement is observed or explicitly derivable from observations.
Position is missing or only a Boolean is given.
Cart velocity
Continuous cart velocity is observed or explicitly derivable from observations.
Velocity is unavailable.
Pole angle
Pole angle relative to upright is observed or equivalently represented.
Pole angle is unavailable or only an upright Boolean is given.
Pole angular velocity
Continuous pole angular velocity is observed or explicitly derivable.
Angular velocity is unavailable.
Action type
Continuous force/effort control has explicit numerical or symbolic bounds.
Control is discrete or its type/bounds are unresolved.
Appendix
Table 8: Operational MSR criteria for CartPole.
Component
Pass criterion
Failure criterion
Structure
Observation/state, action, reward, collapse, and timeout are identifiable.
A required field is missing.
Base state
Orientation/uprightness, height, and 3D linear/angular velocity have explicit raw or derived sources.
Essential base-state information is unavailable.
Joint state
Positions and velocities cover all eight actuated joints.
A required joint or its velocity is missing.
Action dimension
One continuous 8D action vector is fully specified.
The action is noncontinuous, or its dimension is unresolved or differs from eight.
Action semantics
Joint effort/torque or normalized commands map explicitly to the effort actuators.
Commands conflict with the declared actuator interface.
Forward signal
Global +X progress or speed is rewarded as a locomotion objective.
Forward progress is absent, direction is unresolved, or progress is only indirectly gated.
Appendix
Table 9: Operational MSR criteria for Ant.
Component
Pass criterion
Failure criterion
Structure
Observation/state, action, reward, collapse, and timeout are identifiable and consistent.
A field is missing or the schema is contradictory.
Base state
Base position/height, orientation, and linear/angular velocities have explicit raw or derived sources.
Required pose or velocity information is unavailable.
Joint state
Positions and velocities cover every controlled joint.
A controlled joint or its velocity is missing, or the state and action joint sets disagree.
Stability cue
At least one uprightness, heading, height, or foot-contact cue has a declared raw or derived source.
No stability cue is available.
Action schema
One continuous per-joint vector has a uniquely determined dimension.
The action is noncontinuous, its dimension is unresolved, or conflicting schemas remain.
Action semantics
Position targets, deltas, or normalized targets map to the implicit actuators.
Raw effort-only commands or other semantics conflict with the actuators.
Appendix
Table 10: Operational MSR criteria for Humanoid.
Component
Pass criterion
Failure criterion
Structure
Observation/state, action, reward, and timeout are identifiable.
A required field is missing.
Joint state
Positions and velocities cover all seven arm joints.
A required joint or its velocity is missing.
Target command
The commanded 3D end-effector position is in, or explicitly derivable from, realized observations.
The commanded position is unavailable from the realized observations.
Orientation
If orientation is rewarded, actual and commanded orientations have declared observable sources or equivalents.
One orientation source is unavailable.
Action history
If action rate is rewarded, the previous action has an observation or declared manager-memory source.
The previous action is only optional or has no available temporal source.
Action dimension
A continuous 7D arm joint-space command is fully specified.
Cartesian/effort-only control conflicts with the interface.
Appendix
Table 11: Operational MSR criteria for Franka Reach.
Component
Pass criterion
Failure criterion
Structure
Observation/state, action, staged reward, drop, and timeout are identifiable.
A required field is missing.
Robot state
Positions/velocities cover seven arm and two gripper joints.
A required arm or gripper position or velocity is missing.
Object position
The object 3D position is observed or explicitly derivable from observations.
It is absent or available only as latent simulator state.
Object orientation
Object orientation is observed or equivalently represented.
It is optional only, absent, or latent only.
Object linear velocity
The object 3D linear velocity is observed or explicitly derivable.
It is absent or latent only.
Object angular velocity
The object 3D angular velocity is observed or explicitly derivable.
It is absent or latent only.
Appendix
Table 12: Operational MSR criteria for Franka Lift.
Successes
Rate
95% CI
0/10
0.0
[0.00, 0.28]
1/10
0.1
[0.02, 0.40]
2/10
0.2
[0.06, 0.51]
3/10
0.3
[0.11, 0.60]
4/10
0.4
[0.17, 0.69]
5/10
0.5
[0.24, 0.76]
Appendix
Table 13: 95% Wilson CIs.
Task
Task type
Backend
Objective
Reported evidence
Diagonal CartPole
Robotics variant
Isaac Sim
Balance a cart-pole system with a diagonal initial pole.
Rollout video
Jumping Ant
Robotics variant
Isaac Sim
Perform repeated vertical jumps in place.
Rollout video
In-place rotation Ant
Robotics variant
Isaac Sim
Rotate the Ant in place without translational drift.
Rollout video
Jumping Humanoid
Robotics variant
Isaac Sim
Generate vertical jumping behavior for a humanoid robot.
Rollout video
Pick-and-place
Robotics variant
Isaac Sim
Transfer a cube between two tables.
Rollout video
CartPole
Classic control
Gymnasium
Balance the pole with discrete cart forces.
Evaluation curve
Appendix
Table 14: Summary of additional demonstrations beyond the main Isaac Lab benchmark tasks.
Figure 5: Evaluation results for additional standard RL and stochastic decision-making tasks.
Figure 6: Comparison with the expert-designed DQN policy for the wireless scheduling task.
Figure 7: Comparison of daily patterns with the expert-designed DQN policy for the energy management task.
Policy
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Total
Expert
13.65
11.91
9.50
9.27
7.99
10.27
19.27
20.69
11.35
9.27
10.59
13.62
147.40
ARPG
13.63
11.86
9.52
9.15
8.00
10.31
20.05
21.11
11.39
9.27
10.64
13.75
148.72
Appendix
Table 15: Monthly and annual electricity costs for the expert and ARPG-generated energy management policies.
Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions. Recent work, such as Eureka, automates reward design by using an LLM to iteratively generate and refine reward code from task descriptions. However, they rely on coarse feedback signals such as success rate, which provide little semantic insight into the learned behavior. As a result, their trained policies achieve the final goal but are frequently poorly aligned with task instructions. We introduce the Reward Design Agent (RDA), a VLM-based agentic framework that injects semantic understanding into reward design. RDA decomposes tasks, visually evaluates trajectories, summarizes failure modes, and iteratively revises reward code to better align with task instructions. Across 12 tabletop manipulation tasks from ManiSkill and 4 whole-body manipulation tasks from HumanoidBench, RDA produces policies substantially more instruction-aligned than those of other baselines, while achieving comparable task success rates. Videos and the generated reward code are available on https://nitinkamra1992.github.io/reward-design-agent.
Task planning, the problem of sequencing actions to reach a goal from an initial state, is a core capability requirement for autonomous robotic systems. Whether large language models (LLMs) can serve as viable planners alongside classical symbolic methods remains an open question. We present PyPDDLEngine, an open-source Planning Domain Definition Language (PDDL) simulation engine that exposes planning operations as LLM tool calls through a Model Context Protocol (MCP) interface. Rather than committing to a complete action sequence upfront, the LLM acts as an interactive search policy that selects one action at a time, observes each resulting state, and can reset and retry. We evaluate four approaches on 102 International Planning Competition (IPC) Blocksworld instances under a uniform 180-second budget: Fast Downward lama-first and seq-sat-lama-2011 as classical baselines, direct LLM planning (Claude Haiku 4.5), and agentic LLM planning via PyPDDLEngine. Fast Downward achieves 85.3% success. The direct and agentic LLM approaches achieve 63.7% and 66.7%, respectively, a consistent but modest three-percentage-point advantage for the agentic approach at 5.7× higher token cost per solution. Across most co-solved difficulty blocks, both LLM approaches produce shorter plans than seq-sat-lama-2011 despite its iterative quality improvement, a result consistent with training-data recall rather than generalisable planning. These results suggest that agentic gains depend on the nature of environmental feedback. Coding agents benefit from externally grounded signals such as compiler errors and test failures, whereas PDDL step feedback is self-assessed, leaving the agent to evaluate its own progress without external verification.
Kai Göbel, Pierrick Lorang, Patrik Zips +1
AIT Austrian Institute of Technology GmbH, Vienna, Austria
Given a task and an evaluator, a language model can rewrite a candidate program while a search loop decides which rewrites survive, offering a practical route to algorithm discovery. But that loop is governed by five constants set by hand: which parent to select, how hard to mutate, how to keep diversity, what to remember, and a scalar score that never says which part of the program earned it. Reinforcement learning already has an estimator for each. The obstacle is that program evolution is not usually written down as a decision process. We formalize it as a Markov decision process whose action is the modular prefix the model is conditioned on, rather than the program it emits. Credit assignment, value estimation, adaptive exploration, and experience memory can then attach to distinct components. AGAR (Algorithm Generation As RL) provides the resulting substrate: any estimator can be replaced or switched off without changing the controller, making the transfer auditable one mechanism at a time, with no gradient training of the backend model. Across 19 tasks, two backends, and three seeds under one harness, AGAR improves on the stronger of two published baselines on most tasks, with gains concentrated in the competitive-programming family. The formalization also yields a checkable reading of prior work: these systems are implicitly zero-discount, not by choice, but because fitness is exogenous to an individual rather than a return over successors, leaving a discount factor nothing to act on.
Haoran Li, Zengle Ge, Xiaomin Yuan +11
University of the Chinese Academy of Sciences · Baidu · Tsinghua University +2