cs.RONov 3, 2026

LLMs as MDP Designers: A Dependency-Aware Agentic Framework for Automated Robot Policy Generation

Authors: Hong Je-Gal, Jinwoo Kim, Chanbin Yi, Hyun-Suk Lee

Organizations: Department of AI & Robotics, Sejong University, Seoul 05006 · Department of AI, Sejong University, Seoul 05006 · Department of AI & Robotics, AI Robotics Institute (AIRI), Sejong University, Seoul 05006

Abstract

We investigate the automated generation of executable robot reinforcement learning (RL) policies from natural-language task descriptions. Rather than using large language models (LLMs) as execution-time decision makers, we use them as designers that construct RL pipelines before deployment, including Markov decision process (MDP) formulation and simulator implementation. We identify a phase-wise structure in this problem: MDP formulation is dependency-structured and sequential, whereas simulator implementation becomes modular once the formulation is fixed. Based on this structure, we propose ARPG, a dependency-aware agentic framework that performs sequential MDP modeling, localized verification and correction, and simulator-structured implementation while externalizing intermediate artifacts. ARPG targets a key failure mode of automated policy generation: executable code may still yield non-learnable policies when state, action, reward, and termination artifacts are not semantically aligned. Experiments in Isaac Sim show that ARPG improves modeling correctness, environment executability, and policy-generation success compared with monolithic LLM baselines and off-the-shelf agentic systems. Additional demonstrations on custom robotics, standard RL, and engineering-domain MDPs further suggest that the same modeling pipeline can be reused beyond the main benchmark suite.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. RDA: Reward Design Agent for Reinforcement Learning

    Jun 1, 2026Hojoon Lee, Ajay Subramanian, Ben Abbatematteo +4Agentic RLReinforcement Learning

  2. Agentic LLM Planning via Step-Wise PDDL Simulation: An Empirical Characterisation

    Mar 6, 2026Kai Göbel, Pierrick Lorang, Patrik Zips +1LLM PlanningLLM Agent Planning

  3. AGAR: a reinforcement learning substrate for LLM program evolution

    Oct 6, 2026Haoran Li, Zengle Ge, Xiaomin Yuan +11LLM-Based Program SynthesisRL for Code Generation