Agentic LLM Planning via Step-Wise PDDL Simulation: An Empirical Characterisation

LLMs remain unreliable for long-horizon planning, often generating logically inconsistent or non-applicable plans. Recent hybrid methods instead translate natural language into the Planning Domain Definition Language (PDDL), allowing symbolic planners to produce verifiable plans. However, existing methods frequently rely on rigid generation pipelines, a partial PDDL definition, or human feedback. Furthermore, their evaluation is hindered by the lack of standardized benchmarks with automated verification. To address these limitations, we present PDDLCoder, an agentic framework for PDDL generation from natural language that iteratively generates, analyzes, and refines planning specifications. We further introduce NL-pddlgym, a benchmark dataset comprising 711 planning problems across 23 domains with executable gym environments for the automated verification of plan applicability. Experiments on the NL-pddlgym test set containing 106 problems across 4 held-out domains show that PDDLCoder generates applicable plans for 89.6% of tested planning problems. This improves upon our adaptations of previous PDDL generation methods, which achieved up to 45.3%, and outperforms direct LLM planning approaches, which reached up to 74.5% on the same test set. Our work demonstrates the effectiveness of agentic PDDL generation for planning and establishes a reproducible benchmark for future research on LLM-assisted symbolic planning.

While Large Language Models (LLMs) provide semantic flexibility for robotic task planning, their susceptibility to hallucination and logical inconsistency limits their reliability in long-horizon domains. To bridge the gap between unstructured environments and rigorous plan synthesis, we propose DUPLEX, an agentic dual-system neuro-symbolic architecture that strictly confines the LLM to schema-guided information extraction rather than end-to-end planning or code generation. In our framework, a feed-forward Fast System utilizes a lightweight LLM to extract entities, relations etc. from natural language, deterministically mapping them into a Planning Domain Definition Language (PDDL) problem file for a classical symbolic planner. To resolve complex or underspecified scenarios, a Slow System is activated exclusively upon planning failure, leveraging solver diagnostics to drive a high-capacity LLM in iterative reflection and repair. Extensive evaluations across 12 classical and household planning domains demonstrate that DUPLEX significantly outperforms existing end-to-end and hybrid LLM baselines in both success rate and reliability. These results confirm that The key is not to make the LLM plan better, but to restrict the LLM to the part it is good at - structured semantic grounding - and leave logical plan synthesis to a symbolic planner.

Interactive benchmarks for LLMs usually report a single metric, even though succeeding in them requires models to show several different skills. While those benchmarks are useful for measuring the overall capability of models to perform the downstream task, it is harder to identify which skill was lacking and caused failure solely from the aggregate score. For example, in benchmarks where models are supposed to navigate a simulated world to perform a task, failures can come from perception, an ambiguous instruction, retrieval, missing commonsense about what actions do, unknown dynamics, or planning. \benchmark recasts the procedurally generated BabyAI gridworld as a fully observable, purely textual environment designed to evaluate planning separately from those factors. The nature of the environment and the structure of the prompt control for the other factors cited above. To evaluate the models, we define the PPD suite: Predict asks for the state that follows an action sequence, Plan asks for an action sequence that reaches a goal, and Decompose asks for a subgoal sequence that achieves a mission. Across seven frontier and open models, queried zero-shot, we find simulation far ahead of planning for every model, close to saturated for the strongest. The length of the required solution governs planning failure, and each of the four strongest models has a horizon beyond which single-attempt success collapses. The responses of claude-opus-4-6 and claude-sonnet-4-6, which show their working step by step, reveal that they plan by proposing L-shaped corridors, which is consistent with their successful plans staying near-optimal while success falls. At the subgoal level, decomposition precision likewise falls to zero on long missions, while comprehension of those missions persists.