Zero-Shot Goal Recognition with Large Language Models

Bridging the gap between human intent and machine execution remains a challenge in automated planning, where expressing goals in formal languages like PDDL restricts accessibility to non-experts. This paper empirically evaluates whether current Large Language Models (LLMs) can reliably translate natural language testing goals, written in informal language by video game testers, into well-formed PDDL targets suitable for classical planning. We present a carefully designed prompt template, integrating insights from iterative experimentation, aimed at maximizing both accuracy and response coherence from multiple state-of-the-art LLMs. Six contemporary models are systematically assessed on correctness, speed, and error tendencies using real-world, domain-specific benchmarks. All models demonstrate high correctness, exceeding 92%, with Gemini 2.5 Flash achieving the highest accuracy at 96% and the lowest incidence of false positives, while GPT-4.1 leads in response speed. Despite these advances, critical distinctions exist in model performance, and occasional failures arise from language ambiguity and limitations in domain representation. Our analysis underscores both the significant progress and ongoing gaps in enabling LLMs to act as robust bridges between natural language objectives and automated planning pipelines.

When LLMs exhibit uneven performance across planning tasks, these gaps are often attributed to task difficulty. We argue that this explanation is incomplete, as task-level variation may reflect distinct latent planning competencies rather than differences along a single ability spectrum. We study this question on ACPBench-Hard by evaluating multiple LLM families under varying test-time reasoning budgets and applying a multidimensional item response theory model to uncover the latent competency structure underlying LLM planning. The analysis reveals two principal dimensions that shape planning performance: operational reasoning, the ability to evaluate local action applicability and immediate state transitions, and structural enumeration, the ability to reason about goal reachability and landmark structure. Operational reasoning improving under model scaling and longer reasoning traces, while structural enumeration remains comparatively insensitive. Our findings motivate competency-level evaluation of LLM planning, shifting the focus from whether models improve overall to which planning competencies improve, under what conditions, and why.

When an interactive benchmark reports a single success rate for a language-model agent, it is rarely clear what that number measures. A failure can come from perception, ambiguous instructions, retrieval, missing commonsense about what actions do, an incorrect model of the dynamics, or planning, and an aggregate score does not separate them. LLM-BabyBench recasts the procedurally generated BabyAI gridworld as a fully observable, purely textual environment in which every source of failure but planning is removed by construction. The whole grid is serialised into the prompt, instructions come from a small formal grammar, every object's coordinate is stated, the six actions and their effects are specified, and a deterministic expert validates each answer by executing it rather than judging it. On this substrate we define the PPD suite: Predict asks for the state that follows an action sequence, Plan for an action sequence that reaches a goal, and Decompose for a subgoal sequence that achieves a mission, scored by three assistance-aware metrics that separate understanding a mission from sequencing it. Across seven frontier and open models, simulation is far ahead of planning for every model, near saturation for the strongest and well short of it for the weakest, and the length of the required solution, not grid size or obstacle count, governs planning failure. Each model has a characteristic horizon beyond which single-attempt success collapses. Where enough instances are solved to support the ratio, returned plans stay near-optimal. Models that write out their working show why: they commit to one family of corridor-shaped route and verify it with no means of backing out, so what they return is either near-optimal or invalid. The same pattern holds one level up: decomposition precision falls to zero on long missions even where comprehension persists.