Authors: Nicola J. Müller, Naya Rudolph, Katharina Stein, Jörg Hoffmann, Ayal Taitler, Timo P. Gros
Organizations: German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Saarland University, Saarland Informatics Campus, Saarbrücken, Germany · Center for European Research in Trusted Artificial Intelligence (CERTAIN) · Ben-Gurion University of the Negev
PDDL, the de-facto standard language in the AI Planning community, is designed to specify planning domains: sets of instances that share the same predicates and action schemas. Yet it does not provide any means to specify the actual instance set, i.e., legality constraints on initial states and goal conditions, as well as possibly domain subset constraints specifying an instance subset we are interested in. One consequence of this is that instance generation has always been ad-hoc, with manually written domain- and subset-specific instance generators. Recent work has started to address this, through reasoning and learning methods that however suffer from scalability limitations. Here we introduce an alternative approach, leveraging LLMs to generate instance-generation programs, with built-in soundness guarantees through prescribed checks. We show that these automatically generated instance generators return large numbers of sound and diverse instances efficiently.
Figures & tables
Bug Category
Test
Description
Error Message
Code Bugs
Class Loading Test
Try loading the generator class.
Python error message.
Instance Generation Test
Check whether instance generation terminates without a runtime error.
Python error message.
Efficiency Test
Check whether instance generation exceeds the time limit.
“Generation exceeded the time limit of N seconds.”
Instance File Bugs
Parsing Test
Check whether the instance file can be parsed.
Error message of PDDL parser.
Instance Size Test
Check whether the number of objects in the instance matches the requested size.
“Expected N objects, but got M instead.”
Quality Bugs
Goal Fulfilled Test
Check whether the initial state already satisfies the goal.
“The initial state already fulfills the goal.”
Table 1: Overview of bug categories, tests, and their error messages for LLM-generated instance generators. 1 1 footnotemark: 1 An instance with a finite hFF value may still be unsolvable.
Domain
Iter.
Sound
Diversity Relative to IPC
Blocksworld
3
100
+836.7
Blocksworld S
1
100
-100.0
Childsnack
22
100
+34.8
Childsnack S
14
100
-44.7
Ferry
29
100
+41.8
Ferry S
11
100
+11.5
Table 2: Soundness & diversity of I2G’s generators. We show the increase or decrease (+-) in diversity w.r.t. the IPC generators. S indicates the usage of subset constraints, which can enforce a lower diversity by restricting the instance subset.
Easy
Medium
Hard
Domain
I2G
Gr25
I2G
Gr25
I2G
Gr25
Blocksworld
100
93
100
0
100
0
Childsnack
100
97
100
3
97
0
Ferry
100
100
100
90
100
0
Floortile
100
10
100
0
100
0
Miconic
100
100
100
67
100
0
Table 3: Percentages of successfully generated IPC instances per difficulty. The bottom row shows the total runtime of successful generation attempts.
Domain
Easy
Medium
Hard
Blocksworld S
100
100
100
Childsnack S
100
100
100
Ferry S
100
100
100
Floortile S
100
100
93
Miconic S
100
100
100
Rovers S
83
100
43
Table 4: Percentages of successfully generated IPC instances per difficulty when using subset constraints. The bottom row shows the total runtime of successful generation attempts.
Domain
I2G
I2G S
NeSIG
Blocksworld
6s
6s
26s
Miconic
6s
6s
26s
Satellite
6s
6s
29s
Sokoban
6s
7s
221s
Table 5: Time needed to generate 100 instances from NeSIG’s training sizes. S indicates the usage of subset constraints.
Figure 1: Evaluation of GNN & GenPlan policies on Satellite ( top ) and Satellite S ( bottom ).
Figure 2: Evaluation of GPT-5.4 models on Transport ( top ) and Transport S ( bottom ).
Large language model (LLM)-based program synthesis turns natural-language goals and examples into executable programs, often using tests or formal constraints.
Saarland Informatics Campus, Saarland University, Saarbrücken, Germany · German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany