Continuation-Guided LLM Evolution of Program Policies and Goal Compilers in Craftax
Organizations: Komorebi AI Technologies
Abstract
Coding agents can synthesize and optimize code-based policies from simulator feedback, allowing the resulting program to act with zero inference model calls. But for long horizon tasks, two challenges arise: local improvements can hurt final success and switching to a new objective requires another expensive synthesis run. We propose a framework that first develops competence by refining the policy through a feedback cascade, moving from episode returns to hierarchical goals and belief updates which can be scored densely from simulator. In the Craftax game, a partially observed and procedurally generated environment, refinements via episode level feedback stall at <1% final-level arrival, whereas with the feedback cascade the optimized policy reaches the final stage 50% of time (28% final boss defeats). Furthermore, to make this competence reusable towards new goals, we factor the previous controller into an environment interpreter (models, memory, skills) and a rule program , and we show that coding agents can optimize a reusable goal compiler that maps structured tasks into new rule programs . Across five independent compiler searches trained only on short tasks, the resulting compilers successfully transfer zero-shot to longer tasks without any task-specific tuning nor inference-time LLM calls: continuation-scored feedback and a fixed machine with an editable program together turn a single-task competent policy into a reusable one. Code at: https://github.com/vicgalle/craftax-program-policies
Figures & tables
| Rounds | Start states | Acting program | Statistic | |
|---|---|---|---|---|
| A | 1–10 | fresh worlds | throughout | return (rounds 1–4); reaching the frontier floor (5–10) |
| B | 11–13 | fresh worlds in which reaches floor | throughout | reaching floor |
| C | 14 | saved floor-milestone states | from on | reaching floor 8 |
| D | 15–16, ext. | saved floor and goal entries; fresh worlds | from on (repeated use); once, then ; from reset | reaching floor 8 |
| E | ext. | every step of incumbent runs | (the candidate’s beliefs computed, not read) | negative belief error against |
| Frozen comparison | Worlds | Before | After | Gain | 95% interval |
|---|---|---|---|---|---|
| Goal-level: R14 to R16 | 2048 | 13.82 | 31.45 | +17.63 | [+15.26, +19.99] |
| World model | 2048 | 31.88 | 37.55 | +5.66 | [+2.97, +8.36] |
| Modules over composition | 4096 | 40.28 | 45.09 | +4.81 | [+3.08, +6.54] |
| Recursive refinement | 1024 | 44.43 | 48.24 | +3.81 | [+0.71, +6.91] |
| Method | Success | Violations | Steps † |
|---|---|---|---|
| Original | 32.36% | 164 | 38,358 |
| Guards | 36.20% | 0 | 37,258 |
| Reference | 36.91% | 0 | 35,104 |
| Shared compiler | 71.30% | 0 | 23,531 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Purpose | Task programs | Stages | Worlds/task | Seed base |
|---|---|---|---|---|
| Shared training | 16 | 2–4 | 4 | 2,000,000,000 |
| Shared validation | 8 | 2–4 | 8 | 2,010,000,000 |
| Held-out test | 48 | 2–4 / 5–6 / 7–8 | 32 | 2,020,000,000 |
| Per-task tuning | 12 fixed specifications | 2–8 | 8 | 2,030,000,000 |
| Per-task selection | Same 12 | 2–8 | 8 | 2,040,000,000 |
| Checkpoint | Regime | Tokens (M) | Arrival (%) | Wins (%) | Return |
|---|---|---|---|---|---|
| Initial | A | 0.12 | 0.00 | 0.00 | 56.0 |
| R1 | A | 0.17 | 0.00 | 0.00 | 66.2 |
| R2 | A | 0.30 | 0.00 | 0.00 | 75.9 |
| R3 | A | 0.35 | 0.00 | 0.00 | 78.6 |
| R4 | A | 0.42 | 0.00 | 0.00 | 84.0 |
| R5 | A | 0.46 | 0.00 | 0.00 | 84.1 |
| Campaign | Worlds | Before | After | Gain | Decision |
|---|---|---|---|---|---|
| R1 | 4,096 | 47.63 | 48.39 | +0.76 | retain incumbent |
| R2 | 4,096 | 47.09 | 48.32 | +1.22 | retain incumbent |
| R3 | 1,024 | 44.43 | 48.24 | +3.81 | retain candidate |
| R4 | 1,024 | 49.22 | 48.34 | -0.88 | retain incumbent |
| R5 | 1,024 | 51.46 | 51.07 | -0.39 | retain incumbent |
| Campaign | Worlds | Before | After | Gain | Decision |
|---|---|---|---|---|---|
| R1 | 256 | 10.16 | 21.88 | +11.72 | retain candidate |
| R2 | 512 | 25.00 | 27.93 | +2.93 | retain incumbent |
| R3 | 512 | 20.70 | 25.20 | +4.49 | retain candidate |
LLM-Based Program Synthesis
Large language model (LLM)-based program synthesis turns natural-language goals and examples into executable programs, often using tests or formal constraints.
See all 339 papers