cs.AINov 14, 2026

Continuation-Guided LLM Evolution of Program Policies and Goal Compilers in Craftax

Authors: Víctor Gallego

Organizations: Komorebi AI Technologies

Abstract

Coding agents can synthesize and optimize code-based policies from simulator feedback, allowing the resulting program to act with zero inference model calls. But for long horizon tasks, two challenges arise: local improvements can hurt final success and switching to a new objective requires another expensive synthesis run. We propose a framework that first develops competence by refining the policy through a feedback cascade, moving from episode returns to hierarchical goals and belief updates which can be scored densely from simulator. In the Craftax game, a partially observed and procedurally generated environment, refinements via episode level feedback stall at <1% final-level arrival, whereas with the feedback cascade the optimized policy reaches the final stage 50% of time (28% final boss defeats). Furthermore, to make this competence reusable towards new goals, we factor the previous controller into an environment interpreter II (models, memory, skills) and a rule program p0p_0, and we show that coding agents can optimize a reusable goal compiler CθC_θ that maps structured tasks gg into new rule programs pgp_g. Across five independent compiler searches trained only on short tasks, the resulting compilers successfully transfer zero-shot to longer tasks without any task-specific tuning nor inference-time LLM calls: continuation-scored feedback and a fixed machine with an editable program together turn a single-task competent policy into a reusable one. Code at: https://github.com/vicgalle/craftax-program-policies

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents

    Jun 11, 2026Saehun Chun, Wonje Choi, Sera Choi +2LLM-Based Program SynthesisAgentic Code Generation

  2. CODE-SHARP: Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs

    Feb 10, 2026Richard Bornemann, Pierluigi Vito Amadori, Antoine CullyAgent Skill LearningLifelong Learning Agents

  3. Coding Agents are Strong Prompt Optimizers

    Aug 13, 2026Agamdeep Singh, Srishti Gautam, Priyanshu Gupta +3Automatic Prompt OptimizationCoding Agents