A core quality of general intelligence is the ability to open-endedly expand and evolve its set of mastered skills autonomously. While recent Foundation Model (FM) driven approaches have shown promising results towards this goal, they typically rely on significant human-in-the-loop engineering, limiting their transferability to novel environments. To address this, we introduce Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs (CODE-SHARP), a framework that leverages FMs to open-endedly grow and evolve an archive of Python programs encoding skills to train a generalist agent policy entirely from scratch via reinforcement learning, directly from source code. These programs, termed Skills as Hierarchical Reward Programs (SHARPs), each encode a local success condition and a set of prerequisites delegated to previously discovered SHARPs. At runtime, SHARPs dynamically route the agent through their prerequisite chain based on the current state, rewarding each completion along the way, requiring the agent to learn only the marginal behaviour each new SHARP introduces, enabling efficient learning of long-horizon skills without any pre-defined rewards. On Craftax-Classic and XLand-MiniGrid, agents trained fully autonomously by CODE-SHARP outperform previous works by 6x and 2.7x in median performance and are the only agents capable of crafting iron tools and mining diamonds. On AI2-THOR, CODE-SHARP achieves 66.7% median success versus 0% for ELLM and OMNI, using only the initial environment state instead of the full source code. Scaled to Craftax-Extended, CODE-SHARP trains a generalist agent on over 90 discovered SHARPs, enabling the agent to solve challenging long-horizon tasks zero-shot, matching agents trained on ground-truth rewards.
Figures & tables
Figure 1: CODE-SHARP consists of two FM-driven iterative processes that discover and evolve skill encoding SHARPs. The skill proposal generator , implementor , and judge generate and filter novel SHARPs before environment evaluation. The skill mutation generator and implementor produce mutated versions of existing SHARPs, evaluated directly in the environment.
Figure 2: Pseudo-code of a SHARP
Figure 3: a) : Distribution of success rates on all achievements in Craftax-Classic aggregated over five runs. Shaded area indicates the interquartile range. b) : Average success rate on selected achievements on Craftax-Classic, error bars denote the bootstrapped 95% CI around the mean. c) : Distribution of success rates on all achievements in XLand-MiniGrid across five runs.
Figure 4: Distribution of achievement success rates in AI2-THOR across three runs.
Figure 5: CODE-SHARP continuously builds on existing SHARPs in the archive to define novel, meaningful skills mirroring the natural skill curriculum of Craftax.
Figure 6: (a) Score achieved by CODE-SHARP across benchmark tasks. (b) The policy planner utilises increasingly complex SHARPs to define policies-in-code throughout training, resulting in large performance gains relative to the average baseline.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Achievement
CODE-SHARP (Ours)
CODE-FRP (Ours)
ELLM
OMNI
Place Table
1.000
0.998
0.971
0.989
Collect Sapling
0.999
0.998
0.994
0.044
Make Wood Pickaxe
0.999
0.994
0.727
0.779
Make Wood Sword
0.999
0.800
0.230
0.965
Place Furnace
0.998
0.575
0.011
0.273
Collect Stone
0.998
0.200
0.097
0.699
Appendix
Table 1: Per-achievement mean success rate on Craftax Achievements.
Achievement
CODE-SHARP (Ours)
CODE-FRP (Ours)
ELLM
OMNI
Pick Up Square Red
0.978
0.683
0.908
0.493
Pick Up Ball Green
0.969
0.479
0.877
0.552
Pick Up Ball Blue
0.962
0.576
0.885
0.396
Pick Up Square Purple
0.959
0.346
0.848
0.163
Pick Up Square Green
0.953
0.446
0.864
0.408
Pick Up Ball Red
0.938
0.454
0.889
0.636
Appendix
Table 2: Per-achievement mean success rate on XLand-MiniGrid achievements.
Depth
# Ach.
CODE-SHARP (OURS)
CODE-FRP (OURS)
ELLM
OMNI
1
32
0.903
0.893
0.611
0.342
2
37
0.485
0.314
0.048
0.218
3
23
0.342
0.130
0.009
0.029
4
1
0.125
0.000
0.000
0.000
5
3
0.000
0.000
0.000
0.000
6
2
0.083
0.000
0.000
0.000
Appendix
Table 3: Per-sub-task-depth achievement success rate by method on AI2-THOR.
Milestone
CODE-SHARP
CODE-FRP
Finetuned
task-specific
ReAct
Craft wood pickaxe
92.6 (80.9, 98.8)
79.0 (54.6, 96.6)
98.7 (98.3, 99.0)
99.0 (98.8, 99.2)
100.0 (100.0, 100.0)
Craft stone pickaxe
87.5 (65.9, 98.6)
63.5 (38.7, 85.4)
96.0 (95.3, 96.9)
97.7 (96.9, 98.4)
88.9 (66.7, 100.0)
Craft iron pickaxe
75.8 (51.7, 93.0)
17.6 (1.7, 39.6)
84.8 (83.5, 86.1)
0.0 (0.0, 0.0)
33.3 (0.0, 66.7)
Craft iron sword
69.7 (48.5, 86.4)
11.7 (0.0, 28.7)
65.9 (64.1, 67.9)
0.0 (0.0, 0.0)
11.1 (0.0, 33.3)
Craft iron armour
34.9 (21.2, 48.6)
3.4 (0.0, 8.5)
12.2 (10.6, 14.0)
0.0 (0.0, 0.0)
0.0 (0.0, 0.0)
Overall Score
66.52 (48.17, 79.96)
16.43 (4.77, 29.09)
58.08 (56.33, 60.00)
5.29 (5.28, 5.31)
10.80 (4.91, 18.73)
Appendix
Table 4: Per-milestone success rates (%) on the Crafting benchmark. Values are mean (95% CI), computed via bootstrap over 2000 resamples.
Milestone
CODE-SHARP
CODE-FRP
Finetuned
task-specific
ReAct
Craft wood pickaxe
98.5 (98.1, 98.9)
96.4 (95.1, 97.5)
98.7 (98.3, 98.9)
98.7 (98.1, 99.1)
100.0 (100.0, 100.0)
Craft stone pickaxe
98.3 (97.7, 98.7)
81.8 (75.4, 87.7)
96.7 (96.1, 97.4)
98.4 (97.7, 98.9)
88.9 (66.7, 100.0)
Craft torches
97.6 (96.7, 98.4)
73.7 (65.9, 80.3)
96.2 (95.4, 97.0)
97.4 (96.4, 98.3)
77.8 (44.4, 100.0)
Descend to dungeon
89.1 (84.2, 93.4)
34.7 (28.2, 41.9)
81.7 (80.3, 83.2)
34.5 (28.9, 40.3)
11.1 (0.0, 33.3)
Kill 8 dungeon monsters
69.6 (58.4, 81.3)
8.9 (4.8, 13.4)
57.4 (54.1, 60.7)
0.0 (0.0, 0.0)
11.1 (0.0, 33.3)
Descend to gnomish mines
47.3 (35.4, 59.6)
2.6 (1.8, 3.3)
41.6 (38.3, 44.9)
0.0 (0.0, 0.0)
11.1 (0.0, 33.3)
Appendix
Table 5: Per-milestone success rates (%) on the Mines benchmark. Values are mean (95% CI), computed via bootstrap 2000 resamples.
Milestone
CODE-SHARP
CODE-FRP
Finetuned
task-specific
ReAct
Craft wood pickaxe
98.3 (97.0, 99.2)
93.8 (88.7, 97.2)
99.0 (98.7, 99.4)
99.2 (98.8, 99.6)
100.0 (100.0, 100.0)
Craft stone pickaxe
98.0 (96.4, 99.0)
82.0 (74.3, 89.3)
98.4 (97.7, 99.0)
99.0 (98.4, 99.6)
88.9 (66.7, 100.0)
Craft stone sword
97.8 (96.1, 99.0)
76.2 (65.6, 86.3)
97.7 (96.8, 98.6)
99.0 (98.4, 99.6)
88.9 (66.7, 100.0)
Craft torches
97.0 (95.0, 98.4)
69.4 (57.4, 79.8)
97.6 (96.6, 98.5)
98.8 (98.0, 99.6)
77.8 (44.4, 100.0)
Craft arrows
96.8 (95.1, 98.3)
67.4 (56.5, 77.7)
97.3 (96.2, 98.4)
98.8 (98.0, 99.6)
77.8 (44.4, 100.0)
Collect sapling
96.8 (95.0, 98.3)
67.3 (56.9, 77.9)
93.6 (92.0, 95.3)
98.8 (98.0, 99.6)
33.3 (0.0, 66.7)
Appendix
Table 6: Per-milestone success rates (%) on the Dungeon benchmark. Values are mean (95% CI), computed via percentile bootstrap over 2000 resamples
Milestone
CODE-SHARP
CODE-FRP
Finetuned
task-specific
ReAct
Descend to dungeon
86.2 (80.6, 91.1)
36.5 (26.0, 46.1)
92.1 (91.2, 93.0)
10.3 (9.4, 12.1)
33.3 (0.0, 66.7)
Kill 8 dungeon monsters
78.9 (72.0, 85.1)
10.2 (4.0, 18.4)
56.9 (52.0, 61.2)
0.0 (0.0, 0.0)
11.1 (0.0, 33.3)
Descend to gnomish mines
59.8 (50.6, 68.8)
5.0 (1.4, 9.5)
38.9 (36.2, 41.5)
0.0 (0.0, 0.0)
11.1 (0.0, 33.3)
Kill 8 gnomish mines monsters
12.6 (4.9, 22.4)
0.0 (0.0, 0.0)
9.7 (8.2, 11.4)
0.0 (0.0, 0.0)
0.0 (0.0, 0.0)
Descend to sewers
0.9 (0.3, 1.6)
0.0 (0.0, 0.0)
5.6 (4.7, 6.6)
0.0 (0.0, 0.0)
0.0 (0.0, 0.0)
Find enchantment table
0.1 (0.0, 0.3)
0.0 (0.0, 0.0)
1.6 (1.2, 2.1)
0.0 (0.0, 0.0)
0.0 (0.0, 0.0)
Appendix
Table 7: Per-milestone success rates (%) on the Navigation benchmark. Values are mean (95% CI), computed via bootstrap over 2000 resamples.
Table 17: Model Architecture Hyperparameters (AI2-THOR)
Condition
Average Score
Absolute Decrease
CODE-SHARP
50.55
–
No OS
31.93
-18.62
No SR+OS
21.20
-29.35
No OE+SR+OS
13.50
-37.05
Appendix
Table 18: Ablation Impact on Average Score over Benchmarks
Figure 7: Detailed Ablation Results per Benchmark
Audit measure
Craftax-Classic
XLand
AI2-THOR
Exact duplicates
0.0%
0.0%
4.6%
Misspecified predicates
9.6%
0.0%
0.0%
Learnability rejection
7.1%
11.7%
18.9%
Appendix
Table 19: LLM judge audit. Novelty and correctness rates are computed over admitted skills; learnability rejection rates are computed over judge-accepted candidates.
Method
Source code
Mean
Median
ELLM
Full
0.398
0.081
OMNI
Full
0.392
0.158
CODE-SHARP
None
0.623
0.968
CODE-SHARP
Full
0.733
0.963
Appendix
Table 20: Craftax-Classic success rates with and without source-code access. Baselines retain full source-code access and use Qwen3-235B-A22B.
Method
Pre-training
Task training
Total steps
Success
CODE-SHARP
108
2×108
3×108
63.2%±2.14%
SCALAR
—
109
109
45%
Appendix
Table 21: Diamond-mining success after autonomous pre-training and goal-directed fine-tuning, compared with SCALAR’s reported task-specific training result.
Figure 8: Evolution of average SHARP complexity present in the skill archive