CODE-SHARP: Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs
Organizations: Imperial College London · Sony Interactive Entertainment
Abstract
A core quality of general intelligence is the ability to open-endedly expand and evolve its set of mastered skills autonomously. While recent Foundation Model (FM) driven approaches have shown promising results towards this goal, they typically rely on significant human-in-the-loop engineering, limiting their transferability to novel environments. To address this, we introduce Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs (CODE-SHARP), a framework that leverages FMs to open-endedly grow and evolve an archive of Python programs encoding skills to train a generalist agent policy entirely from scratch via reinforcement learning, directly from source code. These programs, termed Skills as Hierarchical Reward Programs (SHARPs), each encode a local success condition and a set of prerequisites delegated to previously discovered SHARPs. At runtime, SHARPs dynamically route the agent through their prerequisite chain based on the current state, rewarding each completion along the way, requiring the agent to learn only the marginal behaviour each new SHARP introduces, enabling efficient learning of long-horizon skills without any pre-defined rewards. On Craftax-Classic and XLand-MiniGrid, agents trained fully autonomously by CODE-SHARP outperform previous works by 6x and 2.7x in median performance and are the only agents capable of crafting iron tools and mining diamonds. On AI2-THOR, CODE-SHARP achieves 66.7% median success versus 0% for ELLM and OMNI, using only the initial environment state instead of the full source code. Scaled to Craftax-Extended, CODE-SHARP trains a generalist agent on over 90 discovered SHARPs, enabling the agent to solve challenging long-horizon tasks zero-shot, matching agents trained on ground-truth rewards.
Figures & tables
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Achievement | CODE-SHARP (Ours) | CODE-FRP (Ours) | ELLM | OMNI |
|---|---|---|---|---|
| Place Table | 1.000 | 0.998 | 0.971 | 0.989 |
| Collect Sapling | 0.999 | 0.998 | 0.994 | 0.044 |
| Make Wood Pickaxe | 0.999 | 0.994 | 0.727 | 0.779 |
| Make Wood Sword | 0.999 | 0.800 | 0.230 | 0.965 |
| Place Furnace | 0.998 | 0.575 | 0.011 | 0.273 |
| Collect Stone | 0.998 | 0.200 | 0.097 | 0.699 |
| Achievement | CODE-SHARP (Ours) | CODE-FRP (Ours) | ELLM | OMNI |
|---|---|---|---|---|
| Pick Up Square Red | 0.978 | 0.683 | 0.908 | 0.493 |
| Pick Up Ball Green | 0.969 | 0.479 | 0.877 | 0.552 |
| Pick Up Ball Blue | 0.962 | 0.576 | 0.885 | 0.396 |
| Pick Up Square Purple | 0.959 | 0.346 | 0.848 | 0.163 |
| Pick Up Square Green | 0.953 | 0.446 | 0.864 | 0.408 |
| Pick Up Ball Red | 0.938 | 0.454 | 0.889 | 0.636 |
| Depth | # Ach. | CODE-SHARP (OURS) | CODE-FRP (OURS) | ELLM | OMNI |
|---|---|---|---|---|---|
| 1 | 32 | 0.903 | 0.893 | 0.611 | 0.342 |
| 2 | 37 | 0.485 | 0.314 | 0.048 | 0.218 |
| 3 | 23 | 0.342 | 0.130 | 0.009 | 0.029 |
| 4 | 1 | 0.125 | 0.000 | 0.000 | 0.000 |
| 5 | 3 | 0.000 | 0.000 | 0.000 | 0.000 |
| 6 | 2 | 0.083 | 0.000 | 0.000 | 0.000 |
| Milestone | CODE-SHARP | CODE-FRP | Finetuned | task-specific | ReAct |
|---|---|---|---|---|---|
| Craft wood pickaxe | 92.6 (80.9, 98.8) | 79.0 (54.6, 96.6) | 98.7 (98.3, 99.0) | 99.0 (98.8, 99.2) | 100.0 (100.0, 100.0) |
| Craft stone pickaxe | 87.5 (65.9, 98.6) | 63.5 (38.7, 85.4) | 96.0 (95.3, 96.9) | 97.7 (96.9, 98.4) | 88.9 (66.7, 100.0) |
| Craft iron pickaxe | 75.8 (51.7, 93.0) | 17.6 (1.7, 39.6) | 84.8 (83.5, 86.1) | 0.0 (0.0, 0.0) | 33.3 (0.0, 66.7) |
| Craft iron sword | 69.7 (48.5, 86.4) | 11.7 (0.0, 28.7) | 65.9 (64.1, 67.9) | 0.0 (0.0, 0.0) | 11.1 (0.0, 33.3) |
| Craft iron armour | 34.9 (21.2, 48.6) | 3.4 (0.0, 8.5) | 12.2 (10.6, 14.0) | 0.0 (0.0, 0.0) | 0.0 (0.0, 0.0) |
| Overall Score | 66.52 (48.17, 79.96) | 16.43 (4.77, 29.09) | 58.08 (56.33, 60.00) | 5.29 (5.28, 5.31) | 10.80 (4.91, 18.73) |
| Milestone | CODE-SHARP | CODE-FRP | Finetuned | task-specific | ReAct |
|---|---|---|---|---|---|
| Craft wood pickaxe | 98.5 (98.1, 98.9) | 96.4 (95.1, 97.5) | 98.7 (98.3, 98.9) | 98.7 (98.1, 99.1) | 100.0 (100.0, 100.0) |
| Craft stone pickaxe | 98.3 (97.7, 98.7) | 81.8 (75.4, 87.7) | 96.7 (96.1, 97.4) | 98.4 (97.7, 98.9) | 88.9 (66.7, 100.0) |
| Craft torches | 97.6 (96.7, 98.4) | 73.7 (65.9, 80.3) | 96.2 (95.4, 97.0) | 97.4 (96.4, 98.3) | 77.8 (44.4, 100.0) |
| Descend to dungeon | 89.1 (84.2, 93.4) | 34.7 (28.2, 41.9) | 81.7 (80.3, 83.2) | 34.5 (28.9, 40.3) | 11.1 (0.0, 33.3) |
| Kill 8 dungeon monsters | 69.6 (58.4, 81.3) | 8.9 (4.8, 13.4) | 57.4 (54.1, 60.7) | 0.0 (0.0, 0.0) | 11.1 (0.0, 33.3) |
| Descend to gnomish mines | 47.3 (35.4, 59.6) | 2.6 (1.8, 3.3) | 41.6 (38.3, 44.9) | 0.0 (0.0, 0.0) | 11.1 (0.0, 33.3) |
| Milestone | CODE-SHARP | CODE-FRP | Finetuned | task-specific | ReAct |
|---|---|---|---|---|---|
| Craft wood pickaxe | 98.3 (97.0, 99.2) | 93.8 (88.7, 97.2) | 99.0 (98.7, 99.4) | 99.2 (98.8, 99.6) | 100.0 (100.0, 100.0) |
| Craft stone pickaxe | 98.0 (96.4, 99.0) | 82.0 (74.3, 89.3) | 98.4 (97.7, 99.0) | 99.0 (98.4, 99.6) | 88.9 (66.7, 100.0) |
| Craft stone sword | 97.8 (96.1, 99.0) | 76.2 (65.6, 86.3) | 97.7 (96.8, 98.6) | 99.0 (98.4, 99.6) | 88.9 (66.7, 100.0) |
| Craft torches | 97.0 (95.0, 98.4) | 69.4 (57.4, 79.8) | 97.6 (96.6, 98.5) | 98.8 (98.0, 99.6) | 77.8 (44.4, 100.0) |
| Craft arrows | 96.8 (95.1, 98.3) | 67.4 (56.5, 77.7) | 97.3 (96.2, 98.4) | 98.8 (98.0, 99.6) | 77.8 (44.4, 100.0) |
| Collect sapling | 96.8 (95.0, 98.3) | 67.3 (56.9, 77.9) | 93.6 (92.0, 95.3) | 98.8 (98.0, 99.6) | 33.3 (0.0, 66.7) |
| Milestone | CODE-SHARP | CODE-FRP | Finetuned | task-specific | ReAct |
|---|---|---|---|---|---|
| Descend to dungeon | 86.2 (80.6, 91.1) | 36.5 (26.0, 46.1) | 92.1 (91.2, 93.0) | 10.3 (9.4, 12.1) | 33.3 (0.0, 66.7) |
| Kill 8 dungeon monsters | 78.9 (72.0, 85.1) | 10.2 (4.0, 18.4) | 56.9 (52.0, 61.2) | 0.0 (0.0, 0.0) | 11.1 (0.0, 33.3) |
| Descend to gnomish mines | 59.8 (50.6, 68.8) | 5.0 (1.4, 9.5) | 38.9 (36.2, 41.5) | 0.0 (0.0, 0.0) | 11.1 (0.0, 33.3) |
| Kill 8 gnomish mines monsters | 12.6 (4.9, 22.4) | 0.0 (0.0, 0.0) | 9.7 (8.2, 11.4) | 0.0 (0.0, 0.0) | 0.0 (0.0, 0.0) |
| Descend to sewers | 0.9 (0.3, 1.6) | 0.0 (0.0, 0.0) | 5.6 (4.7, 6.6) | 0.0 (0.0, 0.0) | 0.0 (0.0, 0.0) |
| Find enchantment table | 0.1 (0.0, 0.3) | 0.0 (0.0, 0.0) | 1.6 (1.2, 2.1) | 0.0 (0.0, 0.0) | 0.0 (0.0, 0.0) |
| Hyperparameter | Value |
|---|---|
| Proposal Iterations | 25 |
| Training Epochs per Iteration | 4 |
| Mutation Start Iteration | 7 |
| Proposal Evaluation Epochs | 4 |
| Skill Proposals per Iteration | 10 |
| Mutation Proposals per Iteration | 5 |
| Hyperparameter | Value |
|---|---|
| Proposal Iterations | 40 |
| Training Epochs per Iteration | 5 |
| Mutation Start Iteration | 20 |
| Proposal Evaluation Epochs | 5 |
| Skill Proposals per Iteration | 10 |
| Mutation Proposals per Iteration | 5 |
| Hyperparameter | Value |
|---|---|
| Learning Rate | |
| Batch Size | 512 |
| Optimizer | AdamW |
| Discount Factor ( ) | 0.99 |
| Entropy Coefficient | 0.01 |
| Clip Range ( ) | 0.2 |
| Hyperparameter | Value |
|---|---|
| Architecture | Transformer-XL |
| Number of Layers | 2 |
| Number of Attention Heads | 8 |
| Embedding Dimension ( ) | 256 |
| QKV Dimension ( ) | 256 |
| MLP Hidden Size | 256 |
| Hyperparameter | Value |
|---|---|
| Proposal Iterations | 100 |
| Training Epochs per Iteration | 10 |
| Mutation Start Iteration | 15 |
| Proposal Evaluation Epochs | 5 |
| Skill Proposals per Iteration | 10 |
| Mutation Proposals per Iteration | 5 |
| Hyperparameter | Value |
|---|---|
| Start Learning Rate | |
| Learning Rate Decay | Linear |
| End Learning Rate | |
| Batch Size | 1024 |
| Optimizer | AdamW |
| Discount Factor ( ) | 0.99 |
| Hyperparameter | Value |
|---|---|
| Architecture | Transformer-XL |
| Number of Layers | 2 |
| Number of Attention Heads | 8 |
| Embedding Dimension ( ) | 256 |
| QKV Dimension ( ) | 256 |
| MLP Hidden Size | 256 |
| Hyperparameter | Value |
|---|---|
| Proposal Iterations | 100 |
| Mutation Start Iteration | 25 |
| Proposal Evaluation Epochs | 5 |
| Skill Proposals per Iteration | 10 |
| Mutation Proposals per Iteration | 5 |
| Sampled Skills for Mutation per Iteration | 1 |
| Hyperparameter | Value |
|---|---|
| Start Learning Rate | |
| Learning Rate Decay | None |
| End Learning Rate | |
| Batch Size | 4096 |
| Optimizer | Adam |
| Discount Factor ( ) | 0.99 |
| Hyperparameter | Value |
|---|---|
| Architecture | CNN + FiLM + LSTM |
| Number of Conv Layers | 3 |
| Conv Channels | 32 |
| Number of FiLM Blocks | 2 (BatchNorm) |
| LSTM Hidden Size | 128 |
| Task Embedding Dimension ( ) | 128 |
| Condition | Average Score | Absolute Decrease |
|---|---|---|
| CODE-SHARP | 50.55 | – |
| No OS | 31.93 | -18.62 |
| No SR+OS | 21.20 | -29.35 |
| No OE+SR+OS | 13.50 | -37.05 |
| Audit measure | Craftax-Classic | XLand | AI2-THOR |
|---|---|---|---|
| Exact duplicates | 0.0% | 0.0% | 4.6% |
| Misspecified predicates | 9.6% | 0.0% | 0.0% |
| Learnability rejection | 7.1% | 11.7% | 18.9% |
| Method | Source code | Mean | Median |
|---|---|---|---|
| ELLM | Full | 0.398 | 0.081 |
| OMNI | Full | 0.392 | 0.158 |
| CODE-SHARP | None | 0.623 | 0.968 |
| CODE-SHARP | Full | 0.733 | 0.963 |
| Method | Pre-training | Task training | Total steps | Success |
|---|---|---|---|---|
| CODE-SHARP | ||||
| SCALAR | — |
Agent Skill Learning
Agent skill learning develops reusable procedures and behaviors that help AI agents solve new tasks and adapt from experience.
See all 185 papers