We introduce Anchor-and-Harvest Curriculum (AHC), a domain-informed, rule-based training system for Craftax-Full. AHC combines achievement-oriented tasks, harvested initial-state progress, difficulty adaptation, bounded selection, and original-task practice. It uses no LLM inference and keeps all 43 actions available. At 2,005,401,600 PPO rollout transitions, three primary AHC runs obtain native-task return 46.70±0.57 (sample SD), compared with 41.91±1.01 for fresh native PPO-GTrXL with the same student architecture and evaluation protocol. Evaluation uses ten stochastic rollouts on each of 1024 fixed worlds per model. An additional cohort of three AHC runs reaches 46.08±1.23. Two single-component ablations retain similar mean returns, while jointly freezing difficulty and removing inherited progress gives 45.14±2.79. All-67-achievement profiles show that most of the return difference comes from Intermediate achievements, while the deepest-floor achievements remain unsolved. A Skill-Preflight diagnosis with a 14B teacher shows how prerequisite and floor-access constraints restrict practice. AHC and three 14B diagnostic configurations reach aggregate returns near the reported 80B DiCode mean of 45.88 and below the 235B mean of 48.55. These results show that an explicit curriculum without LLM inference can reach returns comparable to those reported for LLM-generated curricula on Craftax-Full, without action masking.
Figures & tables
Enumerate tasks and task variants →
Select eligible units and one native anchor →
Initialize practice and set difficulty →
Train student and harvest progress
Figure 1: AHC’s data flow. The motivating diagnosis concerns restricted task access, missing deep practice, and limited information in aggregate return.
Item
Setting
Environment
Craftax-Symbolic-v1, Craftax 1.4.5
Budget
2,005,401,600 PPO rollout transitions (15300 updates); harvest and controller statistics reuse these rollouts; evaluation is excluded
Figure 2: The archive lock, drawn from the scheduler’s eligibility record (graph fingerprint cfbb1c9a4558 ). Eligibility under the true prerequisite graph, over five first-round arms’ end-of-run mastery profiles: an item is a legal target only when every prerequisite’s max-ever mastery clears the scheduler’s 0.30 line. enter_gnomish_mines is itself targetable but topped out at 0.20 against that line, and the 27 items behind it — of 67 in the graph, the three floor-2 cluster mobs among them — therefore remain ineligible under these measured profiles. One further under-line gate ( collect_sapphire ) additionally locks the two enchantments inside the same closure. The mastery profiles come from the first round of diagnostic runs.
Warrior (s1)
Recovery
Configuration
Graph and comparison scope
Tail
Final
Tail
Final
longStack
Original graph; floor clause on
4.54
6.74
0/3
0/3
repairF
Repaired graph; API-reference differs
14.36
16.11
3/3
3/3
shufGraph
Fixed shuffled graph; changes both graph consumers
14.97
16.02
3/3
3/3
Table 2: Graph configurations in the diagnostic record. Warrior is the s1 rate (%) averaged over the last ten checkpoints (tail) and at checkpoint 15300 (final). Recovery counts use three training runs and the four-skill boundary; they are identical under both readings. RepairF also changes the API-reference setting relative to longStack.
Figure 3: Per-session mine-entry success on the held-out clean channel for 21 diagnostic runs and two separately marked partner runs. Line style identifies the true, repaired, or shuffled prerequisite graph. No true-graph series reaches the 0.30 scheduler threshold; the closest is partner base2e9 s3 at 27.83%. Three altered-graph runs cross late: shufGraph s3 at session 134, repairF s3 at 137, and shufGraph s2 at 146.
Last ten checkpoints
Checkpoint 15300
Configuration
Return
Individual runs
Return
Individual runs
base15
43.929±1.309
45.016 / 42.476 / 44.296
44.509±1.780
46.024 / 42.549 / 44.955
longStack
44.014±0.795
44.726 / 43.157 / 44.159
44.589±0.909
45.016 / 43.545 / 45.206
Table 4: Diagnostic LLM curriculum comparison under two readings of the same evaluation summaries: the mean of the last ten checkpoints (primary) and the final checkpoint 15300, which is DiCode’s reporting convention. Uncertainty is sample SD across three runs. The checkpoint-15300 values are those compared with AHC in Table 36 ; the last-ten means are not used as comparators for AHC.
Last ten checkpoints
Checkpoint 15300
Skill
base15
longStack
base15
longStack
Mine entry
19.10±3.95
9.64±1.44
19.99±5.52
11.78±3.51
Defeat gnome warrior
15.12±3.64
1.55±2.59
16.67±4.83
2.25±3.89
Defeat gnome archer
8.96±3.15
0.094±0.164
9.96±3.98
0.065±0.113
Eat bat
14.08±3.63
0.50±0.45
15.07±5.01
0.81±0.75
Table 5: Capability profiles (rates in %; three runs per arm) over the last ten checkpoints (primary) and at checkpoint 15300. In the tail window, the smallest base15 value exceeds the largest longStack value for every listed skill. At checkpoint 15300 this holds for warrior, archer, and bat but not for mine entry, where base15 s2 reaches 13.96 against 14.94 for longStack s3. Per-run rates are in Tables 34 and 35 .
Observation
Design response
Some downstream targets are not directly schedulable
Use the graph for downstream value and explicit, separately specified eligibility/frontier rules.
Direct deep-floor practice can be restricted
Represent harvested floor progress through procedurally generated, floor-indexed worlds.
Difficult tasks can lose selection or parent opportunities
Admit fresh eligible units, mix rank priority with staleness, and adapt task difficulty.
Scalar return conceals skill differences
Monitor the original task and report raw held-out skill rates alongside return.
Table 6: Diagnostic observations and AHC design responses. AHC retains hard eligibility rules alongside its soft scores.
Task class
Target
Initialization and dynamics
Candidate qualification
Family L1/L2
Craft the recipe’s mapped achievement
Harvested signature on its recorded floor; recipe tools/resource support; both base labels have σ=0
Usable bank and equipment-chain check; no depth-neighbor gate
Depth L1/L2
Enter floor f∈{1,…,8}
Signature from f−1 , split by gate-progress bin g∈{0,3,6,8} ; L1 native dynamics, L2 starts at σ=1
Bank, neighbor unlock and official-feedback frontier cap
Ordinary achievement L0
One of the 47 non-recipe, non-entry achievements
Signature at its harvested floor; no base surgery/toolkit; σ=0
Bank and applicable controller checks
Family L4 (official-reset level)
Same twelve recipe achievements
Ordinary native reset; no harvested-state initialization or toolkit; σ=0
Requires an official-anchor reading; not eligible for fresh-unit priority
Table 7: Active base task definitions. L is an implementation label, f a physical floor index (surface f=0 ), and σ a dynamics parameter. Bank availability, prerequisite checks, retirement, and selection remain additional conditions; Appendix A.1 gives the exceptions.
Step
Operation
1
Initialize enumerated units, bank, world pool, per-unit statistics, and session RNG. Perform the 200-update initialization.
2
For each of 151 sessions, create scheduled task variants; retain units with supply, then apply unlocking, chain, frontier, and retirement rules.
3
Admit fresh eligible units first; fill the remaining 15-unit capacity by rank/staleness sampling without replacement. Append the original-task anchor.
4
Assemble curriculum states from harvested signatures and official worlds; apply each unit’s difficulty setting. Reweight and normalize environment shares.
5
Run 100 PPO updates. Harvest progress and collect conditional unit success and original-task achievement readings.
6
Update EMA statistics, difficulty, graduation, parking, and regression state; persist the bank/controller and proceed to the next session.
Table 8: AHC session procedure. The total number of transitions is fixed; selection and reweighting change their allocation.
Training cohort
GPU
Update 7600
Update 15300
Primary AHC
L40S
38.69±2.26
46.70±0.57
Additional AHC
A100
40.38±1.69
46.08±1.23
Fresh native PPO-GTrXL
RTX A6000
36.63±1.42
41.91±1.01
Table 9: Core internal comparison. Return is mean ± sample SD across three training seeds per cohort. Each checkpoint is evaluated on the same 1024 native worlds with ten stochastic rollouts per world and an 8192-step cap. Updates 7600 and 15300 correspond to 0.9961472B and 2.0054016B PPO rollout transitions, respectively.
Figure 4: Mean return for the three cohorts at the two evaluated PPO rollout budgets. Points are means across three training seeds and error bars are the across-seed sample SD; error-bar stems are offset horizontally for readability. Lines connect the two measured budgets as visual guides. Within each cohort, both points come from the same three training runs.
metric
seed 0
seed 1
seed 2
mean ± sample SD
mean return
46.945
47.105
46.051
46.700±0.568
mean performance (%)
20.772
20.843
20.377
20.664±0.251
average episode length
1451.2
1583.3
1355.3
1463.2±114.5
enter_gnomish_mines (%)
28.281
27.500
23.369
26.383±2.640
defeat_gnome_warrior (%)
21.504
21.738
17.510
20.251±2.377
defeat_gnome_archer (%)
13.682
14.160
13.555
13.799±0.319
Table 10: AHC at the fixed checkpoint-15300 endpoint, using 1024 fixed worlds and ten stochastic rollouts per world (10,240 episodes/model). Training seeds are 0–2; the world-generation seed is 0. The final column gives the across-seed mean and sample SD. Skill rates are percentages; performance is return as a percentage of the 226-point maximum. Evaluation uses the unfiltered 43-action policy.
Training GPU
Cohort
Seed
Return
Boundary
L40S
Primary
0
46.861
Yes
L40S
Primary
1
47.268
Yes
L40S
Primary
2
46.134
Yes
A100
Additional
1
45.205
No
A100
Additional
4
47.269
Yes
A100
Additional
5
44.951
No
Table 11: Original single-rollout results: one episode on each of 1024 worlds with an 8192-step cap. The L40S rows form the primary cohort; the A100 rows are an additional cohort. Boundary marks retention of the four-skill thresholds in § 2 . Seed 4 resumed training from update 15000 to 15300.
Model
Budget (B)
Return
I-18
Entry
Warrior
W ∣ entry
AHC, checkpoint 7600 (L40S)
0.996
38.69±2.26
34.84±3.58
7.57±1.95
1.51±2.48
23.5±39.0
AHC, checkpoint 7600 (A100)
0.996
40.38±1.69
37.22±2.48
8.05±4.22
1.30±2.23
19.0±32.8
Fresh native PPO, update 7600
0.996
36.63±1.42
32.62±3.06
0.48±0.34
0
0
Native PPO, update 7000 (guard-on)
0.918
36.24±2.42
31.86±4.43
0.47±0.36
0.003±0.006
0.4±0.7
Native PPO, update 8000 (guard-on)
1.049
38.23±1.18
35.49±1.88
0.74±0.35
0
0
L40S AHC, checkpoint 15300
2.005
46.70±0.57
48.18±0.87
26.38±2.64
20.25±2.38
76.7±2.1
Table 12: Repeated-world evaluation with ten rollouts on each of 1024 fixed worlds (10,240 episodes per model). Each row is mean ± sample SD over three training seeds. I-18 is the unweighted macro success rate across the 18 official Intermediate achievements. Other rates are episode percentages. W ∣ entry averages each seed’s conditional success rate. Guard-stopped native-PPO endpoints are historical checkpoints from the original runs; full-budget native PPO is a separate fresh run set. Training GPUs: L40S for primary AHC, A100 for A100 AHC, and RTX A6000 for D4 and native-PPO runs.
Cohort
Total
w=1 (25)
w=3 (18)
w=5 (15)
w=8 (9)
Residual
L40S AHC
4.795
1.115
3.615
0.072
0
−0.007
A100 AHC
4.171
1.128
2.854
0.187
0
0.002
D4-A
4.196
1.046
3.038
0.116
0
−0.004
D4-H
4.512
0.527
3.955
0.021
0
0.009
D4-AH
3.230
0.696
2.508
0.021
0
0.004
Table 13: Native-weight decomposition of the full-budget mean-return gap relative to fresh native PPO. Contributions are in return points. Each cohort has three seeds.
Arm
GPU
Return
Lowest
Highest
M4 (%)
O14 (%)
Full AHC, primary
L40S
46.70±0.57
46.05
47.10
19.86±1.47
56.28±0.70
D4-A: fixed difficulty
A6000
46.10±1.18
45.22
47.45
15.81±6.51
56.06±0.77
D4-H: no progress init.
A6000
46.42±1.55
44.73
47.78
18.64±9.48
57.43±1.90
D4-AH: joint
A6000
45.14±2.79
42.68
48.17
11.51±15.30
56.03±2.10
Full AHC, additional
A100
46.08±1.23
45.33
47.50
11.98±11.02
56.71±0.63
Table 14: Full-budget component comparisons under the same repeated-world evaluation. Each arm has three training seeds; return, M4, and O14 show mean ± sample SD. Lowest and highest are the observed per-seed return means, each estimated from 10,240 episodes.
Training seed
Elapsed (hh:mm:ss)
Allocated L40S GPU ⋅ h
0
17:50:18
17.8383
1
17:33:14
17.5539
2
17:37:21
17.6225
Completed-job total
53:00:53
53.0147
Table 15: Allocation time for the three primary AHC training jobs, including in-job evaluation. Each job used one L40S, so GPU ⋅ h equals elapsed hours.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Part
Fields stored
Bounds / assembly
f
player_level
Selects floor-indexed pool; variant floors are not overridden.
Never copy these from the harvested state; derive already-earned achievement flags.
Appendix
Table 16: Stored signature fields and realization semantics. Clamps limit stored quantities, not fixed amounts granted to the student.
Step
Operation
1
For selected unit u , obtain entries from its bank or variant-parent bank. Official-practice units obtain no entries. Skip entries without a signature.
2
Set S=(f,g,i,a,e) ; for D4-H (and D4-AH), replace it by (0,(),(),(),()) . Set f∗=f ; variant floors remain unspecified in the resolved configuration.
3
Draw W from the official pool at f∗ ; skip if unavailable. Clone W . Its map, actors, position, timestep and vitals are not inherited.
4
Copy each nonempty signature field g,i,a,e , preserving engine dtypes. Copy array prefixes; merge variant scalar prefill as max(signature value,v) , with omitted signature scalars defaulting to zero.
5
Ensure goal-recipe ore supply, then configured lighting. Apply explicit variant softening, otherwise the unit’s sigma specification. Non-axis units have σ=0 (native).
6
Recompute inventory-derived achievement bits; mark entered floors from f∗ . Require the goal bit to remain false. Attach u and build observation.
Appendix
Table 17: State-assembly pseudocode for the active signature path. All unspecified state fields come from the generated world until an explicit task operator changes them. The original-task anchor bypasses curriculum assembly.
Figure 5: All-67 capability differences, part 1: native achievement IDs 0–33. Cells are cohort mean rates minus fresh native PPO in percentage points at update 15300; brackets give native reward weights. All cells use three seed means, each based on 1024 fixed worlds and ten repeats. Zero after rounding need not mean exact equality.
Figure 6: All-67 capability differences, part 2: native achievement IDs 34–66, with the same scale and definitions as Figure 5 . The nine weight-8 achievements have zero successes in every cohort; zero differences alone would not distinguish shared failure from shared mastery.
Task UID
Seed 0
Seed 1
Seed 2
fam:iron_pickaxe|L1|i0 : selected sessions
24
29
34
First / last selection
2 / 144
2 / 148
2 / 148
depth:f1g8|L2 : selected sessions
7
12
32
First / last selection
2 / 21
2 / 23
2 / 54
Appendix
Table 18: Examples of observed task selection in the joint arm. Counts refer to sessions with positive configured task probability; first/last are session indices, not a continuous active interval.
Training seed
Units in export
Variants
Selected
First exit
0
191
44
132
82
1
203
56
145
84
2
200
53
144
82
Appendix
Table 19: Scope of recovered primary-AHC task records. Units include 147 base units and the listed variants. “Selected” counts distinct units with at least one logged seat assignment; “First exit” counts distinct units with a first recorded graduation or parking event.
Cohort
Runs
Cases
Retained updates
Primary L40S AHC
3
6
7600, 15300
Additional A100 AHC
3
6
7600, 15300
Fresh native PPO
3
6
7600, 15300
D4-A / D4-H / D4-AH
3 each
9
15300
Historical guard-stopped PPO
3
9
7000, 8000, last accepted endpoint
Additional A100 D4-A
1
1
15300
Appendix
Table 20: Repeated-world checkpoint inventory. Every case has 10,240 episode records.
Training GPU
Seed
Return
Mine entry
Warrior
Archer
Bat
Remainder
L40S
0
46.861
28.52
22.36
14.45
18.95
44.33
L40S
1
47.268
28.03
21.78
13.28
19.73
44.78
L40S
2
46.134
23.73
18.16
14.94
18.95
43.86
A100
1
45.205
21.78
0.00
0.00
0.00
44.55
A100
4
47.269
29.00
23.54
18.07
22.36
44.48
A100
5
44.951
19.53
1.66
0.29
0.88
44.28
Appendix
Table 21: Deep-floor success rates (%) for the six L40S primary and A100 additional runs at checkpoint 15300 under the original single-rollout protocol. The Remainder column subtracts the four achievements’ contribution (three points times each success rate) from the return.
System
Training seed
Return
Mine entry
Warrior
Archer
Bat
AHC
0
46.860676
28.515625
22.363281
14.453125
18.945312
AHC
1
47.267670
28.027344
21.777344
13.281250
19.726562
AHC
2
46.134350
23.730469
18.164062
14.941406
18.945312
base15
1
46.023933
24.804688
21.093750
14.550781
20.507812
base15
2
42.548748
13.964844
11.523438
7.714844
10.644531
base15
3
44.954964
21.191406
17.382812
7.617188
14.062500
Appendix
Table 22: Inputs to the common-checkpoint comparison. These are single checkpoint-15300 values, not tail-window means. Skill rates are percentages; six decimal places are shown and JSON precision is retained for aggregation.
Model
Seed
Update
Return
SE
I-18
M4
O14
Mine n
W n (%entry)
L40S AHC
0
15300
46.945
0.189
48.70
20.62
56.72
2896
2202 (76.0)
L40S AHC
1
15300
47.105
0.190
48.67
20.79
56.64
2816
2226 (79.0)
L40S AHC
2
15300
46.051
0.198
47.17
18.16
55.46
2393
1793 (74.9)
L40S AHC
0
7600
39.924
0.187
36.92
2.52
46.74
1006
10 (1.0)
L40S AHC
1
7600
40.065
0.190
36.89
1.66
46.96
666
7 (1.1)
L40S AHC
2
7600
36.078
0.237
30.70
4.54
38.18
654
448 (68.5)
Appendix
Table 23: Per-seed repeated-world results (10,240 episodes per model). Return SE is world-clustered within each run, not a training-seed SD. I-18, M4, and O14 are macro-averages (%) over the 18 official Intermediate achievements, the four M4 achievements, and the other 14. Mine n and W n (%entry) are episode counts; the parenthetical value is the warrior success percentage among mine-entry episodes. L40S and A100 AHC denote distinct training cohorts. Fresh native PPO is separate from the historical guard-on PPO runs.
Current matched cohorts
Achievement
M4
L40S AHC 7600
L40S AHC 15300
A100 AHC 7600
A100 AHC 15300
Fresh PPO 7600
Fresh PPO 15300
collect_sapphire
12.22
21.07
13.46
21.62
11.10
18.23
collect_ruby
12.04
21.28
13.53
21.65
11.31
17.96
make_diamond_pickaxe
1.76
3.18
2.01
3.26
1.51
2.56
make_diamond_sword
1.92
4.06
2.17
4.23
1.65
2.95
make_iron_armour
15.71
30.98
18.25
30.33
4.33
8.02
Appendix
Table 24: Success rates (%) for the 18 official Intermediate achievements. Each cell is an equal-weight mean over three training seeds. The upper panel gives six current matched cohorts; PPO denotes fresh native GTrXL runs. The lower panel gives the three D4 intervention arms and historical guard-on PPO cohorts, which are separate training cohorts from the upper panel. D4-AH is the fresh joint intervention. M4 marks the four deep-floor achievements. The source key defeat_orc_solider retains its original spelling.
Training seed
Update 7600: 996,147,200 transitions
Update 15300: 2,005,401,600 transitions
0
Present; 12 files
Present; 12 files
1
Present; 12 files
Present; 11 files
2
Present; 11 files
Present; 11 files
Appendix
Table 25: Archived AHC checkpoints. Entries count files covered by the supplied checksum manifest, not independent models or evaluations. There are three training executions and two checkpoints per execution.
Parameter
Value
GTrXL layers / attention heads
2 / 8
Embedding / QKV / hidden width; activation
256 / 256 / 256; ReLU
Transformer gating; gate bias
Enabled; 2.0
Memory / gradient window (steps)
128 / 64
Task conditioning; action set
67-dimensional achievement indicator; 43 actions
Parallel environments / rollout steps
1024 / 128
Appendix
Table 26: Shared recorded training settings. These are training parameters; the frozen evaluation uses its separately specified protocol.
Arm
Difficulty A
Progress H
Task / completion reward
Entropy stop
Full AHC
Adaptive σ
Enabled
Curriculum + native anchor; +20 on curriculum success
0.15
D4-A
Fixed native dynamics
Enabled
Same task interface
0.15
D4-H
Adaptive σ
Neutral floor-zero signature
Same task interface
0.15
D4-AH
Fixed native dynamics
Neutral floor-zero signature
Same task interface
0.15
Fresh native PPO
Native dynamics
Native resets
Original task only; native reward
Disabled
Appendix
Table 27: Controlled training differences. A/H abbreviate difficulty annealing and harvested-progress initialization. Removing H retains per-unit task operators and variant prefills. Historical guard-stopped PPO is a separate cohort.
Seed
Return
Performance (%)
Mean episode length
Training RTX A6000 GPU ⋅ h
0
41.843
18.514706
1337.300
22.4089
1
40.668
17.994696
1351.456
22.9050
2
42.068
18.614149
1159.646
20.8015
Appendix
Table 28: Single-rollout measurements for the three guard-stopped PPO endpoints (first episode on each of 1024 worlds). Performance is return as a percentage of 226. Training time is elapsed time on one RTX A6000, including in-run evaluation and rejected sessions.
Intervention
Seed
Return
Performance (%)
Episode length
Fixed difficulty
0
47.812920
21.156158
1462.630
Fixed difficulty
1
45.908348
20.313429
1553.788
Fixed difficulty
2
45.132290
19.970039
1457.445
No progress initialization
0
46.757309
20.689075
2115.051
No progress initialization
1
44.585232
19.727980
1216.120
No progress initialization
2
48.070557
21.270159
1616.744
Appendix
Table 29: D4 final-checkpoint results by training seed. These are individual run means over 1024 first episodes. Performance is return as a percentage of the 226-point maximum.
Training seed
Mine entry
Warrior
Archer
Bat
0
22.4
0.1
0.1
0.0
1
41.9
36.8
29.9
36.4
2
34.7
26.8
15.3
25.2
Reported mean
33.0
21.2
15.1
20.5
Appendix
Table 30: DiCode reproduction capability profile at checkpoint 15300, transcribed from the collaborator summary. Rates are percentages; the last row is the supplied across-seed mean.
Arm
Graph change
Scheduler, gate, and prompt change
longStack
Original 67-node, 94-edge graph
Prerequisite scheduler; R3 floor clause enabled
repairF
Replace mine-entry prerequisite with dungeon entry for warrior, archer, and bat only
Scheduler and R3 use repaired graph; mastered-prerequisite exemption retained; API-reference content differs
shufGraph
Fixed double-edge rewiring, seed 20260804; 79 of 94 original edges replaced
Scheduler and R3 use shuffled graph; target identities and scaffold checks both change
Surgical
Original graph retained
Only floor-granted achievements removed from R3 scaffold checks; premark/inventory channels and R1/R2 retained
Appendix
Table 31: Diagnostic interventions relative to longStack. All three training seeds in each arm use the same arm definition. This historical diagnostic comparison is separate from the AHC component ablations.
Figure 7: Mastered-prerequisite exemption timing in the three repairF runs. The blue curve is the clean-channel enter_dungeon success rate. Black lines mark s∗ , the first of three consecutive readings at or above 70% (the 0.7 criterion): sessions 27, 16, and 19, respectively. Orange dots give the proportion of cluster-mob levels whose generated code starts on floor 2 or deeper, among levels with a nonempty training-performance history. Levels target gnome warriors, gnome archers, or bats and are grouped by their creation session, rather than by the session when they were trained. Dot size increases with the number of counted levels and is capped. Orange segments pool level counts before ( s<s∗−2 ) and after ( s>s∗+2 ), excluding the inclusive s∗±2 band. Before/after numerator–denominator pairs are 0/1 and 166/253, 2/7 and 142/269, and 1/1 and 37/180.
Seed
Updates
Valid transitions
Return
Mine entry
Warrior
Archer
Bat
0
14600
1,913,651,200
42.086
7.266
0
0
0.029
1
15000
1,966,080,000
40.249
2.275
0
0
0
2
13600
1,782,579,200
41.867
4.648
0
0.020
0.029
Appendix
Table 32: Original native-PPO runs stopped by the entropy guard, evaluated with ten rollouts on each of 1024 worlds. Updates and PPO rollout transitions describe the last valid checkpoint; each transition count equals updates ×1024×128 . Skill rates are percentages of 10,240 episodes. Every run stops short of the 2,005,401,600-transition target.
System
seed 0
seed 1
seed 2
Mean ± sample SD
AHC (single rollout)
46.861
47.268
46.134
46.754±0.574
DiCode (collaborator reproduction)
46.74
50.93
48.64
48.77±2.10
Appendix
Table 33: Three-seed endpoint returns on the reported common 1024-world evaluation. AHC values come from the original single-rollout JSONs; DiCode values and its sample SD are reported at the supplied summary’s precision.
configuration
seed
mines
warrior
archer
eat_bat
shufGraph
s1
19.06
14.97
11.00
15.05
s2
26.94
22.00
13.89
21.02
s3
26.33
21.13
13.50
19.38
repairF
s1
19.73
14.36
5.15
5.83
s2
23.37
18.17
12.66
15.78
s3
29.30
24.06
18.73
22.67
Appendix
Table 34: Per-seed last-10 means (%) for the four second-floor skills. Supports the base15–longStack separation and the recovery counts; other recorded configurations are included for completeness.
configuration
seed
return
last-ten
mines
warrior
archer
eat_bat
shufGraph
s1
45.480
45.661
20.21
16.02
12.11
15.62
s2
47.652
46.520
29.00
25.10
16.60
23.83
s3
47.848
47.041
29.98
24.41
15.23
22.75
repairF
s1
44.908
44.368
22.85
16.11
4.88
7.23
s2
45.980
46.270
22.36
16.21
11.82
15.33
s3
48.484
47.328
33.50
27.44
21.39
26.37
Appendix
Table 35: Per-run readings at checkpoint 15300 from the same evaluation summaries as Table 34 : return in points (with the last-ten mean beside it) and the four second-floor skill rates in %. Recovery labels are unchanged from the tail window.
System
Return
Mine entry
Warrior
Archer
Bat
AHC
46.754±0.574
26.758±2.633
20.768±2.274
14.225±0.853
19.206±0.451
base15
44.509±1.780
19.987±5.519
16.667±4.825
9.961±3.975
15.072±5.008
longStack
44.589±0.909
11.784±3.508
2.246±3.890
0.065±0.113
0.814±0.746
Appendix
Table 36: Historical single-rollout final-endpoint comparison, recomputed from existing checkpoint-15300 JSON entries. Each cell is mean ± sample SD over three training runs. All rows use Craftax 1.4.5, evaluation seed 0, and 1024 environments with one rollout per world and an 8192-step cap. The AHC row is the original single-rollout result, not Table 10 ’s repeated-world estimate. Skill columns are percentages.
Model
Budget (B)
LLM
Mask
Return
Near 1B transitions
PPO-GTrXL (maintainer board)
1.000
No
None
41.36
Near 2B transitions
PPO-GTrXL (DiCode report)
2.000
No
None
41.59
DiCode 30B (reported)
2.000
Yes
Not audited
44.38±0.66
DiCode 80B (reported)
2.000
Yes
Not audited
45.88±0.40
Appendix
Table 37: External and historical numerical context, separate from the internal two-budget comparison in Table 9 . External results use different software versions, evaluation protocols, and time caps; DiCode uncertainties are standard errors ( Mitsides et al., 2026 ) . The 14B diagnostic panel uses one rollout on each of 1024 worlds: its checkpoint-15300 mean ± SD is followed by the last-ten mean in parentheses. The other two diagnostic endpoint means are base2e9 45.16 and gateOff 43.85 (Appendix A.18 ).
Comparison
Difference
Welch 95% interval
L40S AHC − fresh PPO, update 7600
2.06
[−2.56,6.68]
A100 AHC − fresh PPO, update 7600
3.76
[0.18,7.33]
L40S AHC − fresh PPO, update 15300
4.80
[2.71,6.88]
A100 AHC − fresh PPO, update 15300
4.17
[1.58,6.76]
D4-A − fresh PPO, update 15300
4.20
[1.67,6.72]
D4-H − fresh PPO, update 15300
4.51
[1.35,7.67]
Appendix
Table 38: Return differences and Welch 95% intervals from three training-seed means in each cohort, using the ten-repeat evaluator. The final three rows compare against historical guard-stopped endpoints at their own budgets.
Automatic curriculum learning can improve the effectiveness of reinforcement learning by selecting the training experiences presented to the agent over time. Predicting the consequences of such decisions can, however, be difficult. We analyze automatic curriculum learning as a sequential decision-making problem, highlighting a gap between the quantities that determine the value of curriculum decisions and the information captured by local learning signals commonly used to guide them. We then investigate whether Large Language Models (LLMs) can exploit richer information about the learning problem to better anticipate the consequences of curriculum decisions. We introduce a method that combines online learning-progress estimates with LLM-informed estimates of (i) the potential downstream benefits of learning on each task and (ii) whether direct training on a task is currently likely to produce progress. We evaluate the approach on a custom benchmark of 256 textual goals in Craftax under different curriculum objectives. We observe the strongest gains when optimizing for individual target tasks. When optimizing across the full task set, the benefits vary across learners with different mechanisms for cross-task transfer, ranging from modest improvements in learning speed to larger gains that persist through the end of training.
LM-based agents excel when given high-level action APIs but struggle to ground language into low-level control. Prior work has LLMs generate skills or reward functions for RL, but these one-shot approaches lack feedback to correct specification errors. We introduce SCALAR, a bidirectional framework coupling LLM planning with RL through a learned skill library. The LLM proposes skills with preconditions and effects; RL trains policies for each skill and feeds back execution results to iteratively refine specifications, improving robustness to initial errors. Pivotal Trajectory Analysis corrects LLM priors by analyzing RL trajectories; Frontier Checkpointing optionally saves environment states at skill boundaries to improve sample efficiency. On Craftax, SCALAR achieves 88.2% diamond collection, a 1.9x improvement over the best baseline, and reaches the Gnomish Mines 9.1% of the time where prior methods fail entirely.
Renos Zabounidis, Yue Wu, Simon Stepputtis +4
School of Computer Science, Carnegie Mellon University · Department of Mechanical Engineering, Virginia Tech
Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.