Anchor-and-Harvest Curriculum: A Fully Unmasked Non-LLM Curriculum for Craftax

Jiayu Zhu; Luyuan Li; Yuyuan Shi; Hanyi Dong

cs.LGOct 11, 2026

Anchor-and-Harvest Curriculum: A Fully Unmasked Non-LLM Curriculum for Craftax

Authors: Jiayu Zhu, Luyuan Li, Yuyuan Shi, Hanyi Dong

Organizations: Brown University · York University · Sun Yat-sen University · The Chinese University of Hong Kong, Shenzhen

Abstract

We introduce Anchor-and-Harvest Curriculum (AHC), a domain-informed, rule-based training system for Craftax-Full. AHC combines achievement-oriented tasks, harvested initial-state progress, difficulty adaptation, bounded selection, and original-task practice. It uses no LLM inference and keeps all 43 actions available. At 2,005,401,600 PPO rollout transitions, three primary AHC runs obtain native-task return $46.70\pm0.57$ (sample SD), compared with $41.91\pm1.01$ for fresh native PPO-GTrXL with the same student architecture and evaluation protocol. Evaluation uses ten stochastic rollouts on each of 1024 fixed worlds per model. An additional cohort of three AHC runs reaches $46.08\pm1.23$ . Two single-component ablations retain similar mean returns, while jointly freezing difficulty and removing inherited progress gives $45.14\pm2.79$ . All-67-achievement profiles show that most of the return difference comes from Intermediate achievements, while the deepest-floor achievements remain unsolved. A Skill-Preflight diagnosis with a 14B teacher shows how prerequisite and floor-access constraints restrict practice. AHC and three 14B diagnostic configurations reach aggregate returns near the reported 80B DiCode mean of 45.88 and below the 235B mean of 48.55. These results show that an explicit curriculum without LLM inference can reach returns comparable to those reported for LLM-generated curricula on Craftax-Full, without action masking.

Figures & tables

Enumerate tasks and task variants

\rightarrow

Select eligible units and one native anchor

\rightarrow

Initialize practice and set difficulty

\rightarrow

Train student and harvest progress

Figure 1: AHC’s data flow. The motivating diagnosis concerns restricted task access, missing deep practice, and limited information in aggregate return.

Item	Setting
Environment	Craftax-Symbolic-v1, Craftax 1.4.5
Budget	2,005,401,600 PPO rollout transitions (15300 updates); harvest and controller statistics reuse these rollouts; evaluation is excluded
Endpoints	Update 15300 (final) and 7600 (below 1B); diagnostic tail window 14400–15300
Evaluation	1024 fixed worlds $\times$ 10 stochastic rollouts, 8192-step cap; historical comparisons use one rollout per world
Actions and inputs	43 actions, no filter; 67-dimensional task indicator
Training-state access	Harvested progress, softened dynamics, engine-informed dependency graph

Table 1: Summary of the experimental setting.

Figure 2: The archive lock, drawn from the scheduler’s eligibility record (graph fingerprint cfbb1c9a4558 ). Eligibility under the true prerequisite graph, over five first-round arms’ end-of-run mastery profiles: an item is a legal target only when every prerequisite’s max-ever mastery clears the scheduler’s 0.30 line. enter_gnomish_mines is itself targetable but topped out at 0.20 against that line, and the 27 items behind it — of 67 in the graph, the three floor-2 cluster mobs among them — therefore remain ineligible under these measured profiles. One further under-line gate ( collect_sapphire ) additionally locks the two enchantments inside the same closure. The mastery profiles come from the first round of diagnostic runs.

		Warrior (s1)		Recovery
Configuration	Graph and comparison scope	Tail	Final	Tail	Final
longStack	Original graph; floor clause on	4.54	6.74	0/3	0/3
repairF	Repaired graph; API-reference differs	14.36	16.11	3/3	3/3
shufGraph	Fixed shuffled graph; changes both graph consumers	14.97	16.02	3/3	3/3

Table 2: Graph configurations in the diagnostic record. Warrior is the s1 rate (%) averaged over the last ten checkpoints (tail) and at checkpoint 15300 (final). Recovery counts use three training runs and the four-skill boundary; they are identical under both readings. RepairF also changes the API-reference setting relative to longStack.

Figure 3: Per-session mine-entry success on the held-out clean channel for 21 diagnostic runs and two separately marked partner runs. Line style identifies the true, repaired, or shuffled prerequisite graph. No true-graph series reaches the 0.30 scheduler threshold; the closest is partner base2e9 s3 at 27.83%. Three altered-graph runs cross late: shufGraph s3 at session 134, repairF s3 at 137, and shufGraph s2 at 146.

	Last ten checkpoints		Checkpoint 15300
Configuration	Return	Individual runs	Return	Individual runs
base15	$43.929\pm 1.309$	45.016 / 42.476 / 44.296	$44.509\pm 1.780$	46.024 / 42.549 / 44.955
longStack	$44.014\pm 0.795$	44.726 / 43.157 / 44.159	$44.589\pm 0.909$	45.016 / 43.545 / 45.206

Table 4: Diagnostic LLM curriculum comparison under two readings of the same evaluation summaries: the mean of the last ten checkpoints (primary) and the final checkpoint 15300, which is DiCode’s reporting convention. Uncertainty is sample SD across three runs. The checkpoint-15300 values are those compared with AHC in Table 36 ; the last-ten means are not used as comparators for AHC.

	Last ten checkpoints		Checkpoint 15300
Skill	base15	longStack	base15	longStack
Mine entry	$19.10\pm 3.95$	$9.64\pm 1.44$	$19.99\pm 5.52$	$11.78\pm 3.51$
Defeat gnome warrior	$15.12\pm 3.64$	$1.55\pm 2.59$	$16.67\pm 4.83$	$2.25\pm 3.89$
Defeat gnome archer	$8.96\pm 3.15$	$0.094\pm 0.164$	$9.96\pm 3.98$	$0.065\pm 0.113$
Eat bat	$14.08\pm 3.63$	$0.50\pm 0.45$	$15.07\pm 5.01$	$0.81\pm 0.75$

Table 5: Capability profiles (rates in %; three runs per arm) over the last ten checkpoints (primary) and at checkpoint 15300. In the tail window, the smallest base15 value exceeds the largest longStack value for every listed skill. At checkpoint 15300 this holds for warrior, archer, and bat but not for mine entry, where base15 s2 reaches 13.96 against 14.94 for longStack s3. Per-run rates are in Tables 34 and 35 .

Observation	Design response
Some downstream targets are not directly schedulable	Use the graph for downstream value and explicit, separately specified eligibility/frontier rules.
Direct deep-floor practice can be restricted	Represent harvested floor progress through procedurally generated, floor-indexed worlds.
Difficult tasks can lose selection or parent opportunities	Admit fresh eligible units, mix rank priority with staleness, and adapt task difficulty.
Scalar return conceals skill differences	Monitor the original task and report raw held-out skill rates alongside return.

Table 6: Diagnostic observations and AHC design responses. AHC retains hard eligibility rules alongside its soft scores.

Task class	Target	Initialization and dynamics	Candidate qualification
Family L1/L2	Craft the recipe’s mapped achievement	Harvested signature on its recorded floor; recipe tools/resource support; both base labels have $\sigma=0$	Usable bank and equipment-chain check; no depth-neighbor gate
Depth L1/L2	Enter floor $f\in\{1,\ldots,8\}$	Signature from $f-1$ , split by gate-progress bin $g\in\{0,3,6,8\}$ ; L1 native dynamics, L2 starts at $\sigma=1$	Bank, neighbor unlock and official-feedback frontier cap
Ordinary achievement L0	One of the 47 non-recipe, non-entry achievements	Signature at its harvested floor; no base surgery/toolkit; $\sigma=0$	Bank and applicable controller checks
Family L4 (official-reset level)	Same twelve recipe achievements	Ordinary native reset; no harvested-state initialization or toolkit; $\sigma=0$	Requires an official-anchor reading; not eligible for fresh-unit priority

Table 7: Active base task definitions.

L

is an implementation label,

f

a physical floor index (surface

f=0

), and

\sigma

a dynamics parameter. Bank availability, prerequisite checks, retirement, and selection remain additional conditions; Appendix A.1 gives the exceptions.

Step	Operation
1	Initialize enumerated units, bank, world pool, per-unit statistics, and session RNG. Perform the 200-update initialization.
2	For each of 151 sessions, create scheduled task variants; retain units with supply, then apply unlocking, chain, frontier, and retirement rules.
3	Admit fresh eligible units first; fill the remaining 15-unit capacity by rank/staleness sampling without replacement. Append the original-task anchor.
4	Assemble curriculum states from harvested signatures and official worlds; apply each unit’s difficulty setting. Reweight and normalize environment shares.
5	Run 100 PPO updates. Harvest progress and collect conditional unit success and original-task achievement readings.
6	Update EMA statistics, difficulty, graduation, parking, and regression state; persist the bank/controller and proceed to the next session.

Table 8: AHC session procedure. The total number of transitions is fixed; selection and reweighting change their allocation.

Training cohort	GPU	Update 7600	Update 15300
Primary AHC	L40S	$38.69\pm 2.26$	$46.70\pm 0.57$
Additional AHC	A100	$40.38\pm 1.69$	$46.08\pm 1.23$
Fresh native PPO-GTrXL	RTX A6000	$36.63\pm 1.42$	$41.91\pm 1.01$

Table 9: Core internal comparison. Return is mean

\pm

sample SD across three training seeds per cohort. Each checkpoint is evaluated on the same 1024 native worlds with ten stochastic rollouts per world and an 8192-step cap. Updates 7600 and 15300 correspond to 0.9961472B and 2.0054016B PPO rollout transitions, respectively.

Figure 4: Mean return for the three cohorts at the two evaluated PPO rollout budgets. Points are means across three training seeds and error bars are the across-seed sample SD; error-bar stems are offset horizontally for readability. Lines connect the two measured budgets as visual guides. Within each cohort, both points come from the same three training runs.

metric	seed 0	seed 1	seed 2	mean $\pm$ sample SD
mean return	46.945	47.105	46.051	$46.700\pm 0.568$
mean performance (%)	20.772	20.843	20.377	$20.664\pm 0.251$
average episode length	1451.2	1583.3	1355.3	$1463.2\pm 114.5$
enter_gnomish_mines (%)	28.281	27.500	23.369	$26.383\pm 2.640$
defeat_gnome_warrior (%)	21.504	21.738	17.510	$20.251\pm 2.377$
defeat_gnome_archer (%)	13.682	14.160	13.555	$13.799\pm 0.319$

Table 10: AHC at the fixed checkpoint-15300 endpoint, using 1024 fixed worlds and ten stochastic rollouts per world (10,240 episodes/model). Training seeds are 0–2; the world-generation seed is 0. The final column gives the across-seed mean and sample SD. Skill rates are percentages; performance is return as a percentage of the 226-point maximum. Evaluation uses the unfiltered 43-action policy.

Training GPU	Cohort	Seed	Return	Boundary
L40S	Primary	0	46.861	Yes
L40S	Primary	1	47.268	Yes
L40S	Primary	2	46.134	Yes
A100	Additional	1	45.205	No
A100	Additional	4	47.269	Yes
A100	Additional	5	44.951	No

Table 11: Original single-rollout results: one episode on each of 1024 worlds with an 8192-step cap. The L40S rows form the primary cohort; the A100 rows are an additional cohort. Boundary marks retention of the four-skill thresholds in § 2 . Seed 4 resumed training from update 15000 to 15300.

Model	Budget (B)	Return	I-18	Entry	Warrior	W $\mid$ entry
AHC, checkpoint 7600 (L40S)	0.996	$38.69\pm 2.26$	$34.84\pm 3.58$	$7.57\pm 1.95$	$1.51\pm 2.48$	$23.5\pm 39.0$
AHC, checkpoint 7600 (A100)	0.996	$40.38\pm 1.69$	$37.22\pm 2.48$	$8.05\pm 4.22$	$1.30\pm 2.23$	$19.0\pm 32.8$
Fresh native PPO, update 7600	0.996	$36.63\pm 1.42$	$32.62\pm 3.06$	$0.48\pm 0.34$	0	0
Native PPO, update 7000 (guard-on)	0.918	$36.24\pm 2.42$	$31.86\pm 4.43$	$0.47\pm 0.36$	$0.003\pm 0.006$	$0.4\pm 0.7$
Native PPO, update 8000 (guard-on)	1.049	$38.23\pm 1.18$	$35.49\pm 1.88$	$0.74\pm 0.35$	0	0
L40S AHC, checkpoint 15300	2.005	$46.70\pm 0.57$	$48.18\pm 0.87$	$26.38\pm 2.64$	$20.25\pm 2.38$	$76.7\pm 2.1$

Table 12: Repeated-world evaluation with ten rollouts on each of 1024 fixed worlds (10,240 episodes per model). Each row is mean

\pm

sample SD over three training seeds. I-18 is the unweighted macro success rate across the 18 official Intermediate achievements. Other rates are episode percentages. W

\mid

entry averages each seed’s conditional success rate. Guard-stopped native-PPO endpoints are historical checkpoints from the original runs; full-budget native PPO is a separate fresh run set. Training GPUs: L40S for primary AHC, A100 for A100 AHC, and RTX A6000 for D4 and native-PPO runs.

Cohort	Total	$w=1$ (25)	$w=3$ (18)	$w=5$ (15)	Residual
L40S AHC	4.795	1.115	3.615	0.072	$-0.007$
A100 AHC	4.171	1.128	2.854	0.187	0.002
D4-A	4.196	1.046	3.038	0.116	$-0.004$
D4-H	4.512	0.527	3.955	0.021	0.009
D4-AH	3.230	0.696	2.508	0.021	0.004

Table 13: Native-weight decomposition of the full-budget mean-return gap relative to fresh native PPO. Contributions are in return points. Each cohort has three seeds.

Arm	GPU	Return	Lowest	Highest	M4 (%)	O14 (%)
Full AHC, primary	L40S	$46.70\pm 0.57$	46.05	47.10	$19.86\pm 1.47$	$56.28\pm 0.70$
D4-A: fixed difficulty	A6000	$46.10\pm 1.18$	45.22	47.45	$15.81\pm 6.51$	$56.06\pm 0.77$
D4-H: no progress init.	A6000	$46.42\pm 1.55$	44.73	47.78	$18.64\pm 9.48$	$57.43\pm 1.90$
D4-AH: joint	A6000	$45.14\pm 2.79$	42.68	48.17	$11.51\pm 15.30$	$56.03\pm 2.10$
Full AHC, additional	A100	$46.08\pm 1.23$	45.33	47.50	$11.98\pm 11.02$	$56.71\pm 0.63$

Table 14: Full-budget component comparisons under the same repeated-world evaluation. Each arm has three training seeds; return, M4, and O14 show mean

\pm

sample SD. Lowest and highest are the observed per-seed return means, each estimated from 10,240 episodes.

Training seed	Elapsed (hh:mm:ss)	Allocated L40S GPU $\cdot$ h
0	17:50:18	17.8383
1	17:33:14	17.5539
2	17:37:21	17.6225
Completed-job total	53:00:53	53.0147

Table 15: Allocation time for the three primary AHC training jobs, including in-job evaluation. Each job used one L40S, so GPU

\cdot

h equals elapsed hours.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Part	Fields stored	Bounds / assembly
$f$	player_level	Selects floor-indexed pool; variant floors are not overridden.
$g$	Full monsters_killed vector	Copy compatible vector prefix; retain per-floor gate progress.
$i$	Wood, stone, coal, iron, diamond, sapling; pickaxe and sword tiers	Material caps $(3,2,3,3,3,9)$ ; tool tiers copied as recorded.
$a$	Four inventory.armour slots	Copy recorded values; no fixed armour grant.
$e$	Torches, arrows, ruby, sapphire, bow, books, potion slots	Caps 12 for torches/arrows, 9 for gems, 3 per potion slot; bow/books copied.
Excluded	Map, position, timestep, health/needs, actors, historical achievement bits	Never copy these from the harvested state; derive already-earned achievement flags.

Appendix

Table 16: Stored signature fields and realization semantics. Clamps limit stored quantities, not fixed amounts granted to the student.

Step	Operation
1	For selected unit $u$ , obtain entries from its bank or variant-parent bank. Official-practice units obtain no entries. Skip entries without a signature.
2	Set $S=(f,g,i,a,e)$ ; for D4-H (and D4-AH), replace it by $(0,(),(),(),())$ . Set $f^{*}=f$ ; variant floors remain unspecified in the resolved configuration.
3	Draw $W$ from the official pool at $f^{*}$ ; skip if unavailable. Clone $W$ . Its map, actors, position, timestep and vitals are not inherited.
4	Copy each nonempty signature field $g,i,a,e$ , preserving engine dtypes. Copy array prefixes; merge variant scalar prefill as $\max(\text{signature value},v)$ , with omitted signature scalars defaulting to zero.
5	Ensure goal-recipe ore supply, then configured lighting. Apply explicit variant softening, otherwise the unit’s sigma specification. Non-axis units have $\sigma=0$ (native).
6	Recompute inventory-derived achievement bits; mark entered floors from $f^{*}$ . Require the goal bit to remain false. Attach $u$ and build observation.

Appendix

Table 17: State-assembly pseudocode for the active signature path. All unspecified state fields come from the generated world until an explicit task operator changes them. The original-task anchor bypasses curriculum assembly.

Figure 5: All-67 capability differences, part 1: native achievement IDs 0–33. Cells are cohort mean rates minus fresh native PPO in percentage points at update 15300; brackets give native reward weights. All cells use three seed means, each based on 1024 fixed worlds and ten repeats. Zero after rounding need not mean exact equality.

Figure 6: All-67 capability differences, part 2: native achievement IDs 34–66, with the same scale and definitions as Figure 5 . The nine weight-8 achievements have zero successes in every cohort; zero differences alone would not distinguish shared failure from shared mastery.

Task UID	Seed 0	Seed 1	Seed 2
fam:iron_pickaxe\|L1\|i0 : selected sessions	24	29	34
First / last selection	2 / 144	2 / 148	2 / 148
depth:f1g8\|L2 : selected sessions	7	12	32
First / last selection	2 / 21	2 / 23	2 / 54

Appendix

Table 18: Examples of observed task selection in the joint arm. Counts refer to sessions with positive configured task probability; first/last are session indices, not a continuous active interval.

Training seed	Units in export	Variants	Selected	First exit
0	191	44	132	82
1	203	56	145	84
2	200	53	144	82

Appendix

Table 19: Scope of recovered primary-AHC task records. Units include 147 base units and the listed variants. “Selected” counts distinct units with at least one logged seat assignment; “First exit” counts distinct units with a first recorded graduation or parking event.

Cohort	Runs	Cases	Retained updates
Primary L40S AHC	3	6	7600, 15300
Additional A100 AHC	3	6	7600, 15300
Fresh native PPO	3	6	7600, 15300
D4-A / D4-H / D4-AH	3 each	9	15300
Historical guard-stopped PPO	3	9	7000, 8000, last accepted endpoint
Additional A100 D4-A	1	1	15300

Appendix

Table 20: Repeated-world checkpoint inventory. Every case has 10,240 episode records.

Training GPU	Seed	Return	Mine entry	Warrior	Archer	Bat	Remainder
L40S	0	46.861	28.52	22.36	14.45	18.95	44.33
L40S	1	47.268	28.03	21.78	13.28	19.73	44.78
L40S	2	46.134	23.73	18.16	14.94	18.95	43.86
A100	1	45.205	21.78	0.00	0.00	0.00	44.55
A100	4	47.269	29.00	23.54	18.07	22.36	44.48
A100	5	44.951	19.53	1.66	0.29	0.88	44.28

Appendix

Table 21: Deep-floor success rates (%) for the six L40S primary and A100 additional runs at checkpoint 15300 under the original single-rollout protocol. The Remainder column subtracts the four achievements’ contribution (three points times each success rate) from the return.

System	Training seed	Return	Mine entry	Warrior	Archer	Bat
AHC	0	46.860676	28.515625	22.363281	14.453125	18.945312
AHC	1	47.267670	28.027344	21.777344	13.281250	19.726562
AHC	2	46.134350	23.730469	18.164062	14.941406	18.945312
base15	1	46.023933	24.804688	21.093750	14.550781	20.507812
base15	2	42.548748	13.964844	11.523438	7.714844	10.644531
base15	3	44.954964	21.191406	17.382812	7.617188	14.062500

Appendix

Table 22: Inputs to the common-checkpoint comparison. These are single checkpoint-15300 values, not tail-window means. Skill rates are percentages; six decimal places are shown and JSON precision is retained for aggregation.

Model	Seed	Update	Return	SE	I-18	M4	O14	Mine n	W n (%entry)
L40S AHC	0	15300	46.945	0.189	48.70	20.62	56.72	2896	2202 (76.0)
L40S AHC	1	15300	47.105	0.190	48.67	20.79	56.64	2816	2226 (79.0)
L40S AHC	2	15300	46.051	0.198	47.17	18.16	55.46	2393	1793 (74.9)
L40S AHC	0	7600	39.924	0.187	36.92	2.52	46.74	1006	10 (1.0)
L40S AHC	1	7600	40.065	0.190	36.89	1.66	46.96	666	7 (1.1)
L40S AHC	2	7600	36.078	0.237	30.70	4.54	38.18	654	448 (68.5)

Appendix

Table 23: Per-seed repeated-world results (10,240 episodes per model). Return SE is world-clustered within each run, not a training-seed SD. I-18, M4, and O14 are macro-averages (%) over the 18 official Intermediate achievements, the four M4 achievements, and the other 14. Mine n and W n (%entry) are episode counts; the parenthetical value is the warrior success percentage among mine-entry episodes. L40S and A100 AHC denote distinct training cohorts. Fresh native PPO is separate from the historical guard-on PPO runs.

Current matched cohorts
Achievement	M4	L40S AHC 7600	L40S AHC 15300	A100 AHC 7600	A100 AHC 15300	Fresh PPO 7600	Fresh PPO 15300
collect_sapphire		12.22	21.07	13.46	21.62	11.10	18.23
collect_ruby		12.04	21.28	13.53	21.65	11.31	17.96
make_diamond_pickaxe		1.76	3.18	2.01	3.26	1.51	2.56
make_diamond_sword		1.92	4.06	2.17	4.23	1.65	2.95
make_iron_armour		15.71	30.98	18.25	30.33	4.33	8.02

Appendix

Table 24: Success rates (%) for the 18 official Intermediate achievements. Each cell is an equal-weight mean over three training seeds. The upper panel gives six current matched cohorts; PPO denotes fresh native GTrXL runs. The lower panel gives the three D4 intervention arms and historical guard-on PPO cohorts, which are separate training cohorts from the upper panel. D4-AH is the fresh joint intervention. M4 marks the four deep-floor achievements. The source key defeat_orc_solider retains its original spelling.

Training seed	Update 7600: 996,147,200 transitions	Update 15300: 2,005,401,600 transitions
0	Present; 12 files	Present; 12 files
1	Present; 12 files	Present; 11 files
2	Present; 11 files	Present; 11 files

Appendix

Table 25: Archived AHC checkpoints. Entries count files covered by the supplied checksum manifest, not independent models or evaluations. There are three training executions and two checkpoints per execution.

Parameter	Value
GTrXL layers / attention heads	2 / 8
Embedding / QKV / hidden width; activation	256 / 256 / 256; ReLU
Transformer gating; gate bias	Enabled; 2.0
Memory / gradient window (steps)	128 / 64
Task conditioning; action set	67-dimensional achievement indicator; 43 actions
Parallel environments / rollout steps	1024 / 128

Appendix

Table 26: Shared recorded training settings. These are training parameters; the frozen evaluation uses its separately specified protocol.

Arm	Difficulty A	Progress H	Task / completion reward	Entropy stop
Full AHC	Adaptive $\sigma$	Enabled	Curriculum + native anchor; +20 on curriculum success	0.15
D4-A	Fixed native dynamics	Enabled	Same task interface	0.15
D4-H	Adaptive $\sigma$	Neutral floor-zero signature	Same task interface	0.15
D4-AH	Fixed native dynamics	Neutral floor-zero signature	Same task interface	0.15
Fresh native PPO	Native dynamics	Native resets	Original task only; native reward	Disabled

Appendix

Table 27: Controlled training differences. A/H abbreviate difficulty annealing and harvested-progress initialization. Removing H retains per-unit task operators and variant prefills. Historical guard-stopped PPO is a separate cohort.

Seed	Return	Performance (%)	Mean episode length	Training RTX A6000 GPU $\cdot$ h
0	41.843	18.514706	1337.300	22.4089
1	40.668	17.994696	1351.456	22.9050
2	42.068	18.614149	1159.646	20.8015

Appendix

Table 28: Single-rollout measurements for the three guard-stopped PPO endpoints (first episode on each of 1024 worlds). Performance is return as a percentage of 226. Training time is elapsed time on one RTX A6000, including in-run evaluation and rejected sessions.

Intervention	Seed	Return	Performance (%)	Episode length
Fixed difficulty	0	47.812920	21.156158	1462.630
Fixed difficulty	1	45.908348	20.313429	1553.788
Fixed difficulty	2	45.132290	19.970039	1457.445
No progress initialization	0	46.757309	20.689075	2115.051
No progress initialization	1	44.585232	19.727980	1216.120
No progress initialization	2	48.070557	21.270159	1616.744

Appendix

Table 29: D4 final-checkpoint results by training seed. These are individual run means over 1024 first episodes. Performance is return as a percentage of the 226-point maximum.

Training seed	Mine entry	Warrior	Archer	Bat
0	22.4	0.1	0.1	0.0
1	41.9	36.8	29.9	36.4
2	34.7	26.8	15.3	25.2
Reported mean	33.0	21.2	15.1	20.5

Appendix

Table 30: DiCode reproduction capability profile at checkpoint 15300, transcribed from the collaborator summary. Rates are percentages; the last row is the supplied across-seed mean.

Arm	Graph change	Scheduler, gate, and prompt change
longStack	Original 67-node, 94-edge graph	Prerequisite scheduler; R3 floor clause enabled
repairF	Replace mine-entry prerequisite with dungeon entry for warrior, archer, and bat only	Scheduler and R3 use repaired graph; mastered-prerequisite exemption retained; API-reference content differs
shufGraph	Fixed double-edge rewiring, seed 20260804; 79 of 94 original edges replaced	Scheduler and R3 use shuffled graph; target identities and scaffold checks both change
Surgical	Original graph retained	Only floor-granted achievements removed from R3 scaffold checks; premark/inventory channels and R1/R2 retained

Appendix

Table 31: Diagnostic interventions relative to longStack. All three training seeds in each arm use the same arm definition. This historical diagnostic comparison is separate from the AHC component ablations.

Figure 7: Mastered-prerequisite exemption timing in the three repairF runs. The blue curve is the clean-channel enter_dungeon success rate. Black lines mark

s^{*}

, the first of three consecutive readings at or above 70% (the 0.7 criterion): sessions 27, 16, and 19, respectively. Orange dots give the proportion of cluster-mob levels whose generated code starts on floor 2 or deeper, among levels with a nonempty training-performance history. Levels target gnome warriors, gnome archers, or bats and are grouped by their creation session, rather than by the session when they were trained. Dot size increases with the number of counted levels and is capped. Orange segments pool level counts before (

s<s^{*}-2

) and after (

s>s^{*}+2

), excluding the inclusive

s^{*}\pm 2

band. Before/after numerator–denominator pairs are 0/1 and 166/253, 2/7 and 142/269, and 1/1 and 37/180.

Seed	Updates	Valid transitions	Return	Mine entry	Archer	Bat
0	14600	1,913,651,200	42.086	7.266	0	0.029
1	15000	1,966,080,000	40.249	2.275	0	0
2	13600	1,782,579,200	41.867	4.648	0.020	0.029

Appendix

Table 32: Original native-PPO runs stopped by the entropy guard, evaluated with ten rollouts on each of 1024 worlds. Updates and PPO rollout transitions describe the last valid checkpoint; each transition count equals updates

\times 1024\times 128

. Skill rates are percentages of 10,240 episodes. Every run stops short of the 2,005,401,600-transition target.

System	seed 0	seed 1	seed 2	Mean $\pm$ sample SD
AHC (single rollout)	46.861	47.268	46.134	$46.754\pm 0.574$
DiCode (collaborator reproduction)	46.74	50.93	48.64	$48.77\pm 2.10$

Appendix

Table 33: Three-seed endpoint returns on the reported common 1024-world evaluation. AHC values come from the original single-rollout JSONs; DiCode values and its sample SD are reported at the supplied summary’s precision.

configuration	seed	mines	warrior	archer	eat_bat
shufGraph	s1	19.06	14.97	11.00	15.05
	s2	26.94	22.00	13.89	21.02
	s3	26.33	21.13	13.50	19.38
repairF	s1	19.73	14.36	5.15	5.83
	s2	23.37	18.17	12.66	15.78
	s3	29.30	24.06	18.73	22.67

Appendix

Table 34: Per-seed last-10 means (%) for the four second-floor skills. Supports the base15–longStack separation and the recovery counts; other recorded configurations are included for completeness.

configuration	seed	return	last-ten	mines	warrior	archer	eat_bat
shufGraph	s1	45.480	45.661	20.21	16.02	12.11	15.62
	s2	47.652	46.520	29.00	25.10	16.60	23.83
	s3	47.848	47.041	29.98	24.41	15.23	22.75
repairF	s1	44.908	44.368	22.85	16.11	4.88	7.23
	s2	45.980	46.270	22.36	16.21	11.82	15.33
	s3	48.484	47.328	33.50	27.44	21.39	26.37

Appendix

Table 35: Per-run readings at checkpoint 15300 from the same evaluation summaries as Table 34 : return in points (with the last-ten mean beside it) and the four second-floor skill rates in %. Recovery labels are unchanged from the tail window.

System	Return	Mine entry	Warrior	Archer	Bat
AHC	$46.754\pm 0.574$	$26.758\pm 2.633$	$20.768\pm 2.274$	$14.225\pm 0.853$	$19.206\pm 0.451$
base15	$44.509\pm 1.780$	$19.987\pm 5.519$	$16.667\pm 4.825$	$9.961\pm 3.975$	$15.072\pm 5.008$
longStack	$44.589\pm 0.909$	$11.784\pm 3.508$	$2.246\pm 3.890$	$0.065\pm 0.113$	$0.814\pm 0.746$

Appendix

Table 36: Historical single-rollout final-endpoint comparison, recomputed from existing checkpoint-15300 JSON entries. Each cell is mean

\pm

sample SD over three training runs. All rows use Craftax 1.4.5, evaluation seed 0, and 1024 environments with one rollout per world and an 8192-step cap. The AHC row is the original single-rollout result, not Table 10 ’s repeated-world estimate. Skill columns are percentages.

Near 1B transitions
Model	Budget (B)	LLM	Mask	Return
PPO-GTrXL (maintainer board)	1.000	No	None	41.36
Near 2B transitions
PPO-GTrXL (DiCode report)	2.000	No	None	41.59
DiCode 30B (reported)	2.000	Yes	Not audited	$44.38\pm 0.66$
DiCode 80B (reported)	2.000	Yes	Not audited	$45.88\pm 0.40$

Appendix

Table 37: External and historical numerical context, separate from the internal two-budget comparison in Table 9 . External results use different software versions, evaluation protocols, and time caps; DiCode uncertainties are standard errors ( Mitsides et al., 2026 ) . The 14B diagnostic panel uses one rollout on each of 1024 worlds: its checkpoint-15300 mean

\pm

SD is followed by the last-ten mean in parentheses. The other two diagnostic endpoint means are base2e9 45.16 and gateOff 43.85 (Appendix A.18 ).

Comparison	Difference	Welch 95% interval
L40S AHC $-$ fresh PPO, update 7600	$2.06$	$[-2.56,6.68]$
A100 AHC $-$ fresh PPO, update 7600	$3.76$	$[0.18,7.33]$
L40S AHC $-$ fresh PPO, update 15300	$4.80$	$[2.71,6.88]$
A100 AHC $-$ fresh PPO, update 15300	$4.17$	$[1.58,6.76]$
D4-A $-$ fresh PPO, update 15300	$4.20$	$[1.67,6.72]$
D4-H $-$ fresh PPO, update 15300	$4.51$	$[1.35,7.67]$

Appendix

Table 38: Return differences and Welch 95% intervals from three training-seed means in each cohort, using the ten-repeat evaluator. The final three rows compare against historical guard-stopped endpoints at their own budgets.

Curriculum RL

Curriculum reinforcement learning organizes tasks, examples, or training conditions to help agents learn complex behaviors efficiently.

See all 124 papers

Anticipating the Consequences of Curriculum Decisions with Large Language Models
Oct 3, 2026Octavio Pappalardo, Nathan Herr, Tim RocktäschelCurriculum Learning Curriculum RL
University College London
SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding
Mar 10, 2026Renos Zabounidis, Yue Wu, Simon Stepputtis +4LLM-Guided RL LLM Agent Skill Learning
School of Computer Science, Carnegie Mellon University · Department of Mechanical Engineering, Virginia Tech
S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning
Oct 26, 2026Shuo Shen, Guangran Cheng, Chengqi Lyu +3RL for Language Model Reasoning RL for Language Models
Shanghai AI Laboratory · Independent Researcher

Oct 3, 2026J/K move · Enter open · S save

Anticipating the Consequences of Curriculum Decisions with Large Language Models

Octavio Pappalardo, Nathan Herr, Tim Rocktäschel

University College London

Curriculum Learning Curriculum RL LLM-Guided RL

Open paper AskPDF arXiv

Automatic curriculum learning can improve the effectiveness of reinforcement learning by selecting the training experiences presented to the agent over time. Predicting the consequences of such decisions can, however, be difficult. We analyze automatic curriculum learning as a sequential decision-making problem, highlighting a gap between the quantities that determine the value of curriculum decisions and the information captured by local learning signals commonly used to guide them. We then investigate whether Large Language Models (LLMs) can exploit richer information about the learning problem to better anticipate the consequences of curriculum decisions. We introduce a method that combines online learning-progress estimates with LLM-informed estimates of (i) the potential downstream benefits of learning on each task and (ii) whether direct training on a task is currently likely to produce progress. We evaluate the approach on a custom benchmark of 256 textual goals in Craftax under different curriculum objectives. We observe the strongest gains when optimizing for individual target tasks. When optimizing across the full task set, the benefits vary across learners with different mechanisms for cross-task transfer, ranging from modest improvements in learning speed to larger gains that persist through the end of training.

Abstract

Curriculum RL

Explore similar work

Anticipating the Consequences of Curriculum Decisions with Large Language Models

SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding

S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning