Large language models increasingly produce AI slop in generated code: verbose, redundant, non-idiomatic implementations that pass functional checks but bloat codebases. The problem compounds on SDKs released after the model's training cutoff, where the model has neither idiomatic priors nor compositional understanding. We present Option-GRPO, a hierarchical reinforcement learning framework that adapts a code LLM to an unseen SDK from its source code alone, without human-written usage examples or test cases, and that explicitly optimizes for both functional correctness and code elegance. Our key insight is that SDK source code contains compositional structure that defines a hierarchy of reusable code patterns (code options) which simultaneously serve as the building blocks of a curriculum and the action space for efficient policy optimization. We show that code options induce a valid semi-Markov decision process, give conditional bounds under which shorter option description length implies more concise and, in expectation, more API-efficient programs, and analyze under stated assumptions how the option hierarchy improves credit assignment by a factor of T/K. Using Group Relative Policy Optimization (GRPO) with execution-based rewards at each curriculum level, Option-GRPO discovers SDK idioms, composes them into patterns, and assembles patterns into complete programs. Experiments on 12 unseen SDKs with a 32B-parameter base model demonstrate 61.4% pass@1, outperforming flat GRPO by 8.7 points, ExploraCoder by 11.2 points, and ReCode by 6.3 points. Option-GRPO produces solutions that use 34.7% fewer API calls and are 27% shorter than flat-GRPO outputs while maintaining higher correctness, directly attacking the slop problem.
Figures & tables
Method
RL
No Examples
Structural
Hierarchical
Code Quality
No Critic
CodeRL
✓
×
×
×
×
×
PPOCoder
✓
×
✓
×
×
×
StepCoder
✓
×
×
×
×
×
ExploraCoder
×
✓
×
×
×
N/A
ReCode
✓
✓
×
×
×
✓
Voyager †
×
×
×
✓
×
N/A
Table 1: Comparison with most relevant prior work. Option-GRPO uniquely combines hierarchical RL with structural SDK knowledge for example-free generation. † Voyager builds a skill library via iterative GPT-4 prompting, not gradient-based RL; we mark its RL column accordingly.
Figure 1: Overview of Option-GRPO. The top row constructs three SDK graphs from source code and discovers a hierarchical option vocabulary A0⊂A1⊂A2 (Algorithm 1 ). The middle row runs three curriculum phases of GRPO training, each receiving the option vocabulary from above and the layered execution reward from below (dashed arrows). Successful trajectories from phase k contribute newly extracted options to phase k+1 (Algorithm 2 ). The trained policy πθ generates code at inference. Solid arrows denote training-stage progression; dashed arrows denote artifact dependencies.
Method
pass@1
pass@10
CL ↓
AE ↓
ID ↑
Frozen LLM (32B)
28.5
42.1
94.6
7.8
0.18
RAG
41.3
56.8
88.1
6.4
0.31
DocPrompting
44.8
60.2
85.4
5.9
0.34
Gorilla
47.2
62.7
82.9
5.5
0.36
ExploraCoder
50.2
67.5
78.4
4.9
0.43
ReCode
55.1
71.2
74.1
4.6
0.51
Table 2: Main results on UnseenSDK-Bench. Option-GRPO achieves the highest functional correctness while producing more concise and idiomatic code. ↓ indicates lower is better; ↑ indicates higher is better.
Tasks
Scorer
Flat
Opt
Δ
Synthetic
Property tests
52.7
61.4
+8.7
Synthetic
Reference output
44.7
54.2
+9.5
Tutorial
Property tests
50.8
57.3
+6.5
Tutorial
Reference output
43.6
51.8
+8.2†
Table 3: Independent evaluation (pass@1, %). Tasks: the paper’s synthetic tasks or 132 human-written tutorial tasks. Scorer: the paper’s property tests or reference-output equivalence. † 95% CI +3.1 to +11.6 (paired bootstrap).
Figure 2: Cost–performance curves averaged over the 12 SDKs (Tables 19 and 20 ). Left: pass@1 against aggregate H100 GPU-hours per SDK on the 32B base. Right: pass@1 against dense base size at the default budget. Dashed: Flat GRPO at the default budget on the 32B base ( 52.7% ).
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Model
Base model
Qwen3-32B
Precision
bf16
GRPO
Group size G
16
Sampling temperature
0.8
Appendix
Table 4: Complete training hyperparameters.
SDK
Category
Funcs
L0 Tasks
L1 Tasks
L2 Tasks
Prop. Tests
Daft
DataFrame
156
156
312
33
624
LanceDB
Vector DB
142
142
284
33
568
cuallee
Data Valid.
87
87
174
33
348
Docling
Doc Proc.
198
198
396
33
792
Marker
Doc Proc.
112
112
224
34
448
Surya
Comp. Vision
94
94
188
33
376
Appendix
Table 5: UnseenSDK-Bench statistics per SDK. L0/L1/L2 denote curriculum levels. Property tests are generated automatically from type annotations.
SDK
L0
L1 Cand.
L1 Filt.
Avg Len
Density
Daft
156
284
73
2.7
0.041
LanceDB
142
241
61
2.5
0.038
cuallee
87
159
54
2.4
0.062
Docling
198
312
48
2.9
0.029
Marker
112
187
39
2.6
0.034
Surya
94
148
35
2.4
0.036
Appendix
Table 6: Option discovery results per SDK. Composability density is ∣Ecompose∣/∣F∣2 .
Recall
SDK
Release
Cutoff
Qwen3
GPT 5.5
Opus 4.8
Daft
2022-08
Pre
19.5%
48.5%∗
44.0%∗
LanceDB
2023-03
Pre
16.0%
59.0%∗
52.0%∗
cuallee
2023-01
Pre
12.5%
44.0%∗
41.5%∗
Docling
2024-07
Pre
08.0%
46.0%∗
50.5%∗
Marker
2024-02
Pre
10.5%
52.0%∗
46.5%∗
Appendix
Table 7: Contamination analysis. Release dates and the Cutoff column are relative to the Qwen3-32B training cutoff (approximately Feb 2025). Recall is top- 1 identifier-completion accuracy on 200 probes per SDK at temperature 0 , for the frozen Qwen3-32B and for the two current frontier models. ∗ Recall at or above the 25% threshold: the SDK is functionally seen by that model, so a comparison against that model on that SDK is a seen-SDK comparison rather than a like-for-like win. Controls are probed on the base model only.
SDK
Category
p@1
p@10
CL ↓
AE ↓
ID ↑
OR ↑
cuallee
Data Valid.
78.2
91.0
39.4
2.3
0.89
0.80
LanceDB
Vector DB
74.6
87.4
42.7
2.7
0.83
0.70
Marker
Doc Proc.
74.1
86.8
44.5
3.0
0.82
0.66
Surya
Comp. Vision
63.2
78.4
51.9
3.6
0.80
0.62
Daft
DataFrame
72.9
85.0
45.8
2.7
0.83
0.72
Vortex
Scientific
60.6
77.0
49.6
3.1
0.70
0.63
Appendix
Table 8: Full per-SDK results for Option-GRPO: pass@1, pass@10, code length CL, API efficiency AE (lower is better), idiomaticity ID (higher is better), option reuse rate OR (higher is better).
SDK
Category
p@1
p@10
CL ↓
AE ↓
ID ↑
OR ↑
cuallee
Data Valid.
67.2
83.0
61.6
4.2
0.52
0.38
LanceDB
Vector DB
63.8
79.7
64.7
4.5
0.45
0.31
Marker
Doc Proc.
64.8
81.0
62.4
4.5
0.49
0.32
Surya
Comp. Vision
55.8
70.9
70.2
5.0
0.44
0.25
Daft
DataFrame
59.9
76.3
66.2
4.6
0.49
0.35
Vortex
Scientific
51.7
67.6
71.0
4.8
0.38
0.25
Appendix
Table 9: Full per-SDK results for Flat GRPO (no hierarchy, layered execution reward). Same metrics as Table 8 .
Figure 3: Convergence curves averaged across 12 SDKs. Option-GRPO reaches the flat-GRPO terminal pass@1 in roughly one-third the steps and continues to improve.
Figure 4: Discovered option hierarchy for Daft (excerpt). Blue: Level-0 atomic options (one per SDK function). Orange: Level-1 compositional options (centroid paths of length 3 from Algorithm 1 ). Green: Level-2 program-level composition learned at Phase 3. Edges denote type-composability inclusion. The full Daft hierarchy contains 156 Level-0 options and 73 Level-1 options; we show only the chain underlying a representative Level-2 program.
Figure 5: Per-step relative gradient norm variance (averaged over 12 SDKs). Option-GRPO shows roughly 2.3× lower variance at the end of training (about 1.5× in SNR), in the direction predicted by Proposition 3 but smaller than its T/K scaling.
Config.
p@1
CL ↓
ID ↑
Conv. (steps)
Flat GRPO
52.7
71.3
0.41
24,800
Curric. only
55.4
66.8
0.49
19,200
Options only
58.1
58.7
0.66
15,600
Full (Option-GRPO)
61.4
52.0
0.74
12,400
Appendix
Table 10: Hierarchy ablation. Both options and curriculum contribute; their combination yields the largest gains. Rewards differ across rows: Flat GRPO uses the layered execution reward only; Curric. only and Full use the same schedule (execution reward, plus the quality term from Phase 2 and the consistency term in Phase 3); Options only uses the full reward throughout.
Config.
p@1
Option level
Level 0 only
53.9
Levels 0 + 1
58.6
Full (Levels 0 + 1 + 2)
61.4
Reward
Binary reward only
34.2
Appendix
Table 11: Further ablations (pass@1, %, averaged over the 12 SDKs). Top: curriculum levels used in training. Bottom: reward function.
Table 13: Comparison with prior RL methods on UnseenSDK-Bench. ∗ Adapted to use our layered execution reward in place of unit tests. All use the same Qwen3-32B base.
Method
Recall
Unseen
p@1
p@10
Δ
Frozen Qwen3-32B
10.5%
12/12
28.5
42.1
−32.9
GPT-4o
27%
8/12
54.6
69.4
−6.8
Claude 3.5 Sonnet
35%
6/12
57.2
71.8
−4.2
Gemini 2.0 Flash
32%
7/12
51.8
67.1
−9.6
GPT 5.5
44.5%
3/12
Table 15
Claude Opus 4.8
44.3%
2/12
Table 15
Appendix
Table 14: Comparison with frontier LLMs on the full 12 -SDK suite. The frontier LLMs receive the same top-10 RAG context (source-code chunks). Option-GRPO is shown without retrieval, which makes the comparison conservative: adding RAG improves Option-GRPO to 63.7 pass@1 and 78.5 pass@10 (Table 17 ). The other Qwen3-32B rows repeat Table 2 . Recall is mean top- 1 identifier recall over the 12 SDKs under the probe of Appendix C.2 (for ReCode and Option-GRPO, that of the Qwen3-32B base); Unseen is the number of SDKs below the 25% threshold for that model. A full-suite row is a like-for-like comparison only on the SDKs unseen to that model. GPT 5.5 and Claude Opus 4.8 are functionally seen on most of the suite and are compared only on the SDKs unseen to both (Table 15 ).
SDK
Metric
GPT 5.5
Opus 4.8
Opt-GRPO
Surya
pass@1
55.3
57.5
63.2
Surya
pass@10
71.8
73.7
78.4
Surya
Ref. p@1
45.3
44.7
56.2
Tracecat
pass@1
38.4
42.1
46.8
Tracecat
pass@10
45.8
50.4
63.1
Tracecat
Ref. p@1
31.8
30.2
37.8
Appendix
Table 15: Like-for-like comparison with current frontier models on the two SDKs that are functionally unseen to both (recall below 25% , Table 7 ). Same protocol as Table 14 : the frontier models receive the top-10 source-chunk RAG context and Option-GRPO is evaluated without retrieval; identical prompts, n=10 samples, temperature 0.8 . Ref. p@1 is pass@1 under the stricter reference-output-equivalence scorer.
Table 17: Complementarity with retrieval at inference time.
Figure 6: Code quality metrics against ODL on passing solutions (a sample of the solutions is plotted; the correlations in the legend are computed over all 5,042 ). All four metrics show strong negative correlation with ODL, consistent with Theorem 2 (conciseness), Corollary 1 (API efficiency), and the empirical predictions of Remark 5 ( Qred omitted from plot for legibility; ρred=−0.62 , p<10−50 ).
Phase
Steps
Smp/Step
GPU-hrs
Graph construction
–
–
<0.1
Option discovery
–
–
<0.1
Property test gen.
–
–
<0.1
Level 0 training
5,000
16
113.6
Level 1 training
10,000
16
198.4
Level 2 training
15,000
16
324.0
Appendix
Table 18: Estimated per-phase computational cost in aggregate H100 GPU-hours ( 8 GPUs × wall-clock hours on the 8× H100 node).
Total steps
GPU-hr
Wall (h)
Flat
Opt
Δ
5,000
112
14
33.8
41.2
+7.4
10,000
224
28
44.6
54.1
+9.5
15,000
336
42
49.3
58.7
+9.4
20,000
440
55
51.2
60.4
+9.2
30,000 (default)
636
80
52.7
61.4
+8.7
45,000
976
122
53.1
62.1
+9.0
Appendix
Table 19: Training-budget sweep on Qwen3- 32 B (dense). Step counts are total across all three curriculum phases; we allocate 1/6 , 1/3 , 1/2 of the budget to Phases 1,2,3 respectively. The Flat GRPO baseline runs at the same total step budget and the same 32 B base. GPU-hr is aggregate H100 GPU-hours per SDK; Wall is wall-clock hours on the 8× H100 node.
Base
Params
Mem
Flat
Opt
Δ
Qwen3- 8 B
8.2 B
20 GB
43.2
52.6
+9.4
Qwen3- 14 B
14.8 B
36 GB
48.9
57.3
+8.4
Qwen3- 32 B
32.8 B
75 GB
52.7
61.4
+8.7
Appendix
Table 20: Base-model-size sweep at default ( 30,000 step) budget. All variants are dense models from the Qwen3 family. GPU-memory footprint is per-replica peak during training with ZeRO-3 sharding on 8× H100.