Option-GRPO: Hierarchical RL Against AI Slop in Code Generation for Unseen SDKs
Organizations: Cerby Inc., USA
Abstract
Large language models increasingly produce AI slop in generated code: verbose, redundant, non-idiomatic implementations that pass functional checks but bloat codebases. The problem compounds on SDKs released after the model's training cutoff, where the model has neither idiomatic priors nor compositional understanding. We present Option-GRPO, a hierarchical reinforcement learning framework that adapts a code LLM to an unseen SDK from its source code alone, without human-written usage examples or test cases, and that explicitly optimizes for both functional correctness and code elegance. Our key insight is that SDK source code contains compositional structure that defines a hierarchy of reusable code patterns (code options) which simultaneously serve as the building blocks of a curriculum and the action space for efficient policy optimization. We show that code options induce a valid semi-Markov decision process, give conditional bounds under which shorter option description length implies more concise and, in expectation, more API-efficient programs, and analyze under stated assumptions how the option hierarchy improves credit assignment by a factor of . Using Group Relative Policy Optimization (GRPO) with execution-based rewards at each curriculum level, Option-GRPO discovers SDK idioms, composes them into patterns, and assembles patterns into complete programs. Experiments on 12 unseen SDKs with a 32B-parameter base model demonstrate 61.4% pass@1, outperforming flat GRPO by 8.7 points, ExploraCoder by 11.2 points, and ReCode by 6.3 points. Option-GRPO produces solutions that use 34.7% fewer API calls and are 27% shorter than flat-GRPO outputs while maintaining higher correctness, directly attacking the slop problem.
Figures & tables
| Method | RL | No Examples | Structural | Hierarchical | Code Quality | No Critic |
|---|---|---|---|---|---|---|
| CodeRL | ||||||
| PPOCoder | ||||||
| StepCoder | ||||||
| ExploraCoder | N/A | |||||
| ReCode | ||||||
| Voyager † | N/A |
| Method | pass@1 | pass@10 | CL | AE | ID |
|---|---|---|---|---|---|
| Frozen LLM (32B) | |||||
| RAG | |||||
| DocPrompting | |||||
| Gorilla | |||||
| ExploraCoder | |||||
| ReCode |
| Tasks | Scorer | Flat | Opt | |
|---|---|---|---|---|
| Synthetic | Property tests | |||
| Synthetic | Reference output | |||
| Tutorial | Property tests | |||
| Tutorial | Reference output |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Model | |
| Base model | Qwen3-32B |
| Precision | bf16 |
| GRPO | |
| Group size | |
| Sampling temperature | |
| SDK | Category | Funcs | L0 Tasks | L1 Tasks | L2 Tasks | Prop. Tests |
|---|---|---|---|---|---|---|
| Daft | DataFrame | |||||
| LanceDB | Vector DB | |||||
| cuallee | Data Valid. | |||||
| Docling | Doc Proc. | |||||
| Marker | Doc Proc. | |||||
| Surya | Comp. Vision |
| SDK | L0 | L1 Cand. | L1 Filt. | Avg Len | Density |
|---|---|---|---|---|---|
| Daft | |||||
| LanceDB | |||||
| cuallee | |||||
| Docling | |||||
| Marker | |||||
| Surya |
| Recall | |||||
| SDK | Release | Cutoff | Qwen3 | GPT 5.5 | Opus 4.8 |
| Daft | 2022-08 | Pre | |||
| LanceDB | 2023-03 | Pre | |||
| cuallee | 2023-01 | Pre | |||
| Docling | 2024-07 | Pre | |||
| Marker | 2024-02 | Pre | |||
| SDK | Category | p@1 | p@10 | CL | AE | ID | OR |
|---|---|---|---|---|---|---|---|
| cuallee | Data Valid. | ||||||
| LanceDB | Vector DB | ||||||
| Marker | Doc Proc. | ||||||
| Surya | Comp. Vision | ||||||
| Daft | DataFrame | ||||||
| Vortex | Scientific |
| SDK | Category | p@1 | p@10 | CL | AE | ID | OR |
|---|---|---|---|---|---|---|---|
| cuallee | Data Valid. | ||||||
| LanceDB | Vector DB | ||||||
| Marker | Doc Proc. | ||||||
| Surya | Comp. Vision | ||||||
| Daft | DataFrame | ||||||
| Vortex | Scientific |
| Config. | p@1 | CL | ID | Conv. (steps) |
|---|---|---|---|---|
| Flat GRPO | ||||
| Curric. only | ||||
| Options only | ||||
| Full (Option-GRPO) |
| Config. | p@1 |
|---|---|
| Option level | |
| Level 0 only | |
| Levels 0 + 1 | |
| Full (Levels 0 + 1 + 2) | |
| Reward | |
| Binary reward only | |
| Source Target | Daft | LanceDB | cuallee |
|---|---|---|---|
| No transfer (from scratch) | |||
| Daft | – | ||
| LanceDB | – | ||
| cuallee | – |
| Method | p@1 | p@10 | Critic | CL | ID |
|---|---|---|---|---|---|
| CodeRL-adapted ∗ | Yes | ||||
| PPOCoder-adapted ∗ | Yes | ||||
| StepCoder-adapted ∗ | Yes | ||||
| ExploraCoder | No | ||||
| ReCode | No | ||||
| Flat GRPO | No |
| Method | Recall | Unseen | p@1 | p@10 | |
|---|---|---|---|---|---|
| Frozen Qwen3-32B | |||||
| GPT-4o | |||||
| Claude 3.5 Sonnet | |||||
| Gemini 2.0 Flash | |||||
| GPT 5.5 | Table 15 | ||||
| Claude Opus 4.8 | Table 15 | ||||
| SDK | Metric | GPT 5.5 | Opus 4.8 | Opt-GRPO |
|---|---|---|---|---|
| Surya | pass@1 | |||
| Surya | pass@10 | |||
| Surya | Ref. p@1 | |||
| Tracecat | pass@1 | |||
| Tracecat | pass@10 | |||
| Tracecat | Ref. p@1 |
| SDK | Frozen | ExpC. | ReCode | Flat GRPO | Opt-GRPO | vs ReCode | Category |
|---|---|---|---|---|---|---|---|
| cuallee | Data Valid. | ||||||
| LanceDB | Vector DB | ||||||
| Marker | Doc Proc. | ||||||
| Surya | Comp. Vision | ||||||
| Daft | DataFrame | ||||||
| Vortex | Scientific |
| Configuration | p@1 | p@10 |
|---|---|---|
| RAG only (32B) | ||
| DocPrompting only (32B) | ||
| Option-GRPO (no retrieval) | ||
| Option-GRPO + RAG | ||
| Option-GRPO + DocPrompting |
| Phase | Steps | Smp/Step | GPU-hrs |
| Graph construction | – | – | |
| Option discovery | – | – | |
| Property test gen. | – | – | |
| Level 0 training | |||
| Level 1 training | |||
| Level 2 training |
| Total steps | GPU-hr | Wall (h) | Flat | Opt | |
|---|---|---|---|---|---|
| (default) | |||||
| Base | Params | Mem | Flat | Opt | |
|---|---|---|---|---|---|
| Qwen3- B | B | GB | |||
| Qwen3- B | B | GB | |||
| Qwen3- B | B | GB |
Code Generation
Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.
See all 348 papers