BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
Organizations: Google Research · Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign · Google DeepMind · Georgia Institute of Technology College of Computing
Abstract
Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks (), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.
Figures & tables
| Line of work | Sandbox environment | |||||
| Work | Synthetic | Training | Multi-file | Unrestricted | Agentic | Difficulty |
| data gen. | framework | repository | shell | evaluation | scaling | |
| MLAgentBench ( Huang et al., 2023 ) | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
| MLGym ( Nathani et al., 2025 ) | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ |
| MLE-Dojo ( Qiang et al., 2025 ) | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
| MLE-Smith ( Qiang et al., 2026b ) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| 1 Bug | 5 Bugs | 10 Bugs | 50 Bugs | |||||||||
| Model | Full Repair | Recovery | Percentile | Full Repair | Recovery | Percentile | Full Repair | Recovery | Percentile | Full Repair | Recovery | Percentile |
| Reference | ||||||||||||
| Clean repository (no bugs) | – | – | 49.1 | – | – | 49.1 | – | – | 49.1 | – | – | 52.0 |
| Frontier Models | ||||||||||||
| Claude Opus 5.5 | 95.5 | 99.8 | 57.2 | 31.8 | 88.4 | 44.6 | 13.6 | 92.2 | 46.7 | 9.1 | 77.8 | 43.9 |
| Claude Sonnet 5.5 | 90.9 | 95.5 | 54.0 | 27.3 | 85.5 | 42.4 | 31.8 | 86.4 | 43.6 | 0.0 | 64.3 | 30.6 |
| SWE-bench Lite | LiveCodeBench v6 | |||||||||
| Model | Resolved (%) | Fail-to-Pass Tests (%) | Tokens / Ep. (k) | Tool-Call Success (%) | vs. Qwen3.5-9B | Pass@1 All (%) | Pass@1 Easy (%) | Pass@1 Medium (%) | Pass@1 Hard (%) | vs. Qwen3.5-9B |
| Base Model | ||||||||||
| Qwen3.5-9B | 24.3 | 26.4 | 794 | 75.5 | – | 69.3 | 97.4 | 76.5 | 35.7 | – |
| Ours: Curriculum Learning with BuggyMLE (9B) | ||||||||||
| SFT Only (9B) | 18.3 | 23.0 | 361 | 95.1 | – | 61.3 | 94.7 | 65.7 | 25.9 | – |
| SFT+RL Curriculum: | ||||||||||
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
| Group | Feature (p50) | Description |
| Repo Scale | lines_of_code | Total source lines |
| file_count | Number of source files | |
| max_dir_depth | Deepest directory nesting | |
| Dependency | dependency_count | Number of inter-file dependencies |
| max_dep_chain | Longest dependency chain | |
| Patch | files_changed | Files modified by the bug fix |
| Benchmark | Tasks | Task type | Language | CCI | Solve rate |
| SWE-bench Verified | 500 | Repo issue repair | Python | 0.00 | 0.71 |
| FrontierCS Research | 61 | Research code generation | Python | 0.31 | 0.46 |
| FrontierCS Algo | 188 | Algorithmic | C++ | 0.38 | 0.39 |
| SWE-bench Pro | 731 | Long-horizon repo repair | Multi | 0.48 | 0.31 |
| FrontierSWE | 17 | From-scratch engineering | Multi | 0.58 | 0.30 |
| DeepSWE | 113 | Long-horizon engineering | Multi | 0.68 | 0.27 |
| Feature ( ) | Raw skew | skew | Improved? |
| lines_of_code | 1.70 | yes | |
| file_count | 1.46 | yes | |
| max_dir_depth | 0.27 | yes | |
| dependency_count | 1.14 | yes | |
| max_dep_chain | 2.23 | 1.26 | yes |
| files_changed | 2.66 | 1.32 | yes |
| Approach | Params | Predicts in | LOBO MAE |
| 3 complexity groups (logistic regression) | 4 | ✓ | 0.115 |
| + model-family fixed effects | 17 | ✓ | 0.130 |
| Bug patch only | 2 | ✓ | 0.166 |
| Patch + dependency | 3 | ✓ | 0.167 |
| + intelligence index | 5 | ✓ | 0.173 |
| Intercept only (baseline) | 1 | ✓ | 0.227 |
| task | source |
| airlines-dataset-to-predict-a-delay | Synthetic Kaggle |
| arabic-company-reviews | Synthetic Kaggle |
| bbc-news-articles | Synthetic Kaggle |
| car-dataset | Synthetic Kaggle |
| civil-engineering-global-project-dataset | Synthetic Kaggle |
| corona-virus-report | Synthetic Kaggle |
| corpus | trains | source | rows | tasks | hunks | reasoning | gate | validation |
| Teacher | All Difficulties, step 1 | teacher models | 2,724 | 128 | 1–52 (10) | 0 | 98,304 | 85 / 14 |
| RFT | All Difficulties, step 2 | 27B sibling | 411 | 61 | 1–34 (5) | 250 | 98,304 | 34 / 19 |
| Easy Only ( 5 hunks) | Easy Only | teacher models | 936 | 130 | 1–5 (1) | 0 | 49,152 | 63 / 12 |
| Hard Only ( 30 hunks) | Hard Only | teacher models | 628 | 107 | 30–52 (31) | 0 | 98,304 | 59 / 34 |
| teacher model | Teacher corpus | Hard Only corpus |
| Gemini 3.6 Flash ( Google DeepMind, 2026b ) | 1,012 | 222 |
| Claude Opus 4.8 ( Anthropic, 2026b ) | 654 | 139 |
| Claude Opus 5 ( Anthropic, 2026c ) | 402 | 101 |
| Claude Sonnet 5 ( Anthropic, 2026f ) | 333 | 79 |
| Gemini 3.7 Flash ( Google DeepMind, 2026c ) | 323 | 87 |
| total | 2,724 | 628 |
| band | train instances | train hunks | eval instances | eval repositories | eval hunks |
| 1–5 hunks | 40 | 1–5 (1) | 31 | 26 | 1–5 |
| 10–12 hunks | 40 | 10–12 (10) | 32 | 23 | 10–11 |
| 30–34 hunks | 40 | 30–34 (32) | 29 | 25 | 30–32 |
| repository | modality | 1–5 | 10–12 | 30–34 |
| agriculture-crop-images | image | 1 | 11 | 32 |
| agriculture-crop-yield | tabular | 1 | 10 | 31 |
| arabic-customer-reviews | text | 5 | 10 | 32 |
| arabic-sentiment-twitter-corpus | text | 1 | 12 | 32 |
| bbc-news | text | 1 | 10 | 31 |
| bitcoin-tweets | text | 1 | 10 | 32 |
| competition | Lite-Original | 10 Bugs | 5 Bugs | 1 Bug |
| aerial-cactus-identification † | 18 | 10 | 5 | 1 |
| aptos2019-blindness-detection | 24 | 13 | 5 | 1 |
| denoising-dirty-documents † | 22 | 15 | 8 | 2 |
| detecting-insults-in-social-commentary † | 19 | 12 | 6 | 2 |
| dog-breed-identification | 33 | 23 | 9 | 2 |
| dogs-vs-cats-redux-kernels-edition | 27 | 17 | 6 | 2 |
| set | tasks | held-out | BuggyMLEBench-Lite |
| Teacher SFT (train) | 128 | 0 | 0 |
| Teacher SFT (validation) | 14 | 0 | 0 |
| RFT (train) | 61 | 0 | 0 |
| RFT (validation) | 19 | 0 | 0 |
| Easy Only SFT (validation) | 12 | 0 | 0 |
| Hard Only SFT (train) | 107 | 0 | 0 |
| competition | clean | oracle | (%) | |
| aerial-cactus-identification | 0.99992 | 0.99992 | 0.000 | |
| aptos2019-blindness-detection | 0.83104 | 0.83104 | 0.000 | |
| denoising-dirty-documents | 0.04195 | 0.04195 | 0.002 | |
| detecting-insults-in-social-commentary | 0.80907 | 0.80907 | 0.000 | |
| dog-breed-identification | 3.06187 | 3.06187 | 0.000 | |
| dogs-vs-cats-redux-kernels-edition | 0.15686 | 0.15686 | 0.000 |
| Stage | Practice |
| Data | Keep a true holdout; fit scalers on training data only; apply the same preprocessing to train and test. |
| Alignment | Permute features and labels together; keep ids and predictions in one ordering; match a positional submission’s row order. |
| Training | Re-assert train mode in every training loop and eval mode for inference; rescale a scheduler’s horizon when stepping it per batch; zero gradients every step; freeze only pretrained weights; size the batch to the update budget. |
| Evaluation | Optimize the loss the metric grades; match the checkpoint comparison to the metric’s direction; compute the metric on the full holdout; pass the explicit label set. |
| Output | Match the output activation and clipping to the target range; write predictions with enough precision. |
| Step | Data | Length | Objective |
| SFT, teacher | 2,724 evaluator-verified repair trajectories from frontier teacher models, 128 tasks, no reasoning | 682 optimizer steps (2 epochs, batch size 8), lr | next-token loss, 98,304-token context |
| SFT, RFT | 411 evaluator-verified trajectories sampled from our SFT’d Qwen3.8-27B, with reasoning (41% of trained tokens inside <think> ), 61 tasks | 103 optimizer steps (2 epochs), lr | same adapter, continued |
| RL, 1–5 hunks | 40 buggy repositories, one instance each with 1–5 diff hunks (median 1) | 3 rounds 48 = 144 steps | mlperf_converge |
| RL, 10–12 hunks | the same 40 repositories, each re-injected with 10–12 hunks | 100 steps | judge_progress |
| RL, 30–34 hunks | the same 40 repositories, re-injected with 30–34 hunks | 72 steps | judge_progress , KL 0.001 |
| Model | Stage | Steps | Turns | Context | Reward | KL |
| All Difficulties | 1–5 hunks, rounds 1–3 | 3 48 | 40 | 40,960 | mlperf_converge | 0 |
| 10–12 hunks | 100 | 40 | 40,960 | judge_progress | 0 | |
| 30–34 hunks | 72 | 80, then 60 | 81,920 | judge_progress | 0.001 | |
| Easy Only | 1–5 hunks | 150 | 40 | 40,960 | judge_progress | 0 |
| Hard Only | 30–34 hunks | 400 | 60 | 49,152 | judge_progress | 0 |
| Tool | Effect |
| bash(command, workdir) | runs a shell command in the container, under a per-call timeout |
| read_file(path) | returns a file’s contents |
| write_file(path, content) | creates or overwrites a file |
| modify_file(path, old, new) | replaces an exact substring of a file, so a single function can be patched without rewriting the file |
| list_dir(path) | lists a directory |
| finalize(verdict, description) | ends the episode; the verdict is recorded but not used in scoring |
| RL 1–5, 10–12 hunks | RL 30–34 hunks | BuggyMLEBench-Lite 1 / 5 / 10 / 50 Bugs | |
| turns per episode | 40 | 60 § | 40 / 60 / 80 / 80 ‡ |
| context / output per turn (tokens) | 40,960 / 4,096 | 81,920 / 4,096 † | 98,304 / 4,096 |
| malformed limit | 3 | 3 | the turn budget |
| sampling | temperature 1.0 | temperature 1.0 | temperature 0.7 |
| Model | Turns (1 / 5 / 10 / 50 Bugs) | Context and output |
| Claude Opus 5.5 ( Anthropic, 2026d ) | 80 / 80 / 80 / 80 | shared |
| Claude Sonnet 5.5 ( Anthropic, 2026g ) | 80 / 80 / 80 / 80 | shared |
| Claude Sonnet 5 ( Anthropic, 2026f ) | 80 / 80 / 80 / 80 | shared |
| Claude Sonnet 4.6 ( Anthropic, 2026e ) | 40 / 60 / 80 / 80 | 262k; 40,960 per turn |
| Gemini 3.8 Flash ( Google DeepMind, 2026d ) | 40 / 60 / 80 / 80 | 262k; 40,960 per turn |
| Claude Haiku 5.5 ( Anthropic, 2026a ) | 40 / 60 / 80 / 80 | shared |
| Variant | Model | Recovery (%) | Full Repair (%) | Bugs Fixed (%) | Improved Repo (%) | Ended with Finalize Tool (%) | Tool-Call Success (%) | Hit Turn Cap (%) | Leaderboard Percentile (%) | Tasks Above Clean Repo |
| Reference | Clean repository (1, 5, 10 Bugs) | – | – | – | – | – | – | – | 49.1 | – |
| 1 Bug | Claude Opus 5.5 | 99.8 | 95.5 | 95.5 | 100.0 | 100.0 | 96.7 | 0.0 | 57.2 | 14 |
| Claude Sonnet 5.5 | 95.5 | 90.9 | 90.9 | 95.5 | 100.0 | 91.8 | 0.0 | 54.0 | 11 | |
| Claude Sonnet 5 | 80.2 | 63.6 | 63.6 | 86.4 | 100.0 | 89.9 | 0.0 | 39.2 | 5 | |
| Claude Sonnet 4.6 | 30.9 | 27.3 | 27.3 | 27.3 | 72.7 | 95.9 | 0.0 | 17.5 | 1 | |
| Gemini 3.8 Flash | 27.3 | 27.3 | 27.3 | 27.3 | 90.9 | 97.1 | 0.0 | 15.6 | 3 |
| Frontier | Budget Models | ||||
| Task | Metric | Claude Sonnet 5 | Qwen3.5-9B | Ours (9B SFT+RL) | Ours Qwen3.5-9B |
| Code only (no tools, up to 10 debug attempts) 66 episodes per model | |||||
| Quality | Leaderboard Percentile | 61.2 | 16.4 | 17.9 | +1.5 |
| Quality | Valid Submission (%) | 100 | 61 | 50 | 10.6 |
| Efficiency | Input + Output Tokens / Ep. (k) | 13 | 78 | 65 | 12.7 † |
| Efficiency | Tool-Call Success (%) | – | – | – | – |