BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
Authors: John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu
Organizations: Google Research · Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign · Google DeepMind · Georgia Institute of Technology College of Computing
Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks (R2=0.951), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.
Figures & tables
Figure 1: Code complexity predicts task difficulty across BuggyMLEBench-Lite and eight public coding benchmarks. Top: An agent injects bugs into the clean repositories of all 22 BuggyMLEBench-Lite tasks to form the 1 Bug, 5 Bugs, 10 Bugs, and 50 Bugs tiers. (a) A representative BuggyMLEBench-Lite repository (Tabular Playground Series) is plotted at the CCI of its code after each injected bug. Its ROC AUC falls with each bug, dropping below the Kaggle competition leaderboard median after the first injection, with dots colored by the pipeline stage each bug modifies. (b) On the same CCI axis, gray points show eight public coding benchmarks at their average solve rate across all evaluated models , along with their linear fit ( R2=0.951 ; CCI uses no model information). The blue band spans judge-verified Full Repair across nine baseline models from Table 2 (excluding Qwen3.5-9B and its trained variants) on the four BuggyMLEBench-Lite tiers (all 22 tasks per tier, each plotted at its median CCI ), highlighting the median model (solid) and best model (dashed; Claude Opus 5.5). Solve rates decline with rising CCI along the same trajectory as the public benchmarks. We further verify this trend on 200 additional BuggyMLE tasks in Appendix A.5 .
Line of work
Sandbox environment
Work
Synthetic
Training
Multi-file
Unrestricted
Agentic
Difficulty
data gen.
framework
repository
shell
evaluation
scaling
MLAgentBench ( Huang et al., 2023 )
✗
✗
✗
✗
✓
✗
MLGym ( Nathani et al., 2025 )
✗
✓
✓
✗
✓
✗
MLE-Dojo ( Qiang et al., 2025 )
✗
✗
✗
✗
✓
✗
MLE-Smith ( Qiang et al., 2026b )
✓
✗
✗
✗
✗
✗
Table 1: Work on agentic MLE falls into three lines: synthetic data generators , which emit new tasks; training frameworks , which provide the harness in which an agent is trained or evaluated; and sandbox environments , which define what an agent may touch while solving. A check mark indicates the capability is present. BuggyMLE spans all three, and is the only work giving the agent an unrestricted shell over a multi-file repository together with explicit control over task difficulty.
Figure 2: The CCI robustly predicts average solve rates across eight public coding benchmarks. (a) Linear regressions of average solve rate on CCI : both the full fit (blue, R2=0.951 ) and all eight leave-one-benchmark-out refits (lavender) slope downward consistently. (b) Each held-out benchmark’s actual pass rate versus the pass rate predicted from the remaining seven benchmarks stays close to the identity line ( MAE=0.048 ); each benchmark retains the same color and marker across both panels.
Figure 3: BuggyMLE data curation workflow: every multi-bug repository is built from a vetted clean repository and decomposed into verified, individually impactful bugs. (a) Repository curation: an auditor keeps candidate ML repositories that run fast, give a stable metric across runs, and have a responsive metric (one that moves when the code changes). (b) Bug injection: an injector agent turns the clean repository into a mutated one at a target complexity (the CCI slider); a mutation is kept only if automated checks confirm it changes ML performance. (c) Bug attribution: in two passes, an oracle first cumulatively reverts mutations until baseline performance recovers (identifying a minimal reverting set), then applies each mutation alone onto the clean repository to measure isolated ML performance against the clean baseline (dashed), confirming impactful bugs and marking no-effect edits as decoys. (d) Evaluation: a solver’s fixed repository is graded against the clean repository and the oracle’s findings for correctness, recovered ML performance, and cheating (anti-bypass checks and an oracle-grounded LLM judge).
Figure 4: Distribution of tasks across data modalities. Proportion of tasks by modality for (a) SFT + RFT training, (b) RL training, and (c) the BuggyMLEBench-Lite evaluation benchmark, where image tasks form the largest share. Detailed recipe statistics are provided in Appendix Figure 15 .
Figure 5: Distribution of injected bug impacts across complexity tiers, pipeline stages, and defect categories. Proportion of bugs stratified by their isolated impact on the competition test metric. (a) Breakdown across complexity tiers, showing that mild and strong impacts dominate the 1, 5, and 10 Bugs tiers, whereas catastrophic failures are rare and execution crashes occur only at the 50 Bugs tier. (b) Distribution of unique bugs across the 1, 5, and 10 Bugs tiers by ML pipeline stage, and (c) across the six most frequent defect categories (total counts in parentheses; individual bugs may span multiple categories). A bug is classified as impactful if applying it in isolation shifts the graded test score beyond the task’s significance threshold, and is categorized by magnitude as mild ( ∣Δ∣<0.1 ), strong ( 0.1≤∣Δ∣<1 ), or catastrophic ( ∣Δ∣≥1 ). A crash fails to produce a valid submission file, whereas a decoy alters at most the validation score.
Figure 6: Representative examples of injected benchmark bugs. (a)–(c) Three injected defects from BuggyMLEBench-Lite spanning image, audio, and tabular tasks, showing removed lines (red) and added lines (green), including the inline comments generated during injection to rationalize each modification. Reported scores reflect each defect’s isolated impact on the competition test metric, alongside its 1–5 realism rating assigned by the LLM judge (RQ6, Section 5 ).
1 Bug
5 Bugs
10 Bugs
50 Bugs
Model
Full Repair ↑
Recovery ↑
Percentile ↑
Full Repair ↑
Recovery ↑
Percentile ↑
Full Repair ↑
Recovery ↑
Percentile ↑
Full Repair ↑
Recovery ↑
Percentile ↑
Reference
Clean repository (no bugs)
–
–
49.1
–
–
49.1
–
–
49.1
–
–
52.0
Frontier Models
Claude Opus 5.5
95.5
99.8
57.2
31.8
88.4
44.6
13.6
92.2
46.7
9.1
77.8
43.9
Claude Sonnet 5.5
90.9
95.5
54.0
27.3
85.5
42.4
31.8
86.4
43.6
0.0
64.3
30.6
Table 2: Evaluation results on the held-out BuggyMLEBench-Lite benchmark (22 tasks per difficulty tier; all values in percentages). Reference denotes the clean baseline repository prior to bug injection. The 50 Bugs tier expands each repository to roughly twice its original size with 23–60 injected bugs per task. Arrows indicate the direction of better performance. Shading, bold (best), and underline (second best) highlight rankings within the budget and 9B model comparison class. Model tiers and metrics are defined in Section 4 .
Figure 7: High-complexity curriculum stages drive turn efficiency. Training dynamics of the All Difficulties curriculum across Easy (1–5 hunks), Medium (10–12), and Hard (30–34) RL stages (Hard stage shaded). (a) Per-step training reward (gray), 8-step moving average (blue), held-out evaluation score on the corresponding difficulty band (lavender), and the selected evaluation checkpoint (dashed). Reward recovers within each stage and remains stable through the Hard stage. (b) Mean turns per episode with stage-wise linear trends relative to the turn budget (dashed). Episode length remains nearly flat during the Easy and Medium stages but drops sharply during the Hard stage without degrading reward. An earlier turn-budget adjustment during the Hard stage is analyzed in Appendix Figure 22 .
SWE-bench Lite
LiveCodeBench v6
Model
Resolved (%) ↑
Fail-to-Pass Tests (%) ↑
Tokens / Ep. (k) ↓
Tool-Call Success (%) ↑
Δ vs. Qwen3.5-9B
Pass@1 All (%) ↑
Pass@1 Easy (%) ↑
Pass@1 Medium (%) ↑
Pass@1 Hard (%) ↑
Δ vs. Qwen3.5-9B
Base Model
Qwen3.5-9B
24.3
26.4
794
75.5
–
69.3
97.4
76.5
35.7
–
Ours: Curriculum Learning with BuggyMLE (9B)
SFT Only (9B)
18.3
23.0
361
95.1
–
61.3
94.7
65.7
25.9
–
SFT+RL Curriculum:
Table 3: OOD generalization on general coding benchmarks (experimental settings in Section 4 ). Arrows indicate the direction of better performance, with shading, bold , and underline ranked across all rows. Δ denotes the paired difference relative to the untuned base model, where † indicates a 95% bootstrap confidence interval excluding zero.
Figure 8: Impact of metric recovery, fault localization, observability, and realism on bug resolution. Each row corresponds to one of the ten models audited at the per-bug level, ordered as in Table 2 , showing frontier models (lavender), budget baselines (gray), and our trained 9B models (blue). (a) 10 Bugs task solve rate under test-score recovery alone (open circles) versus judge-verified full repair requiring all injected bugs to be fixed (filled circles); the gap indicates tasks where the test metric recovered despite remaining bugs, which is widest for Claude Opus 5.5 and Sonnet 5.5. (b) Proportion of injected bugs modified (light) versus fully resolved (dark), showing that top agents (All Difficulties, Opus 5.5, and Sonnet 5.5) localize nearly all bugs and resolve most modified sites, whereas the 9B base model modifies roughly one-third. (c) Resolution rate for visible bugs (which degrade validation metrics) versus silent bugs (which affect only test predictions), demonstrating consistently lower resolution on silent bugs across all models. (d) Resolution rate stratified by LLM-judged realism score (4–5, 2–3, and 1). Panels (b)–(d) pool tasks across the 1, 5, and 10 Bugs tiers; all x-axes report percentages.
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Feature (p50)
Description
Repo Scale
lines_of_code
Total source lines
file_count
Number of source files
max_dir_depth
Deepest directory nesting
Dependency
dependency_count
Number of inter-file dependencies
max_dep_chain
Longest dependency chain
Patch
files_changed
Files modified by the bug fix
Appendix
Table 4: Raw features used in CCI .
Benchmark
Tasks
Task type
Language
CCI
Solve rate
SWE-bench Verified
500
Repo issue repair
Python
0.00
0.71
FrontierCS Research
61
Research code generation
Python
0.31
0.46
FrontierCS Algo
188
Algorithmic
C++
0.38
0.39
SWE-bench Pro
731
Long-horizon repo repair
Multi
0.48
0.31
FrontierSWE
17
From-scratch engineering
Multi
0.58
0.30
DeepSWE
113
Long-horizon engineering
Multi
0.68
0.27
Appendix
Table 5: Benchmarks ordered by CCI . Solve rates average all evaluated frontier models. SWE-Marathon counts the 17 of its 20 tasks with readable reference code.
Feature ( fi )
Raw skew
log(1+fi) skew
Improved?
lines_of_code
1.70
−0.64
yes
file_count
1.46
−0.05
yes
max_dir_depth
0.27
−0.07
yes
dependency_count
1.14
−0.10
yes
max_dep_chain
2.23
1.26
yes
files_changed
2.66
1.32
yes
Appendix
Table 6: Skewness before and after log(1+fi) .
Figure 9: Grouping the ten structural features into three dimensions eliminates severe within-group collinearity. Pairwise Pearson correlations across the eight public coding benchmarks (log-transformed task medians; blue: positive, lavender: negative). (a) Raw features exhibit near-collinearity within conceptual blocks (e.g., file_count vs. max_dir_depth , r=0.97 ; files_changed vs. hunks , r=0.93 ), which destabilizes an ungrouped regression. (b) Aggregating into Repo Scale, Dependency, and Bug Patch z-score averages lowers the maximum off-diagonal correlation to r=0.82 while keeping Bug Patch nearly orthogonal to both repository-level groups ( r=0.15 and −0.03 ).
Figure 10: All three CCI group coefficients preserve their sign and relative magnitude across every LOBO refit. Binomial GLM log-odds coefficients when each of the eight benchmarks is held out in turn (lavender dots and horizontal offsets, exact values at right) compared against the full eight-benchmark fit (vertical blue line; zero shown dashed) for (a) Repo Scale ( +0.64 ), (b) Dependency Complexity ( −0.34 ), and (c) Bug/Patch Complexity ( −0.99 ). No single benchmark drives the fit or flips any coefficient’s sign, and Bug/Patch Complexity remains the dominant predictor in every refit ( −0.91 to −1.18 ).
Approach
Params
Predicts in [0,1]
LOBO MAE
3 complexity groups (logistic regression)
4
✓
0.115
+ model-family fixed effects
∼ 17
✓
0.130
Bug patch only
2
✓
0.166
Patch + dependency
3
✓
0.167
+ intelligence index
5
✓
0.173
Intercept only (baseline)
1
✓
0.227
Appendix
Table 7: LOBO prediction error by model specification. MAE: mean absolute error of the held-out solve rate; OLS: ordinary least squares; WLS: weighted least squares. Linear specifications can predict solve rates outside [0,1] and are clipped before scoring. Bold : lowest error among specifications whose predictions stay within [0,1] .
Figure 11: Bug/patch complexity alone explains three-quarters of benchmark difficulty variance, and combining all three groups reaches R2=0.951 . Average benchmark pass rate across evaluated models (gray points, directly labeled) regressed via OLS (blue line) onto each min–max scaled score in [0,1] : (a) Repo Scale ( R2=0.047 ), (b) Dependency Complexity ( R2=0.045 ), (c) Bug/Patch Complexity ( R2=0.756 ), and (d) the full three-group CCI ( R2=0.951 ). Repository size and dependency structure are weak standalone predictors because small single-file algorithmic tasks (e.g., FrontierCS Algo) can still be difficult, whereas combining them with patch scope aligns all eight benchmarks tightly along a single linear frontier.
Figure 12: Model capability discriminates within individual benchmarks but explains none of the variance across benchmarks ( R2=0.001 ). Each point plots one (model, benchmark) pair ( n=100 pairs with an available index) across the eight public coding benchmarks (distinct marker and color per benchmark), comparing external Model Intelligence Index against benchmark pass rate. Although higher-intelligence models generally score higher within a fixed benchmark, the pooled least-squares trend (dashed black line) is flat because benchmark-level floor and ceiling effects (from near-saturated SWE-bench Verified down to unsaturated ProgramBench) dominate cross-benchmark variation.
Figure 13: Scaling the injected hunk budget on a fixed 200-task pool predictably lowers solve rates across six frontier and budget solvers. Solve rate versus projected CCI at 1, 10, and 30 injected diff hunks ( 1h , 10h , 30h vertical guides), with Claude Opus 4.8 held fixed as injector, oracle, and reviewer. Black line and gray dots show the eight public human benchmarks and their linear fit ( R2=0.951 ); colored markers and least-squares lines track each solver across hunk budgets (solid: Claude family; dashed: Gemini family). A task counts as solved only when it achieves full metric recovery and the oracle-grounded reviewer confirms that the verified bugs are fixed. Gemini 3.6 Flash’s 30-hunk cell is omitted due to incomplete coverage (Appendix A.5 ).
Figure 14: Negative CCI scaling and solver rankings hold across evaluation strictness levels and repository sources. Solver solve rates at 1, 10, and 30 injected hunks across human-authored PyHealth repositories ( top , (a)–(c) ; 90 tasks per cell) and synthetic Kaggle repositories ( bottom , (d)–(f) ; 190 tasks per cell) as the success criterion is progressively tightened: (a, d) metric recovery alone ( Metric Recovery ), (b, e) additionally requiring the oracle-grounded reviewer to confirm that the patch fixes the verified bugs ( + Fix Addresses Bug ), and (c, f) further requiring a principally valid repair ( + Principally Valid ; Section 3.2 ). Filtering for causal bug resolution removes superficial metric recoveries without altering the downward CCI slope, while requiring principal validity changes solve rates minimally and preserves solver ordering. Black line and gray dots denote the human-benchmark reference frontier; incomplete runs are excluded (Appendix A.5 ).
Figure 15: Training corpus statistics: most repositories supply SFT data, and trajectories span all three difficulty bands. (a) Allocation of the 200 training repositories: 142 supply SFT trajectories, 37 are held out for validation, 5 are used exclusively in RL, and 16 are excluded prior to training. (b) Distribution of the 3,135 SFT + RFT trajectories across bug-difficulty bands: 1,328 easy ( ≤ 5 hunks; 42%), 1,100 medium (6–29 hunks; 35%), and 707 hard ( ≥ 30 hunks; 23%); the Easy Only (936 rows) and Hard Only (628 rows) corpora each train on a single band.
task
source
airlines-dataset-to-predict-a-delay
Synthetic Kaggle
arabic-company-reviews
Synthetic Kaggle
bbc-news-articles
Synthetic Kaggle
car-dataset
Synthetic Kaggle
civil-engineering-global-project-dataset
Synthetic Kaggle
corona-virus-report
Synthetic Kaggle
Appendix
Table 8: The 37 held-out tasks and the source family each comes from.
corpus
trains
source
rows
tasks
hunks
reasoning
gate
validation
Teacher
All Difficulties, step 1
teacher models
2,724
128
1–52 (10)
0
98,304
85 / 14
RFT
All Difficulties, step 2
27B sibling
411
61
1–34 (5)
250
98,304
34 / 19
Easy Only ( ≤ 5 hunks)
Easy Only
teacher models
936
130
1–5 (1)
0
49,152
63 / 12
Hard Only ( ≥ 30 hunks)
Hard Only
teacher models
628
107
30–52 (31)
0
98,304
59 / 34
Appendix
Table 9: The four SFT corpora. Hunks: range (median) per trajectory. Reasoning: trajectories with a reasoning block. Gate: longest trajectory kept, in tokens. Validation: trajectories / tasks.
teacher model
Teacher corpus
Hard Only corpus
Gemini 3.6 Flash ( Google DeepMind, 2026b )
1,012
222
Claude Opus 4.8 ( Anthropic, 2026b )
654
139
Claude Opus 5 ( Anthropic, 2026c )
402
101
Claude Sonnet 5 ( Anthropic, 2026f )
333
79
Gemini 3.7 Flash ( Google DeepMind, 2026c )
323
87
total
2,724
628
Appendix
Table 10: Teacher trajectories by teacher model for the Teacher and Hard Only corpora (the Easy Only corpus draws from the same five teacher models).
band
train instances
train hunks
eval instances
eval repositories
eval hunks
1–5 hunks
40
1–5 (1)
31
26
1–5
10–12 hunks
40
10–12 (10)
32
23
10–11
30–34 hunks
40
30–34 (32)
29
25
30–32
Appendix
Table 11: The three RL pools. Hunks: range and median of injected diff hunks per instance.
repository
modality
1–5
10–12
30–34
agriculture-crop-images
image
1
11
32
agriculture-crop-yield
tabular
1
10
31
arabic-customer-reviews
text
5
10
32
arabic-sentiment-twitter-corpus
text
1
12
32
bbc-news
text
1
10
31
bitcoin-tweets
text
1
10
32
Appendix
Table 12: The 40 RL repositories and the injected diff hunks of the training instance each contributes to the 1–5, 10–12, and 30–34-hunk pools.
competition
Lite-Original
10 Bugs
5 Bugs
1 Bug
aerial-cactus-identification †
18
10
5
1
aptos2019-blindness-detection
24
13
5
1
denoising-dirty-documents †
22
15
8
2
detecting-insults-in-social-commentary †
19
12
6
2
dog-breed-identification
33
23
9
2
dogs-vs-cats-redux-kernels-edition
27
17
6
2
Appendix
Table 13: BuggyMLEBench-Lite tasks and injected diff hunks per tier. † : clean repository above the leaderboard median.
set
tasks
∩ held-out
∩ BuggyMLEBench-Lite
Teacher SFT (train)
128
0
0
Teacher SFT (validation)
14
0
0
RFT (train)
61
0
0
RFT (validation)
19
0
0
Easy Only SFT (validation)
12
0
0
Hard Only SFT (train)
107
0
0
Appendix
Table 14: Task overlaps with the held-out tasks and BuggyMLEBench-Lite.
competition
clean
oracle
∣Δ∣ (%)
aerial-cactus-identification
↑
0.99992
0.99992
0.000
aptos2019-blindness-detection
↑
0.83104
0.83104
0.000
denoising-dirty-documents
↓
0.04195
0.04195
0.002
detecting-insults-in-social-commentary
↑
0.80907
0.80907
0.000
dog-breed-identification
↓
3.06187
3.06187
0.000
dogs-vs-cats-redux-kernels-edition
↓
0.15686
0.15686
0.000
Appendix
Table 15: Oracle repair on BuggyMLEBench-Lite. Clean: the graded score of the clean repository. Oracle: the graded score after every injected bug is reversed. ∣Δ∣ : their relative difference, in percent. Arrows give the metric’s better direction.
Figure 16: Nearly three-quarters of raw Lite-Original injections independently degrade graded test performance across every pipeline stage and defect category. Isolated test-metric impact of all 364 bugs in the unpruned Lite-Original pool, stratified by (a) ML pipeline layer and (b) the nine most frequent defect types (total bug counts in parentheses; a multi-aspect bug may appear in multiple types). Horizontal bars partition bugs into catastrophic ( ∣Δ∣≥1 , dark blue), strong ( 0.1≤∣Δ∣<1 , medium blue), and mild ( ∣Δ∣<0.1 , light blue) test-score degradation, below-threshold shifts (pale blue), validation-only decoys (lavender), and execution crashes (charcoal). Filtering out decoys when constructing the 1 Bug, 5 Bugs, and 10 Bugs tiers removes nearly all non-impactful patches (Figure 17 e).
Figure 17: Automated realism audit, human expert calibration, oracle decoy filtering, and downstream solver resolution on BuggyMLEBench-Lite. (a) LLM-judged 1–5 realism distribution across all 364 raw Lite-Original patches (mean 3.40/5 ), ranging from plausible oversights (score 5) to overt sabotage (score 1). (b) Realism distribution across the 222 unique evaluated bugs in the 1 Bug, 5 Bugs, and 10 Bugs tiers, where 93% score ≥2 and stratum shares remain consistent across tiers. (c) Breakdown across the four task modalities (patch count and mean realism μ in parentheses), showing the share accompanied by a plausible rationalizing comment versus overt sabotage ( 1/5 ). (d) Blind human calibration by an ML practitioner on 30 randomly sampled defects across the evaluated tiers, reporting real-repository plausibility and agreement with the LLM stratum (90% on both). (e) Oracle-measured test impact versus decoys and below-threshold shifts across Lite-Original and all four BuggyMLEBench-Lite tiers. (f) Line-level bug resolution by Claude Sonnet 5 pooled across the 1, 5, and 10 Bugs tiers, stratified by realism score.
Stage
Practice
Data
Keep a true holdout; fit scalers on training data only; apply the same preprocessing to train and test.
Alignment
Permute features and labels together; keep ids and predictions in one ordering; match a positional submission’s row order.
Training
Re-assert train mode in every training loop and eval mode for inference; rescale a scheduler’s horizon when stepping it per batch; zero gradients every step; freeze only pretrained weights; size the batch to the update budget.
Evaluation
Optimize the loss the metric grades; match the checkpoint comparison to the metric’s direction; compute the metric on the full holdout; pass the explicit label set.
Output
Match the output activation and clipping to the target range; write predictions with enough precision.
Appendix
Table 16: Practices encoded by reverting the injected bugs.
Figure 18: Injected bugs rated 1/5 for realism represent overt sabotage justified by fabricated comments. (a)–(c) Three randomly sampled score-1 defects from BuggyMLEBench-Lite spanning audio, tabular, and text tasks ( Bird Songs , Tabular May 2022 , and Detecting Insults ). Left column reports task modality, pipeline stage, bug summary, the code-level rationale for its score of 1 ( Rating ), and isolated test-metric degradation (blue pill; validation unchanged marks silent bugs). Right column shows the unified diff (pink: removed; green: added), where each mutation relies on an invented premise about the dataset or library API.
Figure 19: Injected bugs rated 2–3/5 apply recognizable ML operations in questionable contexts. (a)–(c) Three randomly sampled score 2–3 defects across image, tabular, and text tasks ( Aerial Cactus , Tabular May 2022 , and Spooky Authors ), formatted as in Figure 18 . Each mutation invokes a real ML concept (batch shuffling for batch normalization, rounding probabilities to match binary labels, or strided subsampling), but applies it in a way a careful code review would flag (e.g., permuting inputs without labels or rounding predictions under a ranking metric).
Figure 20: Injected bugs rated 4–5/5 closely resemble subtle, honest MLE mistakes. (a)–(c) Three randomly sampled score 4–5 defects across image, audio, and text tasks ( Histopathology , Right Whale Calls , and Text Norm. Russian ), formatted as in Figure 18 . Examples include emitting single- argsort sort indices in place of prediction ranks, replacing ReLU with saturating Sigmoid activations across convolutional blocks, and a shared additive smoothing constant that silently alters a threshold gate in another module; all three compile cleanly and mirror real research bugs.
Step
Data
Length
Objective
SFT, teacher
2,724 evaluator-verified repair trajectories from frontier teacher models, 128 tasks, no reasoning
682 optimizer steps (2 epochs, batch size 8), lr 10−4
next-token loss, 98,304-token context
SFT, RFT
411 evaluator-verified trajectories sampled from our SFT’d Qwen3.8-27B, with reasoning (41% of trained tokens inside <think> ), 61 tasks
103 optimizer steps (2 epochs), lr 3×10−5
same adapter, continued
RL, 1–5 hunks
40 buggy repositories, one instance each with 1–5 diff hunks (median 1)
3 rounds × 48 = 144 steps
mlperf_converge
RL, 10–12 hunks
the same 40 repositories, each re-injected with 10–12 hunks
100 steps
judge_progress
RL, 30–34 hunks
the same 40 repositories, re-injected with 30–34 hunks
72 steps
judge_progress , KL 0.001
Appendix
Table 17: The five training steps of the All Difficulties model.
Model
Stage
Steps
Turns
Context
Reward
KL
All Difficulties
1–5 hunks, rounds 1–3
3 × 48
40
40,960
mlperf_converge
0
10–12 hunks
100
40
40,960
judge_progress
0
30–34 hunks
72
80, then 60
81,920
judge_progress
0.001
Easy Only
1–5 hunks
150
40
40,960
judge_progress
0
Hard Only
30–34 hunks
400
60
49,152
judge_progress
0
Appendix
Table 18: Settings of every RL stage. Steps: the checkpoint used. All stages use 10 tasks × 8 rollouts per step, temperature 1.0, learning rate 3×10−5 , and 4,096 output tokens per turn.
Figure 21: RL training reward and held-out validation curves across all three curriculum ablations show consistent learning followed by late-stage overfitting. Per-step training reward (thin gray), 8-step moving average (blue), held-out evaluation score on each run’s target difficulty band (lavender markers), and the exported checkpoint selected for final evaluation (vertical dashed line) for (a) Easy Only (1–5 hunks, peaking at step 150 before a 50-step unused extension), (b) Hard Only (30–34 hunks, evaluated at step 400 prior to late collapse), and (c) All Difficulties across cumulative Easy, Medium, and Hard stages (selected at Hard step 72). In (c) , the Easy stage uses mlperf_converge while Medium and Hard use judge_progress , so vertical reward levels are comparable within, rather than across, that stage boundary.
Figure 22: Cutting the turn budget from 80 to 60 mid-stage costs reward briefly, then the agent learns to finish within it. The full hard stage of the All Difficulties run, including its first 20 steps at an 80-turn budget (shaded). (a) Mean turns per episode with each budget (dashed). (b) Share of episodes that hit the turn cap: it jumps when the budget drops, then falls steadily. (c) Training reward per step (gray) and 8-step mean (blue), held-out score (lavender), and the checkpoint we evaluate (dashed).
Tool
Effect
bash(command, workdir)
runs a shell command in the container, under a per-call timeout
read_file(path)
returns a file’s contents
write_file(path, content)
creates or overwrites a file
modify_file(path, old, new)
replaces an exact substring of a file, so a single function can be patched without rewriting the file
list_dir(path)
lists a directory
finalize(verdict, description)
ends the episode; the verdict is recorded but not used in scoring
Appendix
Table 19: The agent’s tools. Searching the repository is done through bash .
Table 21: Evaluation setup. Context and output list total context and per-turn output limits where they differ from the shared harness (Table 20 ).
Figure 23: Curriculum RL replaces blind pipeline rewrites with targeted multi-file diagnosis and verification on a 5 Bugs task. Side-by-side episode trace on Aerial Cactus (each model’s reported sample; T : turn; quotes verbatim from model reasoning). (a) Untuned Qwen3.5-9B spends 80 turns rewriting files and standalone scripts, fixing 3/5 bugs without finalizing (recovery 0.48; red boxes mark failure modes). (b) All Difficulties diagnoses four suspects across modules, reverts them in one turn, catches the fifth bug from a chance-level validation AUC, and finalizes in 14 turns with 5/5 bugs fixed (recovery 1.00; green boxes mark diagnostic reasoning and repairs). (c) Per-bug judge audit and fixing turn.
Figure 24: On a silent 1 Bug task, the RL agent inspects submission code directly whereas the base model overfits validation noise. Side-by-side episode trace on Detecting Insults (layout as in Figure 23 ). (a) Untuned Qwen3.5-9B misinterprets a 0.005 validation gap as the defect, tunes vectorizer hyperparameters across 43 turns without inspecting submit.py , and finalizes with 0/1 bugs fixed (recovery 0.72). (b) All Difficulties locates and removes the injected test-score Gaussian noise in submit.py by turn 6, verifies on main.py , and finalizes in 12 turns with full recovery (1.00). (c) Per-bug judge audit.
Figure 25: On a 10 Bugs repository, the RL agent systematically clears all ten defects across library modules while the base model rewrites main.py around them. Side-by-side episode trace on Spooky Authors (layout as in Figure 23 ). (a) Untuned Qwen3.5-9B repeatedly rewrites main.py from scratch over 60 turns without editing any underlying library/ module, fixing 0/10 bugs (recovery 0.60). (b) All Difficulties patches seven bugs across eight files in a single turn ( T6 ), uses residual validation log-loss gaps against the baseline to isolate the remaining three subtle hyperparameter and data-slice bugs ( T13--T16 ), and resolves all 10/10 bugs in 20 turns (recovery 0.99). (c) Per-bug judge audit across all ten injected defects.
Variant
Model
Recovery (%) ↑
Full Repair (%) ↑
Bugs Fixed (%) ↑
Improved Repo (%) ↑
Ended with Finalize Tool (%) ↑
Tool-Call Success (%) ↑
Hit Turn Cap (%) ↓
Leaderboard Percentile (%) ↑
Tasks Above Clean Repo ↑
Reference
Clean repository (1, 5, 10 Bugs)
–
–
–
–
–
–
–
49.1
–
1 Bug
Claude Opus 5.5
99.8
95.5
95.5
100.0
100.0
96.7
0.0
57.2
14
Claude Sonnet 5.5
95.5
90.9
90.9
95.5
100.0
91.8
0.0
54.0
11
Claude Sonnet 5
80.2
63.6
63.6
86.4
100.0
89.9
0.0
39.2
5
Claude Sonnet 4.6
30.9
27.3
27.3
27.3
72.7
95.9
0.0
17.5
1
Gemini 3.8 Flash
27.3
27.3
27.3
27.3
90.9
97.1
0.0
15.6
3
Appendix
Table 22: All metrics on BuggyMLEBench-Lite , 22 episodes per model and tier. Arrows, shading, and the Reference rows follow Table 2 . Bugs fixed: injected bugs judged fixed, whether or not the repository still runs (not judged for Qwen3.8-27B on 50 Bugs). Improved repo: the repository beats its injected start. Ended with finalize tool: episodes terminating via an explicit finalize call. Tool-call success: 1− failed/attempted actions. Hit turn cap: the episode used its whole turn budget. Tasks above clean repo: tasks where the repaired code ranks above the clean repository.
Figure 26: Test-metric recovery substantially overstates true bug repair as defect count grows, especially for frontier models. Task solve rate on BuggyMLEBench-Lite under metric recovery alone ( recovery≥0.99 , open circles) versus strict judge-verified full repair requiring every injected defect to be resolved (filled circles), across (a) 1 Bug, (b) 5 Bugs, (c) 10 Bugs, and (d) 50 Bugs (22 tasks per panel; lavender: frontier models, gray: budget baselines, blue: our trained 9B models). The two criteria coincide on 1 Bug, where fixing the sole bug is necessary to recover the score, but diverge sharply from 5 Bugs onward as strong models compensate for unfixed or silent defects via modeling improvements elsewhere (e.g., Claude Opus 5.5 reaches 64% score recovery on 10 Bugs vs. 14% verified full repair).
Figure 27: Curriculum RL halves episode length and shifts tool usage from exploratory shell commands toward structured file reads and patches. Episode-level execution statistics across the 22 tasks of the 10 Bugs tier (lavender: frontier models, gray: budget baselines, blue: our trained 9B models). (a) Turns per episode and (b) total tool calls per episode (shaded bar: interquartile range; vertical tick: median; dot: mean; 80-turn budget). Whereas Qwen3.5-9B Base and SFT Only routinely exhaust the 80-turn cap, All Difficulties finishes in a median of ∼27 turns, comparable to Claude Haiku 5.5 and Sonnet 5.5. (c) Share of tool calls by tool type: Claude Code baselines execute predominantly through bash (67–96%), whereas All Difficulties balances bash (25%), read_file (32%), and modify_file (36%) while eliminating the whole-file write_file calls seen in the untuned base model.
Figure 28: High-complexity RL closes the gap with frontier models across pipeline stages, defect categories, and data modalities, particularly on output-layer bugs. Percentage of individual injected bugs judged resolved, pooled across the 1 Bug, 5 Bugs, and 10 Bugs tiers (darker blue indicates higher resolution rate), broken down by (a) ML pipeline layer, (b) the eight most frequent defect categories, and (c) competition data modality. Output-formatting defects are the hardest layer for base, SFT, and most external models (4% for Qwen3.5-9B Base, 19% for Claude Sonnet 5, 69% for Sonnet 5.5), whereas Hard Only (78%) and All Difficulties (83%) match or exceed Claude Opus 5.5 (76%). Across modalities (c) , All Difficulties transfers evenly across image, signal, tabular, and text repositories (66–82%).
Frontier
Budget Models
Task
Metric
Claude Sonnet 5
Qwen3.5-9B
Ours (9B SFT+RL)
Δ Ours − Qwen3.5-9B
Code only (no tools, up to 10 debug attempts) 66 episodes per model
Quality
Leaderboard Percentile ↑
61.2
16.4
17.9
+1.5
Quality
Valid Submission (%) ↑
100
61
50
− 10.6
Efficiency
Input + Output Tokens / Ep. (k) ↓
13
78
65
− 12.7 †
Efficiency
Tool-Call Success (%) ↑
–
–
–
–
Appendix
Table 23: OOD tasks. Top rows of each task measure quality, bottom rows efficiency. Tokens: input plus output per episode. Shading and bold (best) compare the base model and ours. Δ : paired per-task difference from the base; † : its 95% task-bootstrap interval excludes zero.
Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints. Existing code benchmarks largely rely on synthetic tasks, binary correctness signals, or single-objective evaluation, limiting their ability to assess holistic optimization behavior. We introduce FormulaCode, a benchmark for evaluating agentic optimization on large, real-world codebases with fine-grained, multi-objective performance metrics. FormulaCode comprises 957 performance bottlenecks mined from scientific Python repositories on GitHub, each paired with expert-authored patches and, on average, 264.6 community-maintained performance workloads per task, enabling the holistic ability of LLM agents to optimize codebases under realistic correctness and performance constraints. Our evaluations reveal that repository-scale, multi-objective optimization remains a major challenge for frontier LLM agents. Project website at: https://formula-code.github.io
Atharva Sehgal, James Hou, Akanksha Sarkar +4
The University of Texas at Austin · California Institute of Technology · Cornell University
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.
Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.