cs.LGSep 30, 2026
Savecua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents
Organizations: Carnegie Mellon University
Abstract
Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA benchmarks currently face a reproducibility crisis. Benchmarks are based on complex infrastructure with varying machine and container configurations that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we propose cua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs. cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks. Across four different CUA benchmarks, we evaluate how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We also demonstrate that we can effectively reduce the evaluation task set of most CUA benchmarks without degrading overall statistical power, allowing for more efficient benchmarking and comparison. We believe cua-speedrun will enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications. All code, infrastructure, and analysis are available at https://cuaspeedrun.com.
Figures & tables
Figure 1: cua-speedrun overview. cua-speedrun is a standardized platform for benchmarking the performance, speed, and cost of computer-use agents. It combines representative task selection with standardized infrastructure on Modal, enabling consistent comparisons across models, agent harnesses, and reasoning settings. Our findings reveal that more reasoning can reduce both task time and model cost for CUAs, while faster environment I/O can make agents slower.
Figure 2: Different agents define the speed and cost frontiers. Mean score vs. mean task time and model cost on OSWorld (top, 54 configs) and OSWorld2 (bottom, 19 configs), excluding MiniMax M3. Dashed lines connect the observed Pareto-frontier points, which are circled. Colors and marker shapes identify models, and darker shades indicate increased reasoning effort. The plots compare complete agent configs. Costs use recorded charges or token usage at the applicable API prices. Astra (low) uses five-seed means for each I/O setting. Full configs and plots in Appendix E .
Figure 3: Less reasoning can make an agent slower. Gemini 3 Flash Preview at low, medium, and high effort on OSWorld. The panels show mean task time, generated tokens including reasoning, and model calls across all 50 tasks. Scores are 33.6%, 57.6%, and 59.6%, respectively. Medium and high effort generate more tokens than low effort but require fewer calls and complete tasks faster.
Figure 4: The useful reasoning setting changes across benchmarks. Gemini 3.8 Flash at low, medium, and high effort; labels also report mean cost per task. Each curve keeps the model, agent harness, and execution settings fixed. Additional reasoning brings no gain on OSWorld, whereas moving from low to medium substantially improves OSWorld2 performance.
Figure 5: Faster I/O can be offset by more waiting and agent interaction. GPT-6 Astra (low), with normal and fast I/O. Panel (a) separates agent time, agent-requested waits, and remaining environment processing, averaged over five seeds per setting; error bars show the standard deviation of mean task time across seeds. Panel (b) shows two consecutive screenshots after opening Save As: the dialog is absent in the first and visible in the second, requested 5.8 s later with no intervening action.
Figure 6: Choosing an agent by performance, time, and cost. Joint Pareto frontier on OSWorld; the OSWorld2 frontier is in Appendix E , Fig. 13 . Circled points are the configurations on the Pareto frontier. Time and cost axes are logarithmic.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Infrastructure | Click | Escape | Ctrl+U | Type 100 characters |
|---|---|---|---|---|
| Gym-Anything | 2758.70 | 2865.00 | 2858.29 | 3521.72 |
| OSWorld | 2515.41 | 2707.74 | 2708.41 | 2756.47 |
| Fast I/O (ours) | 27.58 | 2.43 | 2.91 | 14.98 |
Table 1: Fast I/O reduces action-to-observation latency to milliseconds. Median latency in milliseconds over 20 calls per action. For each system, the timer starts immediately before env.step(action) and stops when it returns the observation. All default action waits are included.
| Component (ms) | Click | Escape | Ctrl+U | Type 100 characters |
|---|---|---|---|---|
| Gym-Anything | ||||
| Fixed waits | 2100.2 | 2100.2 | 2100.2 | 2100.2 |
| Action request | 153.5 | 153.7 | 154.0 | 805.5 |
| Screenshot capture | 258.9 | 354.0 | 354.3 | 354.1 |
| Screenshot transfer | 238.1 | 212.4 | 220.6 | 211.3 |
| Cleanup and other | 53.1 | 53.1 | 53.1 | 53.1 |
Table 2: Fixed waits account for most baseline latency. Mean component times from 10 profiled calls per action for Gym-Anything and 20 for OSWorld. Components sum to the total before rounding. The final row for each system is the profiled total minus its two-second post-action wait.
| Component (ms) | Click | Escape | Ctrl+U | Type 100 characters |
|---|---|---|---|---|
| Input execution | 27.30 | 1.85 | 2.55 | 18.39 |
| Frame capture | 1.03 | 0.77 | 0.77 | 0.80 |
| Foreground image preparation | 0.27 | 0.02 | 0.02 | 0.18 |
| Other environment processing | 0.93 | 0.94 | 1.00 | 0.90 |
| Transport | 4.67 | 4.99 | 4.76 | 4.62 |
| Screenshot-file delivery | 0.43 | 0.45 | 0.43 | 0.44 |
Table 3: Fast I/O completes the four tested actions in under 50 ms. Mean times over 20 commands per action, from the agent issuing a command to receiving the screenshot file. This includes command overhead, communication between sandboxes, and screenshot-file delivery in addition to the env.step call. Components sum to the total before rounding.
| Run | Score (%) | Time (s) | Agent (s) | Steps | Env. (s) | Waits (s) | Env. waits (s) |
|---|---|---|---|---|---|---|---|
| 1 | 91.62 | 103.96 | 98.07 | 8.68 | 5.891 | 5.553 | 0.338 |
| 2 | 91.62 | 93.98 | 89.02 | 8.04 | 4.955 | 4.628 | 0.327 |
| 3 | 91.62 | 96.48 | 91.67 | 7.96 | 4.812 | 4.488 | 0.324 |
| 4 | 87.62 | 100.22 | 95.28 | 8.20 | 4.934 | 4.604 | 0.330 |
| 5 | 89.62 | 100.46 | 95.68 | 8.12 | 4.785 | 4.424 | 0.361 |
| Mean | 90.42 | 99.02 | 93.94 | 8.20 | 5.075 | 4.739 | 0.336 |
Table 4: Fast I/O remains consistent across five evaluations. Each row averages all 50 OSWorld tasks for GPT-6 Astra at low reasoning effort. SD is the sample standard deviation across run-level means; score SD is in percentage points. Steps count action batches, and waits are explicitly requested by the agent.
| ID | Configuration | Batches | Actions | Responses | Tokens |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra / xhigh | 7.14 | 19.90 | – | 2,060 |
| 2 | Gemini 3.8 Flash / low | 16.60 | 20.02 | 18.52 | 1,564 |
| 3 | Claude Opus 5 / high | 19.92 | 26.92 | 20.62 | 3,735 |
| 4 | Gemini 3.8 Flash / medium | 27.16 | 34.70 | 29.10 | 5,365 |
| 5 | GPT-6 Astra / medium | 6.30 | 19.26 | – | 1,366 |
| 6 | GPT-6 Astra / high | 6.52 | 19.76 | – | 1,593 |
Table 5: Interaction and token counts on OSWorld. Means per task, including unsuccessful tasks. IDs match the OSWorld configuration tables in Appendix E . Tokens include reasoning and other output. Astra (low) uses five-seed means for each I/O setting.
| ID | Configuration | Batches | Actions | Responses | Tokens |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra / xhigh | 52.81 | 194.10 | – | 18,332 |
| 2 | GPT-6 Astra / high | 51.29 | 149.02 | – | 14,146 |
| 3 | GPT-6 Astra / low | 42.58 | 151.40 | – | 10,190 |
| 4 | GPT-6 Astra / medium | 45.46 | 151.08 | – | 11,382 |
| 5 | Gemini 3.8 Flash / high | 216.54 | 331.88 | 218.48 | 72,149 |
| 6 | Gemini 3.8 Flash / medium | 167.79 | 252.65 | 169.65 | 49,532 |
Table 6: Interaction and token counts on OSWorld2. Means per task, including unsuccessful tasks. IDs match the OSWorld2 configuration tables in Appendix E . Tokens include reasoning and other output.
Figure 7: Performance–interaction and performance–token frontiers on OSWorld. Mean score against mean action batches, individual actions, and generated tokens per task. Tokens include reasoning and other output. Panels (a)–(b) show all 56 configurations; panel (c) shows the 54 with token counts. Dashed lines connect the circled frontier points. IDs match the configuration tables in Appendix E ; darker green indicates greater reasoning effort. Astra (low) uses five-seed means for each I/O setting.
Figure 8: Performance–interaction and performance–token frontiers on OSWorld2. Mean score against mean action batches, individual actions, and generated tokens per task. Panels (a)–(b) show all 21 configurations; panel (c) shows the 19 with token counts. IDs match the OSWorld2 configuration table in Appendix E . Other plotting conventions follow Figure 7 .
| ID | Configuration | Score (%) | Time (s) | Cost ($) | Tokens |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra / xhigh | 91.6 | 126.8 | 0.708 | 2,060 |
| 2 | Gemini 3.8 Flash / low | 91.6 | 127.1 | 0.108 | 1,564 |
| 3 | Claude Opus 5 / high | 91.6 | 136.0 | 0.661 | 3,735 |
| 4 | Gemini 3.8 Flash / medium | 91.6 | 250.1 | 0.221 | 5,365 |
| 5 | GPT-6 Astra / medium | 89.6 | 90.2 | 0.600 | 1,366 |
| 6 | GPT-6 Astra / high | 89.6 | 100.8 | 0.636 | 1,593 |
Table 7: OSWorld configurations. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output.
| ID | Configuration | Score (%) | Time (s) | Cost ($) | Tokens |
|---|---|---|---|---|---|
| 27 | GPT-5.6 Luna / low / Codex | 75.6 | 144.8 | 0.025 | 3,191 |
| 28 | MiniMax M3 / thinking_off | 75.6 | 253.8 | – | – |
| 29 | GPT-5.6 Luna / high / API | 81.6 | 160.5 | 0.029 | 3,358 |
| 30 | Kimi K3 / low / batched | 73.6 | 149.5 | 0.174 | 1,865 |
| 31 | Muse Spark 1.3 / high | 73.6 | 506.9 | 0.148 | 10,538 |
| 32 | Kimi K3 / low / single | 71.6 | 176.6 | 0.280 | 3,171 |
Table 8: OSWorld configurations, continued. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output.
| ID | Configuration | Score (%) | Time (s) | Cost ($) | Tokens |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra / xhigh | 76.9 | 1237.3 | 8.175 | 18,332 |
| 2 | GPT-6 Astra / high | 75.0 | 914.0 | 7.391 | 14,146 |
| 3 | GPT-6 Astra / low | 68.2 | 904.2 | 6.336 | 10,190 |
| 4 | GPT-6 Astra / medium | 67.2 | 828.1 | 6.519 | 11,382 |
| 5 | Gemini 3.8 Flash / high | 60.2 | 3058.8 | 5.019 | 72,149 |
| 6 | Gemini 3.8 Flash / medium | 59.9 | 2051.8 | 2.915 | 49,532 |
Table 9: OSWorld2 configurations. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output.
Figure 9: Full OSWorld performance–time and performance–cost comparisons. All 56 configurations appear in the time comparison; the cost comparison uses configurations with reported or estimated costs. IDs identify the model, reasoning effort, and harness in Tables 7 – 8 . Dashed lines connect nondominated configurations for each pair of axes. Scores, times, and costs are means across the same 50 tasks, including unsuccessful tasks.
Figure 10: Full OSWorld comparison at a common token rate. Both plots use the same 54 configurations with token counts. The right plot replaces measured mean task time with mean recorded environment time plus generated tokens divided by 100 tokens/s. Tokens include reasoning and other output; scores and trajectories are unchanged. Point IDs refer to the OSWorld configuration tables.
Figure 11: Full OSWorld median-time view and fast-region detail. Mean verifier score versus median task time for all 56 configurations. For Astra (low), the median task times are averaged across five seeds. The right plot enlarges the region containing the fastest configurations; point IDs match the full OSWorld tables. Green dotted lines show the published human references of 72.36% and 111.94 s from the original 369-task OSWorld benchmark ( Xie et al., 2024 ) .
Figure 12: Full OSWorld2 performance–time and performance–cost comparisons. All 21 configurations are evaluated on the same 52 tasks. Point IDs refer to the OSWorld2 configuration table. Marker shapes follow Figure 2 ; darker green indicates greater reasoning effort.
Figure 13: Joint performance–time–cost frontier on OSWorld2. Circled points are the evaluated frontier configurations. Other configurations appear in gray. Time and cost axes are logarithmic. Marker shapes follow Figure 2 ; darker green indicates greater reasoning effort.
Figure 14: Performance–time and performance–cost frontiers on MyPCBench and CUA-World. GPT-6 Astra through Codex at four reasoning settings. Each point averages the same 38 MyPCBench tasks (top) or 26 CUA-World tasks (bottom). Darker green indicates greater reasoning effort. Dashed lines connect the observed Pareto-frontier points, which are circled.
| MyPCBench (38 tasks) | CUA-World (26 tasks) | |||||
|---|---|---|---|---|---|---|
| Effort | Score (%) | Time (s) | Cost ($) | Score (%) | Time (s) | Cost ($) |
| Low | 89.9 | 514 | 4.57 | 90.3 | 807 | 9.29 |
| Medium | 88.3 | 495 | 4.33 | 86.7 | 810 | 8.42 |
| High | 88.6 | 568 | 4.81 | 94.2 | 1,108 | 11.11 |
| Xhigh | 93.6 | 619 | 4.81 | 93.0 | 1,515 | 15.06 |
Table 10: Reasoning effort on MyPCBench and CUA-World. GPT-6 Astra through Codex, reporting mean partial-credit score, time, and model cost per task. Bold indicates the highest score, lowest time, or lowest cost within each benchmark.
| Run | Score (%) | Time (s) | Agent (s) | Steps | Env. (s) | Waits (s) | Env. waits (s) |
|---|---|---|---|---|---|---|---|
| 1 | 87.62 | 86.11 | 72.90 | 5.90 | 13.209 | 1.328 | 11.881 |
| 2 | 91.62 | 89.56 | 75.31 | 5.82 | 14.248 | 1.082 | 13.166 |
| 3 | 91.62 | 90.13 | 76.52 | 5.88 | 13.606 | 1.040 | 12.566 |
| 4 | 91.62 | 89.81 | 75.78 | 5.78 | 14.033 | 1.152 | 12.881 |
| 5 | 91.62 | 92.10 | 78.19 | 5.94 | 13.914 | 1.012 | 12.902 |
| Mean | 90.82 | 89.54 | 75.74 | 5.86 | 13.802 | 1.123 | 12.679 |
Table 11: Variation across five seeds for GPT-6 Astra at low reasoning effort. Each row averages all 50 OSWorld tasks. SD is the sample standard deviation of these means across the five runs; score SD is in percentage points. Steps count action batches, and waits are explicitly requested by the agent.
| I/O | Pass@ (%) | Best-of- partial score (%) | |
|---|---|---|---|
| Standard I/O | 1 | 87.20 | 90.82 |
| 2 | 88.00 | 91.62 | |
| 3 | 88.00 | 91.62 | |
| 4 | 88.00 | 91.62 | |
| 5 | 88.00 | 91.62 | |
| Fast I/O | 1 | 86.80 | 90.42 |
Table 12: Task-level repeatability across five seeds of Astra at low effort. Values average uniformly over all size- subsets of the five runs, then over the 50 tasks. Pass@ measures whether any of the attempts succeeds; best-of- uses the highest partial score.
| Benchmark | Full set | Selected subset |
|---|---|---|
| OSWorld ( Xie et al., 2024 ) | 295 | 50 |
| OSWorld2 ( Yuan et al., 2026 ) | 63 | 52 |
| CUA-World ( Aggarwal et al., 2026 ) | 143 | 26 |
| MyPCBench ( Jang et al., 2026a ) | 184 | 38 |
Table 13: Task counts for the representative subsets selected for cua-speedrun .
Figure 15: Score error and rank correlation on held-out models. OSWorld uses nine leave-one-agent-out folds; OSWorld2 uses seven. The randomized baseline averages 300 difficulty-stratified selections per task count. Dotted vertical lines mark 50 and 52 tasks; horizontal lines mark rank correlation 0.95. Each task count is optimized independently.
| Benchmark | Partial | Exact | Partial MAE | Exact MAE | |
|---|---|---|---|---|---|
| OSWorld | 49 | 0.967 | 0.967 | 4.36 | 4.57 |
| 50 | 0.983 | 0.975 | 2.16 | 2.17 | |
| 51 | 1.000 | 0.992 | 4.21 | 3.95 | |
| OSWorld2 | 51 | 0.964 | 0.955 | 1.70 | 2.28 |
| 52 | 0.964 | 0.955 | 1.33 | 2.03 | |
| 53 | 0.964 | 0.982 | 0.74 | 1.04 |
Table 14: Held-out-model validation around the selected task count. MAE is in percentage points. Each model is excluded from task selection in its fold. Both correlations exceed 0.95 at the selected task count and its two neighbors.
Figure 16: Speed–performance frontiers from full-set scores and held-out subset estimates. Dark points and connecting lines mark nondominated configurations. Model IDs are: 1, Sonnet 5; 2, Gemini 3.6 Flash; 3, Gemini 3 Flash Preview; 4, GLM-5V Turbo; 5, Luna; 6, Kimi K3; 7, Muse Spark; 8, MiniMax M3; 9, Qwen3.5-9B. Each point in (b) uses a 50-task subset selected with that model held out.
| Method | Partial MAE | Exact MAE | Mean MAE |
|---|---|---|---|
| Uniform random | 5.88 | 6.01 | 5.95 |
| Difficulty-stratified random | 5.05 | 5.11 | 5.08 |
| IRT-inspired selection | 7.94 | 8.21 | 8.08 |
| Energy-based selection | 4.50 | 4.34 | 4.42 |
Table 15: Task-selection methods on OSWorld with 32 tasks. Errors are averaged across nine leave-one-agent-out folds and reported in percentage points. Mean MAE averages partial-score and exact-completion MAE. Random baselines average 300 selections per fold.
| Application | Task UUID |
|---|---|
| chrome | 06fe7178-4491-4589-810f-2e2bc9502122 |
| chrome | 2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3 |
| chrome | 3720f614-37fd-4d04-8a6b-76f54f8c222d |
| chrome | 44ee5668-ecd5-4366-a6ce-c1c9b8d4e938 |
| chrome | af630914-714e-4a24-a7bb-f9af687d3b91 |
| gimp | 045bf3ff-9077-4b86-b483-a1040a949cff |
Table 16: Selected OSWorld task identities.
| Environment | Task name |
|---|---|
| ardour_env | broadcast_podcast_stem_delivery |
| dhis2_env | rmncah_scorecard_dashboard |
| docker_desktop_env | diagnose_broken_microservices_stack |
| gpredict_env | poes_downlink_schedule_setup |
| gvsig_desktop_env | vulnerability_map_remote_communities |
| jstock_env | quarterly_portfolio_rebalance |
Table 17: Selected CUA-World task identities.
| Reviewer or outcome | Included | Excluded |
|---|---|---|
| Reviewer 1 | 297 | 72 |
| Reviewer 2 | 302 | 67 |
| Reviewer 3 | 295 | 74 |
| Unanimous decision | 295 | 67 |
| Disagreement | 7 | |
Table 18: Independent inclusion decisions for 369 OSWorld tasks. The seven disagreements are excluded by the unanimous-inclusion rule.
| Requested input | Received input | Cause |
|---|---|---|
| Type < | > | Incorrect X11 key/modifier mapping |
| Type accented text | Accented characters omitted | Characters absent from the key map |
| Type a literal shell variable | Variable expanded into a path | Text interpreted by the command shell |
| Keypad Enter or Menu | No corresponding key event | Missing key-name translation |
| Toggle Caps Lock, then type | Lowercase text | Caps Lock press silently omitted |
Table 19: Input errors detected by CUA-AutoDebug when executing actions through SSH and PyAutoGUI. The tests compare requested input with the keyboard and mouse events received.
| Harness | Model response | Executed behavior |
|---|---|---|
| Gemini | Scroll down by five wheel clicks | Scroll up by 600 ticks |
| Gemini | Press F5 or Page Down | Type the key name as text |
| Qwen3.5 | Middle-click the target | No action |
| Qwen3.5 | Ctrl-click the target | Release Ctrl before clicking |
| Qwen3.5 | Move, then scroll or drag | Execute only the first tool call |
Table 20: Harness errors can change a correct model action. Examples detected by comparing model responses with the actions executed and the input events received.
| Action family | Operations |
|---|---|
| Mouse clicks | Left, right, middle, double, and triple click at a coordinate |
| Pointer motion | Move, drag along coordinates, hold or release a mouse button |
| Scrolling | Signed vertical scroll at the current pointer position |
| Keyboard | Type text, press a key or chord, hold and release modifiers |
| Waiting | Wait for an agent-specified duration |
| Observation | Request a screenshot without changing the desktop |
Table 21: Desktop actions supported by the infrastructure. A batch contains one or more actions executed in order. The model-facing tool schema can differ between harnesses.
| Model | Effort | Interaction rule | Score | Time | Batches | Actions | Tokens |
|---|---|---|---|---|---|---|---|
| Astra | xhigh | Batched | 91.62 | 126.82 | 7.14 | 19.90 | 2,060 |
| Astra | xhigh | One action + image | 91.62 | 182.73 | 16.50 | 16.50 | 2,559 |
| Kimi K3 | low | Single tool | 71.62 | 176.61 | 8.12 | 55.10 | 3,171 |
| Kimi K3 | low | Batched tools | 73.62 | 149.50 | 5.74 | 30.00 | 1,865 |
| Kimi K3 | high | Single tool | 81.62 | 305.07 | 8.92 | 113.60 | 7,567 |
| Kimi K3 | high | Batched tools | 79.62 | 358.13 | 8.76 | 60.20 | 7,125 |
Table 22: Batching comparisons within the same model and reasoning effort on OSWorld. Score is a percentage; time is in seconds. All counts and times are means over the same 50 tasks. Tokens include reasoning and other output.
Figure 17: Harness performance depends on reasoning effort. GPT-5.6 Luna at low, medium, and high reasoning effort, using the direct-API agent or Codex on OSWorld. Each curve connects reasoning settings within one harness. All six evaluations use the same 50-task set.
| Model | Tasks | Effort | Time | Batches | Actions | Tokens | Tok/s |
|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | 5 | low | 1291.0 | 116.40 | 171.60 | 29,596 | 47.57 |
| medium | 1363.9 | 112.60 | 158.40 | 34,446 | 43.10 | ||
| high | 3024.5 | 190.60 | 246.60 | 73,578 | 42.18 | ||
| GPT-6 Astra | 13 | low | 791.8 | 47.08 | 146.69 | 7,878 | 15.18 |
| medium | 752.6 | 47.15 | 138.85 | 8,376 | 17.07 | ||
| high | 817.3 | 53.15 | 171.23 | 11,001 | 21.31 |
Table 23: Reasoning effort on tasks completed fully at every compared effort on OSWorld2. Every reported score is 100%. Time is in seconds per task; token rate divides total generated tokens by total agent time. Each model uses its own common task set.
Figure 18: Validation gains do not generalize to the held-out model. Across 34 candidate methods, the coding agent receives only validation errors. We measure test error separately on the held-out model. Each method selects 32 tasks. Error averages the mean absolute errors in partial score and exact completion, in percentage points.