Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
Organizations: Tencent Hy Team · The Hong Kong University of Science and Technology (Guangzhou) · Peking University · The Hong Kong University of Science and Technology ·
Abstract
Application optimization requires coding agents to discover useful changes and coordinate them across a complete program while preserving the required computation. We introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task provides a working program and specifies the workload, correctness requirements, timing boundaries, and permitted edits, leaving agents to choose where and how to optimize. Public testing supports iterative development, and a separate verifier scores the final submission. On benchmark version 1.1, we evaluate 16 coding agents, each pairing a model with a harness. Their Score@1 ranges from 25.5% to 67.4%. Most of their programs are correct, and the size of the speedup separates them: the two leading agents are at least 2x faster on 22 or more of the 32 tasks, while the other fourteen reach 2x on 8 to 17 and leave 125 of their 412 correct outcomes below 1.25x. Controlled replays show why coordination across an application matters: identical source code runs between 0.29x and 2.12x depending only on build and runtime settings, and one code change is worth 1.01x or 6.4x depending on another. Accelerate Bench pairs executable workloads with agents' programs and trajectories to evaluate both their gains and the process behind them.
Figures & tables
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| Agent | ||||
|---|---|---|---|---|
| Claude Sonnet 5.5 | 100.0 | 71.9 | 34.4 | 12.5 |
| Claude Opus 5.5 | 84.4 | 68.8 | 34.4 | 15.6 |
| GPT-6.1 Sol | 78.1 | 50.0 | 15.6 | 9.4 |
| GPT-6 Astra | 75.0 | 53.1 | 12.5 | 9.4 |
| Hy4-Preview | 71.9 | 50.0 | 18.8 | 6.2 |
| Hy4 | 75.0 | 46.9 | 15.6 | 6.2 |
| Task | Selected | Control | Configuration | Reference |
|---|---|---|---|---|
| \tasknamebox Ag ABySS | R1 | 0.923 | 0.991 | 1.000 |
| \tasknamebox Ag DataLife | unmodified | 0.943 | 1.035 | 1.000 |
| \tasknamebox Ag Fluidity | unmodified | 0.978 | 1.020 | 1.000 |
| \tasknamebox Ag GPAW | unmodified | 0.995 | 0.992 | 1.000 |
| \tasknamebox Ag Graph sparsification | n/a | 1.017 | 1.017 | 1.000 |
| \tasknamebox Ag GROMACS | R2 | 1.006 | 2.225 | 2.225 |
| GPT-6 Astra (96) | Hy4-Preview (77) | |
|---|---|---|
| SD of Score@1 when re-judged (points) | 1.45 (0.70–2.04) | 1.57 |
| SD of Score@1 between three independent runs (points) | 2.48 | 4.42 |
| Median coefficient of variation of speedup | 3.0% | 3.2% |
| Submissions whose correctness changed between judgments | 3/96 | 7/77 |
| Agent | Round 1 | Round 2 | Round 3 | Mean | SD | Reported |
|---|---|---|---|---|---|---|
| Hy4-Preview | 34.61 | 32.74 | 40.18 | 35.84 | 3.87 | 36.46 |
| DeepSeek-V4.1-Flash | 27.87 | 29.27 | 32.07 | 29.74 | 2.14 | 29.94 |
| Grok 4.6 | 27.53 | 26.03 | 24.93 | 26.16 | 1.30 | 26.24 |
| GLM-5.3-Flash | 25.42 | 22.24 | 29.07 | 25.58 | 3.41 | 27.34 |
| DeepSeek-V4-Flash | 19.23 | 20.07 | 22.84 | 20.72 | 1.89 | 19.08 |
| DeepSeek-V4-Pro | 19.47 | 19.78 | 21.70 | 20.32 | 1.21 | 20.63 |
| Category | Runs | Scored |
|---|---|---|
| Scored panel ( ) | 768 | yes |
| API failure after exhausting retries | 215 | no; continuation or new run fills the slot |
| Infrastructure cancellation | 108 | no; as above |
| Infrastructure runtime error | 18 | no; as above |
| Ended with an indeterminate error | 4 | no; as above |
| Verifier result missing or verifier fault | 4 | no |
| Main text | Appendix |
|---|---|
| Benchmark construction (§ 2 ) | Tasks, sources, resources and contracts ( D ); verifier validation controls ( E ) |
| Evaluation metrics (§ 3 ) | Agents, eligibility, usage ( F ); timing calibration, score sensitivity ( G ) |
| Main results (§ 4 ) | Benchmark v1.1 changes, thresholds, configuration baseline, timing noise and what a single run resolves ( A ); v1.0 results and run selection ( B ); per-task rewards of the v1.0 reported runs ( H ) |
| Analysis sample and labels (§ 5 ) | Annotation procedures and their audits ( I ) |
| Changes, candidates (§ 5.1 –§ 5.2 ) | Technique and workflow analysis ( J ) |
| Interacting choices (§ 5.3 ) | Dependency review ( K ); controlled experiments ( L ); case records ( M ) |
| Snapshot | Used for |
|---|---|
| Benchmark v1.1: first run of every agent; three runs of GPT-6 Astra and Hy4-Preview; fixed configuration-only baseline; repeated judgments of fixed submissions | Table 1 , Figures 2 and 3 ; Appendix A |
| Benchmark v1.0 main evaluation: 12 agents, 384 runs | Table 5 , Table 21 ; Appendix B ; scoring sensitivity ( G.4 ) |
| Analysis: 480 runs of five agents, three per task, with trajectories and public-test records | Section 5 ; Appendices I – K , M and N |
| Earlier 842-run snapshot, per-case timings | Baseline perturbation ( G.1 ); workload aggregation check ( G.3 ) |
| Controlled replays of fixed programs | Appendices E , G and L ; conditions are stated per experiment |
| Task | Version | CPUs | GiB | Agent h | Verify min |
|---|---|---|---|---|---|
| \tasknamebox Ag GROMACS | 12.0.3 | 4 | 6 | 8 | 390 |
| \tasknamebox Ag LAMMPS | 4.1.3 | 4 | 8 | 8 | 30 |
| \tasknamebox Ag GPAW | 1.0.2 | 4 | 6 | 8 | 233.3 |
| \tasknamebox Ag Quantum ESPRESSO | 1.2.2 | 4 | 6 | 8 | 233.3 |
| \tasknamebox Ag SIESTA | 1.2.2 | 4 | 6 | 8 | 233.3 |
| \tasknamebox Ag Fluidity | 1.0.2 | 4 | 8 | 8 | 233.3 |
| Application area | Tasks | |
|---|---|---|
| Molecular simulation / electronic structure | 5 | \tasknamebox Ag GROMACS, \tasknamebox Ag LAMMPS, \tasknamebox Ag GPAW, \tasknamebox Ag Quantum ESPRESSO, \tasknamebox Ag SIESTA |
| Fluids / atmosphere / ocean | 7 | \tasknamebox Ag Fluidity, \tasknamebox Ag ICON, \tasknamebox Ag MPAS-A, \tasknamebox Ag NEMO, \tasknamebox Ag OpenFOAM, \tasknamebox Ag Palabos, \tasknamebox Ag SU2 |
| Transport / plasma / astronomy | 4 | \tasknamebox Ag OpenMC, \tasknamebox Ag ShengBTE, \tasknamebox Ag VPIC, \tasknamebox Ag PRESTO |
| Bioinformatics / phylogenetics | 4 | \tasknamebox Ag ABySS, \tasknamebox Ag MHM2, \tasknamebox Ag MrBayes, \tasknamebox Ag wtdbg2 |
| Numerics / tensors / graphs | 6 | \tasknamebox Ag HPCG-derived, \tasknamebox Ag HPL-derived, \tasknamebox Ag NUFFT imaging, \tasknamebox Ag Tucker reconstruction, \tasknamebox Ag Graph sparsification, \tasknamebox Ag miniVite |
| Quantum circuits | 2 | \tasknamebox Ag QuEST, \tasknamebox Ag QiboTN |
| Task | Permitted edits | Timed execution | Required outputs and checks |
|---|---|---|---|
| \tasknamebox Ag GROMACS | Full source tree and supported build settings | Complete mdrun ; excludes build and input preparation | Prescribed physical calculation, trajectories, energies, final state, deterministic replay |
| \tasknamebox Ag HPCG-derived | Source tree, solver algorithm, entry point, build settings | Input, matrix construction, setup, two solves, output | Two full solutions; absolute error , relative residual ; identical repeated vectors |
| \tasknamebox Ag Paged generation | Application modules and helpers | Model loading, prefill, decoding, checkpoint, fresh-process resume | Full-model states, KV caches, sampling probabilities and tokens; numerical tolerances and exact recovery outputs |
| Competition | Tasks | |
|---|---|---|
| Asia Supercomputer Community (ASC) Student Supercomputer Challenge | 14 | \tasknamebox Ag ABySS, \tasknamebox Ag QiboTN, \tasknamebox Ag Fluidity, \tasknamebox Ag HPCG, \tasknamebox Ag HPL, \tasknamebox Ag NEMO, \tasknamebox Ag OpenFOAM, \tasknamebox Ag Palabos, \tasknamebox Ag PRESTO, \tasknamebox Ag QuEST, \tasknamebox Ag ShengBTE, \tasknamebox Ag Siesta, \tasknamebox Ag SU2, \tasknamebox Ag wtdbg2 |
| Student Cluster Competition at the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC) | 10 | \tasknamebox Ag DataLife, \tasknamebox Ag GROMACS, \tasknamebox Ag ICON, \tasknamebox Ag IO500, \tasknamebox Ag LAMMPS, \tasknamebox Ag MPAS-A, \tasknamebox Ag MrBayes, \tasknamebox Ag OpenMC, \tasknamebox Ag Quantum ESPRESSO, \tasknamebox Ag VPIC |
| Parallel Application Challenge | 4 | \tasknamebox Ag NUFFT, \tasknamebox Ag Paged LM generation, \tasknamebox Ag Tensor, \tasknamebox Ag ThunderSVM |
| Student Cluster Competition at the International Supercomputing Conference (ISC) | 2 | \tasknamebox Ag GPAW, \tasknamebox Ag MHM2 |
| IndySCC, the remote Student Cluster Competition at SC | 1 | \tasknamebox Ag miniVite |
| ACM-China International Parallel Computing Challenge | 1 | \tasknamebox Ag Graph sparsification |
| Task / source repository | Release / revision | Supplied code and retained computation |
|---|---|---|
| \tasknamebox Ag ABySS BirolLab/abyss | 1.9.0 07629837046e | Full application source; generated short reads drive assembly, graph/coverage outputs and contig statistics. |
| \tasknamebox Ag QiboTN qiboteam/qibotn | \tasknamebox Ag QiboTN 0.0.6 283bab987e89 | Full \tasknamebox Ag QiboTN tree with a disclosed CPU backend patch; task-authored circuits produce complete states and observables. |
| \tasknamebox Ag DataLife pnnl/datalife | Release name unknown ee9d3939a323 | FlowMonitor/FlowAnalysis/tools source; monitoring and DAG analysis of a fixed I/O driver. Separate HDF5/DaYu projects excluded. |
| \tasknamebox Ag Fluidity FluidityProject/fluidity | 4.1.16 4fc02e3da1cb | Full source; selected MPI redistribution and DG heat workflows with field outputs and checkpoint recovery. |
| \tasknamebox Ag GPAW GitLab: gpaw/gpaw | 21.1.0 sdist c5b9c0c91bf0 Historical tag; sdist mapping unverified | Full release sdist and upstream PAW setups; SCF, two BFGS steps and restart. These steps do not establish geometry convergence. |
| \tasknamebox Ag GROMACS gromacs/gromacs † | 2020.4 Commit unresolved | Full source with build/numerical preparation patches; prescribed water/argon MD trajectories and energies. |
| Controlled change | Component checks | Application checks | Speedup |
|---|---|---|---|
| Omit a generation step | ✓ | ✗ | — |
| Advance checkpoint cursor incorrectly | ✓ | ✗ | — |
| Change saved cache metadata | ✓ | ✗ | — |
| Emit nonfinite probabilities | ✓ | ✗ | — |
| Lossless checkpoint compression | ✓ | ✓ | 0.894 |
| Display label | Recorded model identifier |
|---|---|
| GPT-6 Astra | gpt-6-astra |
| Hy4-Preview | hy4-preview |
| DeepSeek-V4.1-Flash | deepseek-v4.1-flash-expires-on-0910 |
| Gemini-3.8-Flash | google/gemini-3.8-flash |
| GLM-5.3 | glm-5.3 |
| GLM-5.3-Flash | glm-5.3-flash |
| Total speedup | Submissions | Any slower case | Any slower | Equal-case HM below target |
|---|---|---|---|---|
| 342 | 102 | 28 | 5 | |
| 175 | 8 | 5 | 4 | |
| 107 | 0 | 0 | 5 |
| Score at a given speedup | Contribution | |||
|---|---|---|---|---|
| Scoring rule | of the two tasks | |||
| Runtime savings | 0.500 | 0.750 | 0.875 | 22.1% |
| Log speedup | 1.000 | 2.000 | 3.000 | 38.4% |
| Capped log, | 1.000 | 1.000 | 1.000 | 15.7% |
| Capped log, | 0.500 | 1.000 | 1.000 | 22.7% |
| Capped log, | 0.333 | 0.667 | 1.000 | 29.4% |
| Agent | Score@1 (%) | ( ) | (%) | (%) | |
|---|---|---|---|---|---|
| GPT-6 Astra | 40.3 | 1.675 | 0.790 | 71.3 | 67.3 |
| Hy4-Preview | 36.5 | 1.574 | 0.806 | 62.9 | 60.4 |
| DeepSeek-V4.1-Flash | 29.9 | 1.427 | 0.580 | 50.2 | 46.2 |
| Gemini-3.8-Flash | 29.9 | 1.427 | 0.583 | 53.5 | 47.4 |
| GLM-5.3 | 28.1 | 1.391 | 0.570 | 48.0 | 44.6 |
| GLM-5.3-Flash | 27.3 | 1.376 | 0.560 | 53.4 | 47.2 |
| Comparison | Round 1 | Round 2 | Round 3 |
|---|---|---|---|
| Rebuilt pristine source | 1.004 | 0.996 | 1.001 |
| Independent direct solver | 21.180 | 22.216 | 21.261 |
| Agent final submission | 26.325 | 25.862 | 26.209 |
| Independent time / agent time | 1.245 | 1.189 | 1.200 |
| Task | GPT-6 Astra | Hy4-Preview | DeepSeek-V4.1-Flash | Gemini-3.8-Flash | GLM-5.3 | GLM-5.3-Flash | Grok 4.6 | DeepSeek-V4-Pro | GLM-5.2 | DeepSeek-V4-Flash | Kimi K3 | Hy3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GROMACS | 65 | 63 | 0 | 58 | 52 | 53 | 53 | 52 | 53 | 53 | 53 | 0 |
| LAMMPS | 54 | 47 | 56 | 1 | 3 | 7 | 11 | 0 | 1 | 0 | 7 | 3 |
| GPAW | 31 | 5 | 4 | 23 | × | × | × | 12 | × | 12 | × | 0 |
| Quantum ESPRESSO | 9 | × | 0 | 4 | 12 | 4 | × | 1 | 1 | 1 | 0 | × |
| SIESTA | 61 | 55 | × | 39 | 2 | 49.8 | × | × | × | 51 | 49.7 | × |
| Fluidity | 0 | 0 | × | × | × | 0 | × | × | 0 | × | 0 | × |
| Labels | Annotator | Unit | Input | Used in |
|---|---|---|---|---|
| Technique | GPT-5.6 Luna, medium effort, Codex CLI | Final implementation of a run | Supplied and candidate source, build and launch files, trajectory excerpts | § 5.1 |
| Full trajectory | DeepSeek-V4.1-Flash, high effort, Responses API | Tool call; relationship record | Complete trajectory text | § 5.2 , § 5.3 |
| Measurement command | Lexical rules | Run | Shell-command arguments | § 5.2 |
| Technique | Runs | Tasks | Correct | |
|---|---|---|---|---|
| Computation / reuse | 225 | 29 | 201 (89.3%) | 108 (48.0%) |
| Parallel / vectorized | 132 | 22 | 116 (87.9%) | 83 (62.9%) |
| Runtime / numerical | 64 | 17 | 57 (89.1%) | 31 (48.4%) |
| Library / backend | 46 | 11 | 39 (84.8%) | 26 (56.5%) |
| Compiler / build | 40 | 14 | 32 (80.0%) | 10 (25.0%) |
| I/O / serialization | 28 | 8 | 24 (85.7%) | 11 (39.3%) |
| Model | Labeled | C | P | R | L | B | I |
|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash | 96 | 45 | 34 | 8 | 7 | 5 | 7 |
| Grok 4.6 | 95 | 67 | 36 | 6 | 7 | 3 | 13 |
| DeepSeek-V4-Pro | 93 | 33 | 26 | 17 | 8 | 13 | 3 |
| DeepSeek-V4-Flash | 96 | 46 | 22 | 16 | 11 | 8 | 2 |
| Hy3 | 96 | 34 | 14 | 17 | 13 | 11 | 3 |
| Technique | Groups | Tasks | Difference (pp) [95% CI] |
|---|---|---|---|
| Computation / reuse | 53 | 27 | +8.0 [1.4, 15.9] |
| Parallel / vectorized | 46 | 19 | +8.9 [1.9, 17.0] |
| Runtime / numerical | 35 | 17 | +0.6 [-6.3, 8.9] |
| Library / backend | 15 | 10 | -4.9 [-28.0, 14.4] |
| Compiler / build | 25 | 13 | -0.7 [-6.4, 6.4] |
| I/O / serialization | 9 | 7 | +1.7 [-8.1, 12.4] |
| Group | Runs | Correct | Mean reward | |
|---|---|---|---|---|
| P0 | 61 | 48 | 7 | 8.1% |
| P1 | 251 | 197 | 93 | 22.2% |
| P2 | 168 | 151 | 75 | 26.0% |
| Observation | Pooled count | App.-weighted % [95% CI] |
|---|---|---|
| First-to-last speedup | 19/31 | 65.2 [40.5, 86.6] |
| Last is slower than best | 11/31 | 31.3 [9.5, 58.1] |
| Last is the best tested version | 14/31 | 51.3 [29.7, 73.2] |
| Revision improves over predecessor | 36/70 | 62.9 [45.8, 81.2] |
| Revision sets a new observed best | 26/70 | 58.5 [40.0, 78.4] |
| Threshold | Last faster than first | Last slower than best |
|---|---|---|
| 5% | 19 | 11 |
| 10% | 17 | 8 |
| 20% | 13 | 6 |
| Feature | Runs | Variable cells | Reward pp [95% CI] |
|---|---|---|---|
| Local restoration/selection: present | 480 | 100 | [ ] |
| Restoration/retention share: pp | 468 | 145 | [ ] |
| Multi-layer reference: present | 480 | 69 | [ ] |
| Multi-layer group share: pp | 475 | 150 | [ ] |
| Relationship | Runs | Records | Agent | Source | Feedback |
|---|---|---|---|---|---|
| Build requirement | 410 | 3,956 | 1,284 | 2,216 | 456 |
| Execution-path dependency | 440 | 4,267 | 1,711 | 2,004 | 552 |
| Incompatibility | 440 | 3,923 | 2,326 | 397 | 1,200 |
| Replacement | 451 | 4,331 | 1,790 | 2,224 | 317 |
| Bypass | 432 | 3,819 | 1,832 | 1,371 | 616 |
| Performance hypothesis | 418 | 3,549 | 3,390 | 6 | 153 |
| SIMD | Threads | Reference (s) | Candidate (s) | Speedup |
|---|---|---|---|---|
| Disabled | 1 | 30.58 | 105.70 | 0.289 |
| Disabled | 4 | 30.48 | 30.72 | 0.992 |
| AVX2 | 1 | 30.69 | 43.31 | 0.709 |
| AVX2 | 4 | 30.80 | 14.51 | 2.123 |
| B | R | L | Candidate (s) | Gain over all-off | Share of saving (%) |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 76.99–77.12 | 1.000 | 0.00 |
| 1 | 0 | 0 | 50.86–51.00 | 1.512–1.515 | 37.76–37.93 |
| 0 | 1 | 0 | 76.94–77.21 | 0.999–1.002 | to |
| 0 | 0 | 1 | 75.96–76.02 | 1.014–1.015 | 1.50–1.60 |
| 1 | 1 | 0 | 50.89–50.93 | 1.512–1.514 | 37.79–37.88 |
| 1 | 0 | 1 | 7.94–8.04 | 9.592–9.711 | 99.87–100.08 |