Can LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution. The deleted code becomes the human reference that sets the bar. Earlier open-ended benchmarks give each agent a problem statement and grade one program after the run. FrontierCS 2 places the agent inside a research environment. The agent inherits a working repository and may submit a patch at any point. For each patch, a judge returns a development-suite score but keeps the evaluator, the held-out final suite and the reference private, and grades the final submission on that suite against the reference. The tasks span 13 domains, from machine-learning systems and high-performance computing to databases, security and robotics. Beyond these 48 tasks, FrontierCS 2 keeps a private set of more than 100 tasks, to be announced in the future. We evaluate 12 agents. The best one is Claude Opus 5.5, passing 58% of its runs. No run passes on 16 tasks. A higher score is not the same as a pass: most runs that beat the reference on the combined final score still fall short on some metric or workload. On a shared capability scale for agents and tasks, the median task is roughly as difficult as what the strongest agent can handle.
Figures & tables
Figure 1: Pass rate. Share of each agent’s runs whose final submission beats the human reference on every metric and held-out workload; the harness is in gray.
\toprule
FrontierCS
FrontierCS 2
\midrule Task
A problem statement
A repository with a paper’s contribution deleted
Output
One program
A patch
Bar
No known optimum
The authors’ code, on held-out workloads
Feedback
None
A development score for every patch
Domains
Algorithms and six research areas
13 computer-science domains
Metric
Mean score
Pass rate, Mean score, and FCI
Table 1: FrontierCS and FrontierCS 2. Each task moves into a paper’s own repository, with the authors’ code as the bar.
Figure 2: FrontierCS 2 task building. We delete the paper’s contribution from its own repository and give the agent what remains. The judge keeps the deleted code as the human reference, the pass thresholds measured from it on the paper’s workloads, and a final suite with held-out workloads.
Figure 3: FrontierCS 2 judge. Each colored box is a sandbox. The agent’s sandbox reaches only the driver. The agent sees only a development-suite score and a sanitized message per patch; all that decides its result stays with the judge.
Figure 4: Pass rate and cost. Each point is one agent: pass rate on the y-axis against mean cost per run in US$ on a log-scale x-axis. GPT-6.1 Sol passes less often than GPT-6 Astra and sits far to its left on the cost axis.
Figure 5: Final score against the human reference. Each faint point is one run’s final score minus the human reference’s on the same task, one row per agent; the line spans each agent’s range and the dot marks its mean. The vertical line at 0 marks a tie.
Figure 6: FrontierCS Capabilities Index. Each bar shows an agent’s FCI from 0 with its 90% bootstrap interval as a whisker. The intervals separate only agents that sit far apart on the capability scale.
Figure 7: Test-time scaling. FCI of each agent against the cost budget per run, in US$ on a log scale; a run still going at a budget counts as one without a final submission. Each curve starts at the first budget at which its FCI is at least 0 and ends at the agent’s FCI. Two slanted strokes mark an empty budget range skipped on the axis. The leader changes with the budget.
Figure 8: QuarterPIR bytes per lookup. Bytes per private lookup on the 67-million-entry database; the x-axis starts at 1,200. Muse Wonder 1.0’s checksums remove far more of the reference’s bytes than any other passing run does. Kimi K3, far below the rest and not passing, is left out. Step 5 Preview’s run timed out.
Figure 9: BBQ clock period and circuit size. Each point is one agent’s circuit on the largest tree, 4,096 priorities. The x-axis gives its size in lookup tables and the y-axis its clock period, with shorter, faster clocks at the top; the grey dot marks the human reference. Claude Opus 5.5’s circuit has the shortest clock period and is among the largest. Agents without a correct circuit at this size are left out.
Figure 10: SVG-EAR time and PSNR. Each run’s mean time per clip and mean PSNR against full attention on the held-out clips, with the human reference’s in grey; the x-axes start at 600 s and 16 dB. Only GPT-6 Astra is both faster and closer to full attention than the human reference on average, and it still fails. Step 5 Preview, far slower, is off the chart.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
\toprule Domain
Task
Paper
\midrule
Database
ScaleGPM
Accurate and Fast Approximate Graph Pattern Mining at Scale ( 34 )
Yannakakis
Instance-Optimal Acyclic Join Processing Without Regret: Engineering the Yannakakis Algorithm in Column Stores ( 35 )
FARGO Query Stopping
FARGO: Fast Maximum Inner Product Search via Global Multi-Probing ( 36 )
Spectrum
Spectrum: Speedy and Strictly-Deterministic Smart Contract Transactions for Blockchain Ledgers ( 37 )
SCOPE
Fast Local Subgraph Counting ( 38 )
Appendix
Table 2: Task sources. The paper each task is built from.
\toprule
Agent
GPU-hours
Train
Final eval
\cmidrule (lr)3-4 \cmidrule (lr)6-7Domain
Task
Cores
Memory
GPU
Agent
Verifier
evals
time limit
\midrule
Database
ScaleGPM
8
16
–
–
–
100
15
Yannakakis
2
8
–
–
–
100
15
FARGO Query Stopping
4
4
–
–
–
100
15
Spectrum
4
8
–
–
–
100
60
Appendix
Table 13
\toprule Model
Harness
Version
Reasoning effort
Output limit
\midrule Claude Opus 5.5
Claude Code
2.1.296
max
GPT-6 Astra
Codex CLI
0.162.1
xhigh
not set
GPT-6.1 Sol
Codex CLI
0.162.1
xhigh
not set
Gemini 4 Argon
Pi
1.2.0
high
rest of context
GLM 5.3
ZCode
0.16.9
max
262,144
Muse Wonder 1.0
Muse Code
1.4.2
max
128,000
Appendix
Table 3: Agent settings. Harness and version of each agent, with the reasoning effort and output token limit the harness requested. “Not set” leaves the default to the provider. Rows follow FCI order ( \Cref sec:exp-eci); Kimi K2.7 serves only as the 0 point of FCI.
\toprule Symbol
Meaning
Logits
FCI points
\midrule Fixed
λ
Precision of the strengths’ prior
0.35
Prior standard deviation of a tie level ht
6
60
Gauss–Hermite nodes
64
Bootstrap draws for FCI, and for FCI under a budget
500, 100
Weighted draws for an FDI interval
2,000
Appendix
Table 4: Constants and fitted values of the FCI model. Fixed constants are set before the fit; fitted values come from Step 8 or, for the unit floor c , from the data in Step 4. Values in logits are also given in FCI points, at ten points per logit.