On Replit, our agentic engineering platform, users now build entire products without writing code, through successive feature requests to an agent. Yet existing coding benchmarks evaluate single, fully specified tasks, and many are near saturation. We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models. Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does, under a grading protocol validated to keep this strict score reliable. Evaluating nine models, we find a reliability gap: the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds, and no open-weight model delivers one. The gap is closing fast: across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half. We release eight products with the harness and grader, hold out nine, and report lessons from production use.
Figures & tables
Figure 1: The ViBench 2.0 pipeline. The builder receives one PRD per stage and never sees the test plans; a seeding agent then loads each plan’s data, and a browser agent grades the finished app.
Figure 2: Feature dependencies of the product modeled on Uber, numbered in build order; an arrow from A to B means that B builds on A. Orange marks features two plans check together: earnings must show the car and cancellation fees, and a split fare must not alter them.
Model
pass@1
Plan pass
Min.
Cost
GPT-6.1 Sol
64.7 ± 9.3
96.9 ± 1.2
178
$12.3
GPT-6 Astra
60.3 ± 10.4
98.0 ± 0.6
198
$89.9
Opus 5.5
48.5 ± 10.9
93.7 ± 4.2
73
$23.1
Sonnet 5.5
19.1 ± 9.8
79.2 ± 7.3
26
$4.4
Grok 4.7
1.5 (0–7.9)
52.9 ± 8.9
118
$63.9
DeepSeek V4.1 Flash
0.0 (0–5.3)
77.4 ± 3.2
221
$2.5
Table 1: The three strongest models pass 94 to 98% of test plans but deliver a fully working product in only 49 to 65% of builds. Each model built each of the 17 products four times in the independent runs, and scores average over these 68 builds. Values after ± and in parentheses are 95% intervals. Time is in minutes and cost is in USD, both per build.
Model
pass@1
Plan pass
Min.
Cost
GPT-6.1 Sol ∗
53.1
94.9
170
$11.8
GPT-6.1 Terra
46.9
94.5
122
$6.2
GPT-6 Sol
31.2
84.0
88
$19.4
GPT-5.6 Sol
3.1
67.5
73
$26.5
GPT-6 Luna
0.0
59.8
188
$1.7
GPT-5.5
0.0
51.5
46
$30.1
Table 2: Newer GPT releases pass more test plans, but pass@1 rises only once the plan pass rate exceeds about 70%. Each model built the eight public products four times. Columns follow Table 1 . The starred row is the external team’s run; the others are ours.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Seeding and grading one test plan. Grey loops repeat a step, and each retry count is the most that step is retried. A plan that passes its first grade is not graded again, and a thrown-out grade (Section 2.3 ) does not count toward the majority.
Test
Result
Consistency
99.0% of 97 plans receive the same verdict in 10 of 10 grades.
Wrong fails
3 of 1,521 single grades fail a working plan (0.20%; 0.07–0.58%).
Wrong passes
4 of 477 passes are false (0.8%; 0.3–2.1%), settled by hand after re-grading with Sol.
Planted bugs
0 of 96 grades pass a plan while its bug shows (0–3.9%).
Same-family bias
4 of 237 wrong passes on Claude-built apps, 0 of 240 on GPT-built apps; within chance.
Failure audit
Every issue found in 178 failed plans of Sol, Astra and Opus was fixed before release; the other 159 are real bugs.
Appendix
Table 3: Grader validation on our reference runs. Ranges are 95% intervals. Wrong fails are single grades, before the majority of five. All studies used the grader as frozen before the audit fixes.
Bug kind
Passed
Example
Cent off
0/15
An 11.11ridepaysthedriver8.34, not $8.33 (Uber).
Reverts
0/15
A renamed dropdown option looks saved but reverts on reload (Asana).
Dropped
0/15
A one-letter typo on the server discards every edited task description (Asana).
No refusal
0/14
Cancelling a started ride fails without a message (Uber).
Timing
0/10
A scheduled ride enters dispatch 45 seconds early (Uber).
Obvious
0/27
Story points accept 4, 7 and 20, off the Fibonacci scale (Jira).
Appendix
Table 4: The grader caught every planted bug. Passed counts the grades that passed the plan aimed at a visible bug. Each kind has four or five bugs; obvious bugs have ten. Bugs were planted in our reference builds; examples come from the public products.
Figure 4: The file browser of four of our reference builds of the product modeled on Figma, each holding the same seeded files. Plans grade function, not visual design. Logos modeled on the real products are blurred.
Figure 5: The booking page of four of our reference builds of the product modeled on Uber, with the same trip entered. Sonnet shows the fare in dollars, although the plans expect pounds. Wordmarks the builds drew are blurred.
Model
Amazon
Asana
Discord
Figma
GitHub
Google Docs
Jira
Uber
Public 8
pass@1: builds in which every plan passes, of 4
GPT-6.1 Sol
2/4
4/4
2/4
0/4
0/4
1/4
4/4
4/4
53.1 ± 12.4
GPT-6 Astra
4/4
4/4
1/4
0/4
1/4
1/4
3/4
3/4
53.1 ± 14.4
Opus 5.5
0/4
2/4
2/4
0/4
2/4
0/4
4/4
2/4
37.5 ± 14.9
Sonnet 5.5
1/4
0/4
1/4
0/4
0/4
0/4
1/4
1/4
12.5 (3.5–29.0)
Grok 4.7
0/4
0/4
0/4
0/4
0/4
0/4
1/4
0/4
3.1 (0.1–16.2)
Appendix
Table 5: Results of the independent runs on each public product, 4 builds per product (32 builds per model). The last column is the public total in percent with its 95% interval; near 0% we give the exact binomial range. Held-out products are reported only in total (Table 6 ).
pass@1
Plan pass rate
Model
Public
Held out
Public
Held out
GPT-6.1 Sol
53.1
75.0
94.9
98.7
GPT-6 Astra
53.1
66.7
97.2
98.7
Opus 5.5
37.5
58.3
91.6
95.7
Sonnet 5.5
12.5
25.0
78.7
79.6
Grok 4.7
3.1
0.0
41.1
63.3
Appendix
Table 6: Scores of the independent runs on the 8 public products (32 builds per model) and the 9 held-out products (36 builds). Table 5 gives the public products one by one.
Model
pass@1
GPT-6.1 Sol
88.9
GPT-6 Astra
82.2
Opus 5.5
82.2
Sonnet 5.5
60.0
Opus 4.6 ∗
51.1
Appendix
Table 7: Our ViBench 1.0 rerun on its Vibe-on-Vibe tasks: pass@1 in percent, one build per feature (45 per model). ∗ The best model at ViBench’s release in its standard harness, as reported by Zhong et al. (2026a) .
SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.
Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem.
Xinbo Xu, Ruihan Yang, Haiyang Shen +13
UniPat AI · Peking University · Fudan University +4