cs.AINov 3, 2026

ViBench 2.0: A Benchmark for Long-Horizon Software Building

Authors: Daniel Furman, Preeya Kirani, Ibrahim Sheikh, Zhen Li, Peter Zhong, Michele Catasta

Organizations: Replit

Abstract

On Replit, our agentic engineering platform, users now build entire products without writing code, through successive feature requests to an agent. Yet existing coding benchmarks evaluate single, fully specified tasks, and many are near saturation. We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models. Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does, under a grading protocol validated to keep this strict score reliable. Evaluating nine models, we find a reliability gap: the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds, and no open-weight model delivers one. The gap is closing fast: across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half. We release eight products with the harness and grader, hold out nine, and report lessons from production use.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

    Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent BenchmarksSoftware Engineering Benchmarks

  2. ProgramBench: Can Language Models Rebuild Programs From Scratch?

    May 5, 2026John Yang, Kilian Lieret, Jeffrey Ma +9Software Engineering AgentsSoftware Engineering Benchmarks

  3. RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades

    May 15, 2026Xinbo Xu, Ruihan Yang, Haiyang Shen +13AI Coding AgentsSoftware Engineering Benchmarks