Code generation has emerged as one of AI's highest-impact use cases, yet existing benchmarks measure isolated tasks rather than the complete "zero-to-one" process of building a working application from scratch. We introduce Vibe Code Bench, a benchmark of 100 web application specifications (50 private validation, 50 held-out test) with 964 browser-based workflows comprising 10,131 substeps, evaluated against deployed applications by an autonomous browser agent. Across 16 frontier models, the best achieves 61.8% accuracy on the test split, revealing that reliable end-to-end application development remains a frontier challenge. We identify self-testing during generation as a strong performance predictor (Pearson r=0.72), and show through a completed human alignment study that evaluator selection materially affects outcomes (31.8-93.6% pairwise step-level agreement). Our contributions include (1) a novel benchmark dataset and browser-based evaluation pipeline for end-to-end web application development, (2) a comprehensive evaluation of 16 frontier models with cost, latency, and error analysis, and (3) an evaluator alignment protocol with both cross-model and human annotation results.
Figures & tables
Figure 1. Generation flow from natural-language specification to a runnable application artifact. Generation-flow schematic showing the benchmark's app construction pipeline from input specification through agent execution to final runnable artifact.
Model
Accuracy ± SE (%)
GPT-5.3-Codex
61.77±4.71
Claude Opus 4.6
57.57±4.37
GPT-5.2
53.50±5.07
Claude Opus 4.6 Thinking
53.50±4.68
Claude Sonnet 4.6
51.48±4.64
GPT-5.2-Codex
37.91±4.58
Table 1. Top-model accuracy on Vibe Code Bench (test split).
Benchmark
Task Type
Eval Method
Gaps
HumanEval
Single function
Unit tests
No multi-file apps or deployment
SWE-Bench
Bug fixes
Unit Tests
Assumes existing codebase
SWE-Lancer
Freelancing
E2E tests
Scoped tasks, not full apps
LiveCodeBench
Leetcode
Unit tests
No real-world context
Terminal-Bench
Terminal Tasks
Unit Tests
No browser UI or persistence
WebArena
Web navigation
Task Success
Navigation only, no code gen
Table 2. Comparison of code-generation benchmarks.
Split
#Tasks
#Workflows
#Substeps
Validation (public)
50
491
4,995
Test
50
473
5,136
Total
100
964
10,131
Table 3. Dataset splits and evaluation workload.
Figure 2. Automated evaluation flow from deployed app to workflow pass/fail scoring. Evaluation-flow schematic showing deployment verification, browser workflow execution, and scoring outputs.
Figure 3. Accuracy–cost and accuracy–latency trade-offs. Two scatter plots showing accuracy versus cost and accuracy versus latency.
Model
Accuracy
Cost
Latency
Open
GPT-5.3-Codex
61.77±4.71 %
$11.91
75.8m
×
Claude Opus 4.6
57.57±4.37 %
$8.69
21.3m
×
GPT-5.2
53.50±5.07 %
$17.75
82.9m
×
Claude Opus 4.6 Thinking
53.50±4.68 %
$8.28
23.1m
×
Claude Sonnet 4.6
51.48±4.64 %
$5.91
26.2m
×
GPT-5.2-Codex
37.91±4.58 %
$4.15
32.2m
×
Table 4. Model performance on Vibe Code Bench (test split). Open-weight models are marked ✓ ; closed models are marked × .
Model
Easy (23)
Medium (19)
Hard (8)
GPT-5.3-Codex
81.88
57.91
13.13
Opus 4.6
73.20
54.11
20.86
Gemini 3.1 Pro
46.24
28.33
0.00
GLM-5
40.51
12.43
0.00
Kimi-K2.5
27.07
12.32
2.50
Qwen3.5 Plus
31.71
3.03
0.00
Table 5. Accuracy by difficulty (%).
Model
None (37)
Email only (4)
Stripe only (2)
Both (7)
GPT-5.3-Codex
71.25
55.00
12.50
29.58
Opus 4.6
63.58
46.25
25.00
41.59
Gemini 3.1 Pro
38.42
32.50
12.50
3.57
GLM-5
27.60
23.12
0.00
7.74
Kimi-K2.5
20.44
10.00
12.50
7.94
Qwen3.5 Plus
19.75
11.25
0.00
1.59
Table 6. Accuracy by integration (%).
Model
Individual (12)
Solo Founder (22)
Enterprise Tool (16)
GPT-5.3-Codex
76.93
62.50
49.39
Opus 4.6
56.59
63.09
50.73
Gemini 3.1 Pro
49.30
37.07
12.16
GLM-5
44.24
21.58
10.14
Kimi-K2.5
34.00
11.68
13.24
Qwen3.5 Plus
34.17
13.99
4.32
Table 7. Accuracy by domain (%).
Figure 4. Performance by Reasoning Effort (20 app subset). A line chart showing workflow accuracy at low, medium, and maximum available reasoning effort for three representative models.
Figure 5. Vision ablation (20 app subset) A bar chart comparing vision-enabled and vision-disabled accuracy for three representative models.
Model
Acc. (%)
Bash
Edit
Browser
SQL
Migrate
Other
GPT-5.3-Codex
61.8
61.4
10.2
13.2
3.9
2.1
9.1
Claude Opus 4.6
57.6
23.3
32.2
22.9
3.7
11.1
6.7
GPT-5.2
53.5
48.5
32.7
13.6
1.4
1.3
2.6
GPT-5.2-Codex
37.9
47.7
37.1
8.4
1.3
2.5
2.9
GLM-5
23.4
46.0
28.1
12.1
4.4
6.4
3.0
Grok 4.1 Fast Reasoning
1.2
41.8
27.9
1.1
14.4
7.0
7.9
Table 8. Tool invocations by model (% of calls per app).
Figure 6. Aggregate trajectory action composition across all tasks. Vertical stacked bar chart for all evaluated models, showing average trajectory actions per application categorized as exploration, code writing, debugging, running, planning, installation, database setup, frontend initialization, browsing, documenting, and other.
Source model
Generation Acc. (%)
Evaluator Acc. (%)
Gemini 3.1 Pro
19.38±3.47
37.61±0.99
GPT-5.2
51.63±3.95
51.25±1.16
Claude Sonnet 4.6
47.65±5.00
45.71±3.00
Table 9. Generation and evaluator accuracy on the six-app variance panel (mean ± stderr).
Evaluator
Gemini 3.1 Pro
GPT-5.2
Claude Sonnet 4.6
Claude Sonnet 4.5
Reviewer A
Reviewer B
Reviewer C
Mean Reviewer
Gemini 3.1 Pro
–
38.8
86.4
86.5
86.4
84.5
83.7
84.7
GPT-5.2
38.8
–
33.1
36.0
48.0
33.5
31.8
36.1
Claude Sonnet 4.6
86.4
33.1
–
87.8
86.0
86.7
86.0
86.3
Claude Sonnet 4.5
86.5
36.0
87.8
–
87.5
86.0
86.2
86.4
Reviewer A
86.4
48.0
86.0
87.5
–
93.6
88.6
–
Reviewer B
84.5
33.5
86.7
86.0
93.6
–
91.4
–
Table 10. Pairwise step-level agreement (%) across model and human evaluators.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. Application pass-rate histograms for all evaluated models (test split). Grid of histograms, one per model, showing counts of application pass rates across 12.5-point bins.
Figure 8. Trajectory timeline by model on a single application ( bill_splitting_app ). Time-aligned action timelines across eight models, where each colored segment is one categorized agent action over elapsed runtime.
Model
Easy
Med.
Hard
GPT-5.3-Codex
81.88
57.91
13.13
Claude Opus 4.6
73.20
54.11
20.86
GPT-5.2
81.53
33.72
19.88
Opus 4.6 Thinking
71.67
46.49
17.91
Claude Sonnet 4.6
69.19
46.61
12.13
GPT-5.2-Codex
58.97
26.62
4.18
Appendix
Table 11. All-model accuracy by difficulty (%).
Model
None
Email
Stripe
Both
GPT-5.3-Codex
71.25
55.00
12.50
29.58
Claude Opus 4.6
63.58
46.25
25.00
41.59
GPT-5.2
60.63
50.62
31.25
23.81
Opus 4.6 Thinking
60.58
31.25
37.50
33.37
Claude Sonnet 4.6
59.41
38.12
25.00
24.73
GPT-5.2-Codex
44.15
24.38
12.50
19.95
Appendix
Table 12. All-model accuracy by integration (%).
Model
Individual
Solo Founder
Enterprise Tool
GPT-5.3-Codex
76.93
62.50
49.39
Claude Opus 4.6
56.59
63.09
50.73
GPT-5.2
72.53
56.50
35.11
Opus 4.6 Thinking
57.94
61.79
38.77
Claude Sonnet 4.6
73.46
51.30
35.24
GPT-5.2-Codex
45.27
43.06
25.31
Appendix
Table 13. All-model accuracy by domain (%).
Table 22
Model ID
Temp
Top-p
Reasoning
Vision Disabled
openai/gpt-5.3-codex-api-preview
n/a
n/a
xhigh
-
anthropic/claude-opus-4-6
1.0
1
-
-
openai/gpt-5.2
n/a
1
xhigh
-
anthropic/claude-sonnet-4-6
1.0
1
-
-
anthropic/claude-opus-4-6-thinking
1.0
provider default
-
-
openai/gpt-5.2-codex
n/a
1
high
-
Appendix
Table 15. Generation LLM hyper-parameters for all 16 models.
Web application generation uses AI to turn natural-language, visual, or multimedia specifications into functional, interactive websites and applications.
State Key Lab of CAD&CG, Zhejiang University · Ant Group · State Key Lab of CAD&CG, Zhejiang University; Laboratory of Art and Archaeology Image (Zhejiang University), Ministry of Education, China