Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduce StartupBench, an E2E agent benchmark grounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unified agent harness, even the strongest model successfully completes only approximately 30% of StartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects like complex instruction following and domain-specific expertise as major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks.
Figures & tables
Figure 1: Overview of StartupBench: real AI-product workflows, long-horizon agent execution, six-domain coverage, and multi-format deliverable generation.
Benchmark
Task Source
Final Output
E2E
Item-wise Eval.
Process Logic
Open- ended
Evaluation Method
GAIA ( Mialon et al., 2024 )
Manual
Text
△
✗
✗
✗
Rule
OSWorld ( Xie et al., 2024 )
Real+Manual
Env State
△
✗
✗
✗
Execution
DAComp ( Lei et al., 2025 )
Real+Manual
Workspace
✓
✗
✗
✓
Hybrid
GDPval ( Patwardhan et al., 2025 )
Manual
Work products
✓
✗
✗
△
Manual
Workspace-Bench ( Tang et al., 2026 )
Sim.+Manual
Workspace
✓
✗
✗
✓
Agent-as-Judge
OneMillionBench ( Yang et al., 2026 )
Expert-authored
Text
△
✗
△
✓
LLM-judge
Table 1: Comparison of StartupBench with existing benchmarks. E2E denotes end-to-end workflow completion; Item-wise Eval. denotes criterion-level judging. ✓, ✗, and △ indicate full, no, and partial support.
Figure 2: StartupBench data construction pipeline. We select market-validated startup agents, collect real user workflows, construct task artifacts and instances, and calibrate quality and difficulty through multi-stage review.
Category
Count
Percentage (%)
Overall statistics
Total tasks
97
100.0
Top-level domains
6
–
Domain distribution
Medical & HealthCare
21
21.6
Finance
18
18.6
Table 2: Statistics of StartupBench. The benchmark contains 97 real-world workflow tasks across six top-level domains and requires diverse deliverable formats.
Figure 3: Leaderboard of StartupBench.
Model
Overall
Med
Finance
Legal
Business
STEM
Education
Average score
Kimi-K3
73.67
80.03
62.38
69.45
83.90
68.34
77.71
GPT-5.6-sol
73.61
79.32
61.66
73.53
77.61
74.77
73.94
GPT-5.5
72.79
78.59
62.09
71.75
79.59
72.14
68.32
Seed-2.1-Pro
67.19
73.15
57.26
61.35
77.83
65.61
62.91
Kimi-K2.6
59.95
67.70
51.17
54.62
71.91
51.38
58.60
Table 3: Model performance of StartupBench, averaged over three independent runs. Top block: importance-weighted mean per-task score ( 0 – 100 ); bottom block: success rate, the fraction of samples scoring ≥90 (%), pooled over the three runs. Per-column best in bold , second-best underlined .
Figure 4: Performance across rubric dimensions. Importance-weighted pass rates (%) of different models on the six rubric dimensions in StartupBench. Higher values indicate better performance under the corresponding evaluation dimension.
Model
Auxiliary
Important
Core
Kimi-K3
77.89
75.12
73.40
GPT-5.6-sol
77.59
74.47
73.07
GPT-5.5
75.12
75.09
72.07
Seed-2.1-Pro
70.83
69.85
65.77
Kimi-K2.6
66.64
63.56
57.82
GLM-5.1
64.58
61.40
60.68
Table 4: Task-normalized rubric satisfaction rates (%) across different importance levels. For each model–task–trial, we first compute the satisfaction rate within each importance level and then average across tasks and trials.
Figure 5: A representative case of models failing on core rubrics though passing auxiliary rubrics on StartupBench.
Model
Compliance (%)
Kimi-K3
97.6
Seed-2.1-Pro
95.2
GPT-5.6-sol
94.6
Qwen-3.6-Max
94.6
DeepSeek-V4-Pro
94.0
GPT-5.5
93.5
Table 5: Output-format compliance on the 56 StartupBench tasks with explicit output-file requirement rubrics. Models are ranked by compliance rate.
Figure 6: A representative case of complex instruction following failure. Although the generated workbook appears complete, artifact-level evaluation reveals that several critical acceptance conditions remain unsatisfied, illustrating a common pattern of partial compliance.
Model
Hermes
Claude Code
Nanobot
Max–Min
GPT-5.5
71.00
70.11
72.79
2.68
GLM-5.1
60.20
60.18
60.79
0.61
Qwen-3.6-Max
59.10
57.38
59.46
2.08
Average
63.43
62.56
64.35
1.79
Table 6: Performance under different general-purpose agent frameworks. For each model, we report the average score under three general-purpose frameworks and the performance range (max–min) across frameworks.
Agent System
Avg Score
Success Rate
General (All Runs avg)
64.26
19.74
General (Oracle)
71.75
28.06
Specialized Agent
83.50
39.18
Table 7: General-purpose versus specialized agent systems. Oracle General denotes the averages of the highest score achieved in all trials for every model.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Expert Statistics
Value
Number of domain experts
57
Median professional experience
5 years
Experts with ≥3 years of experience
68.4%
Experts with ≥6 years of experience
42.1%
Experts with ≥10 years of experience
24.6%
StartupBench domain coverage
6 / 6
Appendix
Table 8: Summary of domain expert backgrounds. Professional experience refers to reported years of experience in the corresponding or closely related domain.
Category
Percentage
Structure & Completeness
27.31%
Calculation Precision
38.89%
Information Integration
9.46%
Domain-Specific Compliance
15.49%
Engineering & Format
6.20%
Output & Presentation
2.65%
Appendix
Table 9: Distribution of rubric categories in StartupBench.
Figure 7: A representative example of a rubric of a task spanning 3 levels. Core rubrics assess the essential financial reasoning and calculations required to solve the investment problem, Important rubrics evaluate the correctness and completeness of the supporting analysis, while Auxiliary rubrics capture presentation quality and additional quantitative support.
Figure 8: A representative example of self-verification hallucination. The model generates a plausible workbook and concludes that the task has been completed based on its execution process. However, artifact-level spot checks expose acceptance-critical errors in the final deliverable, showing that the model mistakes a progress summary for actual verification instead of validating the generated artifact against the original task requirements.
Figure 9: A representative example of failure caused by insufficient domain-specific clinical actionability.