The rapid deployment of AI agents in commercial settings has outpaced the development of evaluation methodologies that reflect production realities. Existing benchmarks measure agent capabilities through retrospectively curated tasks with well-specified requirements and deterministic metrics -- conditions that diverge fundamentally from production environments where requirements contain implicit constraints, inputs are heterogeneous multi-modal documents with information fragmented across sources, tasks demand undeclared domain expertise, outputs are long-horizon professional deliverables, and success is judged by domain experts whose standards evolve over time. We present AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business, spanning six O*NET (Occupational Information Network) domains. Unlike model-centric benchmarks, AlphaEval evaluates complete agent products -- Claude Code, Codex, etc. -- as commercial systems, capturing performance variations invisible to model-level evaluation. Our evaluation framework covers multiple paradigms (LLM-as-a-Judge, reference-driven metrics, formal verification, rubric-based assessment, automated UI testing, etc.), with individual domains composing multiple paradigms. Beyond the benchmark itself, we contribute a requirement-to-benchmark construction framework -- a systematic methodology that transforms authentic production requirements into executable evaluation tasks in minimal time. This framework standardizes the entire pipeline from requirement to evaluation, providing a reproducible, modular process that any organization can adopt to construct production-grounded benchmarks for their own domains.
Figures & tables
Figure 1: Overview of AlphaEval . Band 1 : The gap between research benchmarks and production reality. Band 2 : Our requirement-to-benchmark framework transforms production requirements into 94 formalized tasks across 6 O*NET domains. Band 3 : Key findings—best agent scores 64.41/100, scaffold matters as much as model, and six production-specific failure modes.
Figure 2: Taxonomy of evaluation methodologies for AI agents. AlphaEval covers multiple evaluation paradigms (excluding Pairwise Comparison and Others).
Benchmark
Domain
Eval Type
Production
Multi-Modal
Underspec.
Diverse Eval
Expert Val.
Dynamic
Cross-Domain
Software Engineering & Coding
SWE-bench ( Jimenez et al., 2024 )
Code
Rule
✗
✗
✗
✗
✗
✗
✗
SWE-bench Multimodal ( Yang et al., 2024 )
Code
Rule
✗
✓
✗
✗
✗
✗
✗
Multi-SWE-bench ( Zan et al., 2025 )
Code
Rule
✗
✗
✗
✗
✗
✓
✗
SWE-Lancer ( Miserendino et al., 2025 )
Code
Rule
✓
✗
✗
✗
✗
✗
✗
SWT-Bench ( Mündler et al., 2024 )
Code
Rule
✗
✗
✗
✗
✗
✗
✗
Table 1: Comprehensive comparison of AlphaEval with existing benchmarks across seven dimensions that characterize production-level evaluation: Production = tasks sourced from real commercial deployments with paying customers; Multi-Modal = requires processing multi-modal heterogeneous inputs (PDFs, spreadsheets, images); Underspec. = tasks deliberately preserve production-level requirement ambiguity; Diverse Eval = employs three or more distinct evaluation paradigms; Expert Val. = evaluation criteria co-developed and validated with domain experts; Dynamic = benchmark designed for continuous evolution rather than static one-time release; Cross-Domain = spans three or more distinct professional domains.
Figure 3: The requirement-to-benchmark construction framework: four stages from production requirements to automated evaluation.
Domain (O*NET)
Representative Task
Tasks
Input
Eval Paradigm
Human Resources (13-1071)
Resume screening vs. JD
11
PDF, JPEG
F1 score
Finance & Investment (13-2051)
Segment research & pitch critique
22
PDF, MD, TXT
LLM-as-a-Judge
Procurement & Operations (13-1020)
BOM cost optimization
23
Excel, CSV, MD
Constraint verif.
Software Engineering (15-1252)
Full-stack app generation
11
YAML, Code
UI testing
Healthcare & Life Sci. (29-9099)
eCRF & insurance policy analysis
16
PDF, MD
LLM + Numerical
Technology Research (15-1221)
AI industry deep analysis
11
MD
LLM-as-a-Judge
Table 2: Overview of AlphaEval task domains classified by O*NET occupational taxonomy ( U.S. Department of Labor, 2025 ) , where Eval Paradigm indicates the primary evaluation method.
Agent Product
Model
HR
F&I
P&O
SE
H&LS
TR
Avg.
Value (USD K)
(11)
(22)
(23)
(11)
(16)
(11)
Claude Code
Claude Opus 4.6
35.91
70.35
83.35
70.95
50.06
75.82
64.41
110–165
Gemini 3 Pro
26.73
54.68
65.39
59.47
39.88
58.55
50.78
89–133
GPT-5.2
26.55
38.64
40.22
55.08
27.94
48.36
39.47
70–106
Kimi K2.5
20.18
47.80
35.04
56.77
44.44
59.18
43.90
78–116
GLM-5
31.91
51.18
66.43
52.46
38.88
51.36
48.70
82–122
Table 3: Average scores (0–100) on AlphaEval across six O*NET domains. Value (USD K) estimates the human replacement cost each configuration delivers, computed as ∑dscored×domain_valued using per-domain expert-calibrated costs (Table 10 ). Bold : best in column. Underline : second best.
Domain
Mean
Std
95% CI
Human Resources
35.91
2.14
[33.77, 38.05]
Finance & Investment
70.35
1.87
[68.48, 72.22]
Procurement & Ops.
83.35
4.72
[78.63, 88.07]
Software Engineering
70.95
3.21
[67.74, 74.16]
Healthcare & Life Sci.
50.06
2.93
[47.13, 52.99]
Technology Research
75.82
3.56
[72.26, 79.38]
Table 4: Repeated evaluation results for Claude Code + Opus 4.6 (3 runs). CI = 95% confidence interval.
Pair
Agr.
κ
ρ
r
A vs. B
84.7%
0.691
0.818
0.870
A vs. LLM-as-a-Judge
85.0%
0.697
0.820
0.861
B vs. LLM-as-a-Judge
89.7%
0.780
0.845
0.885
Three-way (Fleiss)
79.7%
0.720
—
—
Table 5: Meta-evaluation results: pairwise agreement between two expert annotators (A, B) and the automated LLM-as-a-Judge evaluation across 1,000 rubric point judgments.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Agent Product
Version
Interface
Models
Claude Code
2.1.70
claude CLI
All 6 models
Codex
0.111.0
codex CLI
GPT-5.2, Opus 4.6
Codex
0.80.0
codex CLI
GLM-5, Kimi K2.5
GitHub Copilot
1.0.10
copilot-cli
GPT-5.2, Gemini 3 Pro, Opus 4.6
Cursor
2026.03.11
cursor CLI
Opus 4.6
Appendix
Table 6: Agent system configurations with pinned versions. Codex version varies by model due to API compatibility requirements.
Paradigm
Leaf-node Type
Covered
Tasks
Domains
Reference Verif.
Fuzzy matching
✓
42
HR, F&I, P&O
Exact matching
✓
33
HR, F&I, P&O
Word-embed. sim.
—
0
—
LLM semantic
✓
49
F&I, H&LS, TR
Test Case Verif.
Code unit testing
✓
20
P&O
Repo integ. test
—
0
—
Appendix
Table 7: Fine-grained evaluation type coverage across the paradigms implemented by AlphaEval .
Statistic
Value
Total tasks
94
Domains covered
6
Partner companies
7
Input modality distribution
PDF-primary
∼ 42% (39 tasks)
Excel/CSV-primary
∼ 21% (20 tasks)
Appendix
Table 8: Benchmark statistics of AlphaEval . Execution time and turns measured on Claude Code + Opus 4.6.
Domain
Tasks
Avg.
Types Composed
Human Resources
11
2.0
Fuzzy + Exact match
Finance & Investment
22
2.5
Rubric + LLM semantic + Structural verif. + Match
Procurement & Operations
23
3.8
Fuzzy + Exact + Unit test + Math + LLM
Software Engineering
11
2.5
Env. state verif. + Functional verif. + LLM
Healthcare & Life Sci.
16
2.8
Rubric + LLM + Math + Code + Match
Technology Research
11
2.5
Rubric + LLM semantic + Factual verif.
Appendix
Table 9: Multi-label evaluation composition by domain. Each domain composes multiple evaluation types within individual task rubrics.
Domain
Tasks
Hours
USD (K)
CNY (K)
Human Resources
11
39
1.8–2.7
4.1–6.4
Finance & Investment
22
803
49.6–74.4
165.3–248.7
Procurement & Operations
23
236
20.4–30.6
49.1–49.7
Software Engineering
11
946
61.2–91.8
116.9–179.9
Healthcare & Life Sci.
16
154
6.7–9.5
19.4–28.0
Technology Research
11
242
14.6–21.8
36.3–57.7
Appendix
Table 10: Economic value summary by domain after expert calibration. Hours and values reflect expert-adjusted estimates.
Dimension
Ground Truth
Model Output
Error
Funding round
F-round
Series D
Wrong round
Amount
$160M
∼ $300M
1.9 ×
Valuation
$8B
∼ $3B
2.7 × off
Lawyer users
74,000
Not mentioned
Missing
Appendix
Table 11: Factual hallucination example: Claude Opus 4.6 on Harvey AI funding data. The model substitutes outdated Series D data for the requested F-round.
Model
Search
Halluc.
Signature Pattern
Opus 4.6
30–50
High
Fills gaps with stale data confidently
GPT-5.2
5–75
Low
Variable persistence, sometimes outputs templates
Gemini 3 Pro
10–20
Low
Conservative, acknowledges uncertainty
GLM-5
5–10
Med
Conceptual inversion errors
MiniMax M2.5
3–5
Med
Correct direction, wrong details
Kimi K2.5
5–15
Med
Moderate persistence, domain-specific strengths
Appendix
Table 12: Model cognitive profiles on technology research tasks. Search persistence is measured by typical number of search attempts before writing.
Evaluating AI agents on comprehensive benchmarks is expensive because each evaluation requires interactive rollouts with tool use and multi-step reasoning. We study whether small task subsets can preserve agent rankings at substantially lower cost. Unlike static language model benchmarks, agent evaluation is subject to scaffold-driven distribution shift, since performance depends on the framework wrapping the underlying model. Across eight benchmarks, 33 agent scaffolds, and 70+ model configurations, we find that absolute score prediction degrades under this shift, while rank-order prediction remains stable. Exploiting this asymmetry, we propose a simple optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates (30-70%). This mid-range difficulty filter, motivated by Item Response Theory, reduces the number of evaluation tasks by 44-70% while maintaining high rank fidelity under scaffold and temporal shifts. It provides more reliable rankings than random sampling, which exhibits high variance across seeds, and outperforms greedy task selection under distribution shift. These results suggest that reliable leaderboard ranking does not require full-benchmark evaluation.
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.
Yaolun Zhang, Tianyi Xu, Yujie Zhao +3
Oregon State University · AG2 AI · University of California, San Diego +1
Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of ground truth for long-horizon tasks. This paper makes three contributions. First, we present a taxonomy of seven failure modes unique to production agentic systems, each grounded in observations from systems operating at billion-event scale. Second, we demonstrate empirically where standard metrics -- ROUGE, BERTScore, accuracy/AUC, and the agentic benchmarks above -- fail to detect each failure mode. Third, we propose PAEF (Production Agentic Evaluation Framework), a five-dimension evaluation framework with an open-source reference implementation, designed for continuous evaluation on production traffic rather than episodic benchmark runs. Our analysis shows that standard metrics fail to detect four of the seven failure modes entirely and detect three others only after a lag of multiple evaluation cycles.