AlphaEval: Evaluating Agents in Production

Pengrui Lu; Bingyu Xu; Wenjun Zhang; Shengjia Hua; Xuanjian Gao; Ranxiang Ge; Lyumanshan Ye; Linxuan Wu

cs.CLApr 14, 2026

AlphaEval: Evaluating Agents in Production

Authors: Pengrui Lu, Bingyu Xu, Wenjun Zhang, Shengjia Hua, Xuanjian Gao, Ranxiang Ge, Lyumanshan Ye, Linxuan Wu, +19 more

Organizations: SII · SJTU · GAIR · MiraclePlus · UCAS · HIT · HunterAI · KuaFuAI · POET · LangCore · Jiqizhixin · CinoCore

Abstract

The rapid deployment of AI agents in commercial settings has outpaced the development of evaluation methodologies that reflect production realities. Existing benchmarks measure agent capabilities through retrospectively curated tasks with well-specified requirements and deterministic metrics -- conditions that diverge fundamentally from production environments where requirements contain implicit constraints, inputs are heterogeneous multi-modal documents with information fragmented across sources, tasks demand undeclared domain expertise, outputs are long-horizon professional deliverables, and success is judged by domain experts whose standards evolve over time. We present AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business, spanning six O*NET (Occupational Information Network) domains. Unlike model-centric benchmarks, AlphaEval evaluates complete agent products -- Claude Code, Codex, etc. -- as commercial systems, capturing performance variations invisible to model-level evaluation. Our evaluation framework covers multiple paradigms (LLM-as-a-Judge, reference-driven metrics, formal verification, rubric-based assessment, automated UI testing, etc.), with individual domains composing multiple paradigms. Beyond the benchmark itself, we contribute a requirement-to-benchmark construction framework -- a systematic methodology that transforms authentic production requirements into executable evaluation tasks in minimal time. This framework standardizes the entire pipeline from requirement to evaluation, providing a reproducible, modular process that any organization can adopt to construct production-grounded benchmarks for their own domains.

Figures & tables

Figure 1: Overview of AlphaEval . Band 1 : The gap between research benchmarks and production reality. Band 2 : Our requirement-to-benchmark framework transforms production requirements into 94 formalized tasks across 6 O*NET domains. Band 3 : Key findings—best agent scores 64.41/100, scaffold matters as much as model, and six production-specific failure modes.

Figure 2: Taxonomy of evaluation methodologies for AI agents. AlphaEval covers multiple evaluation paradigms (excluding Pairwise Comparison and Others).

Benchmark	Domain	Eval Type	Production	Multi-Modal	Underspec.	Diverse Eval	Expert Val.	Dynamic	Cross-Domain
Software Engineering & Coding
SWE-bench ( Jimenez et al., 2024 )	Code	Rule	✗	✗	✗	✗	✗	✗	✗
SWE-bench Multimodal ( Yang et al., 2024 )	Code	Rule	✗	✓	✗	✗	✗	✗	✗
Multi-SWE-bench ( Zan et al., 2025 )	Code	Rule	✗	✗	✗	✗	✗	✓	✗
SWE-Lancer ( Miserendino et al., 2025 )	Code	Rule	✓	✗	✗	✗	✗	✗	✗
SWT-Bench ( Mündler et al., 2024 )	Code	Rule	✗	✗	✗	✗	✗	✗	✗

Table 1: Comprehensive comparison of AlphaEval with existing benchmarks across seven dimensions that characterize production-level evaluation: Production = tasks sourced from real commercial deployments with paying customers; Multi-Modal = requires processing multi-modal heterogeneous inputs (PDFs, spreadsheets, images); Underspec. = tasks deliberately preserve production-level requirement ambiguity; Diverse Eval = employs three or more distinct evaluation paradigms; Expert Val. = evaluation criteria co-developed and validated with domain experts; Dynamic = benchmark designed for continuous evolution rather than static one-time release; Cross-Domain = spans three or more distinct professional domains.

Figure 3: The requirement-to-benchmark construction framework: four stages from production requirements to automated evaluation.

Domain (O*NET)	Representative Task	Tasks	Input	Eval Paradigm
Human Resources (13-1071)	Resume screening vs. JD	11	PDF, JPEG	F1 score
Finance & Investment (13-2051)	Segment research & pitch critique	22	PDF, MD, TXT	LLM-as-a-Judge
Procurement & Operations (13-1020)	BOM cost optimization	23	Excel, CSV, MD	Constraint verif.
Software Engineering (15-1252)	Full-stack app generation	11	YAML, Code	UI testing
Healthcare & Life Sci. (29-9099)	eCRF & insurance policy analysis	16	PDF, MD	LLM + Numerical
Technology Research (15-1221)	AI industry deep analysis	11	MD	LLM-as-a-Judge

Table 2: Overview of AlphaEval task domains classified by O*NET occupational taxonomy ( U.S. Department of Labor, 2025 ) , where Eval Paradigm indicates the primary evaluation method.

Agent Product	Model	HR	F&I	P&O	SE	H&LS	TR	Avg.	Value (USD K)
		(11)	(22)	(23)	(11)	(16)	(11)
Claude Code	Claude Opus 4.6	35.91	70.35	83.35	70.95	50.06	75.82	64.41	110–165
	Gemini 3 Pro	26.73	54.68	65.39	59.47	39.88	58.55	50.78	89–133
	GPT-5.2	26.55	38.64	40.22	55.08	27.94	48.36	39.47	70–106
	Kimi K2.5	20.18	47.80	35.04	56.77	44.44	59.18	43.90	78–116
	GLM-5	31.91	51.18	66.43	52.46	38.88	51.36	48.70	82–122

Table 3: Average scores (0–100) on AlphaEval across six O*NET domains. Value (USD K) estimates the human replacement cost each configuration delivers, computed as

\sum_{d}\text{score}_{d}\times\text{domain\_value}_{d}

using per-domain expert-calibrated costs (Table 10 ). Bold : best in column. Underline : second best.

Domain	Mean	Std	95% CI
Human Resources	35.91	2.14	[33.77, 38.05]
Finance & Investment	70.35	1.87	[68.48, 72.22]
Procurement & Ops.	83.35	4.72	[78.63, 88.07]
Software Engineering	70.95	3.21	[67.74, 74.16]
Healthcare & Life Sci.	50.06	2.93	[47.13, 52.99]
Technology Research	75.82	3.56	[72.26, 79.38]

Table 4: Repeated evaluation results for Claude Code + Opus 4.6 (3 runs). CI = 95% confidence interval.

Pair	Agr.	$\kappa$	$\rho$	$r$
A vs. B	84.7%	0.691	0.818	0.870
A vs. LLM-as-a-Judge	85.0%	0.697	0.820	0.861
B vs. LLM-as-a-Judge	89.7%	0.780	0.845	0.885
Three-way (Fleiss)	79.7%	0.720	—	—

Table 5: Meta-evaluation results: pairwise agreement between two expert annotators (A, B) and the automated LLM-as-a-Judge evaluation across 1,000 rubric point judgments.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Agent Product	Version	Interface	Models
Claude Code	2.1.70	claude CLI	All 6 models
Codex	0.111.0	codex CLI	GPT-5.2, Opus 4.6
Codex	0.80.0	codex CLI	GLM-5, Kimi K2.5
GitHub Copilot	1.0.10	copilot-cli	GPT-5.2, Gemini 3 Pro, Opus 4.6
Cursor	2026.03.11	cursor CLI	Opus 4.6

Appendix

Table 6: Agent system configurations with pinned versions. Codex version varies by model due to API compatibility requirements.

Paradigm	Leaf-node Type	Covered	Tasks	Domains
Reference Verif.	Fuzzy matching	✓	42	HR, F&I, P&O
	Exact matching	✓	33	HR, F&I, P&O
	Word-embed. sim.	—	0	—
	LLM semantic	✓	49	F&I, H&LS, TR
Test Case Verif.	Code unit testing	✓	20	P&O
	Repo integ. test	—	0	—

Appendix

Table 7: Fine-grained evaluation type coverage across the paradigms implemented by AlphaEval .

Statistic	Value
Total tasks	94
Domains covered	6
Partner companies	7
Input modality distribution
PDF-primary	$\sim$ 42% (39 tasks)
Excel/CSV-primary	$\sim$ 21% (20 tasks)

Appendix

Table 8: Benchmark statistics of AlphaEval . Execution time and turns measured on Claude Code + Opus 4.6.

Domain	Tasks	Avg.	Types Composed
Human Resources	11	2.0	Fuzzy + Exact match
Finance & Investment	22	2.5	Rubric + LLM semantic + Structural verif. + Match
Procurement & Operations	23	3.8	Fuzzy + Exact + Unit test + Math + LLM
Software Engineering	11	2.5	Env. state verif. + Functional verif. + LLM
Healthcare & Life Sci.	16	2.8	Rubric + LLM + Math + Code + Match
Technology Research	11	2.5	Rubric + LLM semantic + Factual verif.

Appendix

Table 9: Multi-label evaluation composition by domain. Each domain composes multiple evaluation types within individual task rubrics.

Domain	Tasks	Hours	USD (K)	CNY (K)
Human Resources	11	39	1.8–2.7	4.1–6.4
Finance & Investment	22	803	49.6–74.4	165.3–248.7
Procurement & Operations	23	236	20.4–30.6	49.1–49.7
Software Engineering	11	946	61.2–91.8	116.9–179.9
Healthcare & Life Sci.	16	154	6.7–9.5	19.4–28.0
Technology Research	11	242	14.6–21.8	36.3–57.7

Appendix

Table 10: Economic value summary by domain after expert calibration. Hours and values reflect expert-adjusted estimates.

Dimension	Ground Truth	Model Output	Error
Funding round	F-round	Series D	Wrong round
Amount	$160M	$\sim$ $300M	1.9 $\times$
Valuation	$8B	$\sim$ $3B	2.7 $\times$ off
Lawyer users	74,000	Not mentioned	Missing

Appendix

Table 11: Factual hallucination example: Claude Opus 4.6 on Harvey AI funding data. The model substitutes outdated Series D data for the requested F-round.

Model	Search	Halluc.	Signature Pattern
Opus 4.6	30–50	High	Fills gaps with stale data confidently
GPT-5.2	5–75	Low	Variable persistence, sometimes outputs templates
Gemini 3 Pro	10–20	Low	Conservative, acknowledges uncertainty
GLM-5	5–10	Med	Conceptual inversion errors
MiniMax M2.5	3–5	Med	Correct direction, wrong details
Kimi K2.5	5–15	Med	Moderate persistence, domain-specific strengths

Appendix

Table 12: Model cognitive profiles on technology research tasks. Search persistence is measured by typical number of search attempts before writing.

Explore similar work

Efficient Benchmarking of AI Agents
Mar 24, 2026Franck NdzomgaAI Agent Evaluation AI Agent Benchmarks
EvalResearchBench: Can AI Agents Design Their Own Evaluations?
Oct 3, 2026Yaolun Zhang, Tianyi Xu, Yujie Zhao +3Automated Evaluation Benchmark Construction
Oregon State University · AG2 AI · University of California, San Diego +1
Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
May 2, 2026Mukund PandeyAI Agent Reliability AI Agent Evaluation
Independent Researcher

Mar 24, 2026J/K move · Enter open · S save

Efficient Benchmarking of AI Agents

Franck Ndzomga

AI Agent Evaluation AI Agent Benchmarks Benchmark Construction AI Agent Reliability

Open paper AskPDF arXiv

Evaluating AI agents on comprehensive benchmarks is expensive because each evaluation requires interactive rollouts with tool use and multi-step reasoning. We study whether small task subsets can preserve agent rankings at substantially lower cost. Unlike static language model benchmarks, agent evaluation is subject to scaffold-driven distribution shift, since performance depends on the framework wrapping the underlying model. Across eight benchmarks, 33 agent scaffolds, and 70+ model configurations, we find that absolute score prediction degrades under this shift, while rank-order prediction remains stable. Exploiting this asymmetry, we propose a simple optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates (30-70%). This mid-range difficulty filter, motivated by Item Response Theory, reduces the number of evaluation tasks by 44-70% while maintaining high rank fidelity under scaffold and temporal shifts. It provides more reliable rankings than random sampling, which exhibits high variance across seeds, and outperforms greedy task selection under distribution shift. These results suggest that reliable leaderboard ranking does not require full-benchmark evaluation.

Abstract

Explore similar work

Efficient Benchmarking of AI Agents

EvalResearchBench: Can AI Agents Design Their Own Evaluations?

Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework