Efficient Benchmarking in Production: A Study of an Evolving LLM Agent

Yining She; Lei Lin

cs.AISep 18, 2026

Efficient Benchmarking in Production: A Study of an Evolving LLM Agent

Authors: Yining She, Lei Lin

Organizations: Carnegie Mellon University · Meta

Abstract

Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation for a production analytics agent serving tens of thousands of monthly active users and report first-hand deployment experience. Using 574 historical runs of the production benchmark, split chronologically into calibration and held-out periods, we compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. The results show that multidimensional 2PL adaptive testing achieves the best overall score fidelity: executing 200 questions, 38.5% of a full run, yields 1.03 pp of MAE. We nevertheless deployed difficulty-stratified fixed subsets because of their operational simplicity, and show they transfer without recalibration to five other agent families and remain stable across calibration windows as short as one day. Drawing on this deployment experience, we report practical recommendations for recurring production-agent evaluation.

Figures & tables

Figure 1: Overview of the study. (a) Historical evaluation runs of a production analytics agent, split by time into calibration and held-out test runs. (b) The four partial-evaluation methods compared: each executes a subset

S\subseteq\mathcal{Q}

and reports an estimate

\hat{s}

of the pass rate

s

of a complete run. (c) Test on the 287 held-out runs: pass-rate MAE against the complete run (lower is better) and Spearman and Kendall rank correlation (higher is better).

Figure 2: Pass-rate MAE v.s. execution fraction. Fixed-budget methods use $k/519$ . Caching points report mean execution fractions across cache thresholds.

Panel of Figure 3 — Figure 3: Ranking fidelity v.s. execution fraction on 287 held-out runs. Higher is better.

Figure 4: Fixed-subset MAE on 287 held-out runs.

Figure 5: Adaptive Testing MAE on the held-out runs.

Figure 6: MAE on 287 original-agent and 299 transferred runs from five families. The subset and estimators use only original-agent calibration data.

Figure 7: MAE across different calibration windows on 287 held-out runs.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Random	Difficulty weighted	Difficulty gp-IRT	Rasch adaptive
10	1.9	10.48	8.10	7.57	10.94
20	3.9	7.22	6.37	5.51	9.79
30	5.8	5.86	4.63	4.35	6.50
50	9.6	4.41	3.37	3.28	6.46
75	14.5	3.49	3.04	3.05	3.51

Appendix

Table 1: Complete fixed-budget headline results on 287 held-out runs. Within each budget and metric, bold marks the best displayed value and underlining marks the second-best; emphasis is descriptive and does not denote statistical significance.

Threshold $q$	Exec. (%)	MAE (pp)	Spearman	Kendall
0.30	20.3	7.36	0.904	0.732
0.29	21.4	7.19	0.913	0.747
0.28	22.4	7.06	0.918	0.755
0.27	23.7	6.86	0.922	0.760
0.26	25.0	6.67	0.925	0.766
0.25	26.2	6.47	0.932	0.780

Appendix

Table 2: Complete historical-caching threshold sweep on 287 held-out runs. Rows are ordered by mean execution fraction. MAE is in percentage points.

Figure 8: Spearman ranking fidelity for the fixed-subset selection-by-estimation comparison. Higher values are better. All methods use the same 287 held-out evaluation runs and the same question budgets; random sampling is averaged over 20 seeds.

Figure 9: Kendall’s

\tau

ranking fidelity for the fixed-subset selection-by-estimation comparison. Higher values are better. The test population, budgets, and random-sampling seeds match Figure 8 .

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Random	Difficulty weighted	Difficulty gp-IRT	Hist. K-means weighted	Hist. K-means gp-IRT	IRT K-means weighted	IRT K-means gp-IRT
10	1.9	10.48	8.10	7.57	6.24	10.11	10.44	6.53
20	3.9	7.22	6.37	5.51	4.51	5.82	8.00	5.00
30	5.8	5.86	4.63	4.35	4.37	3.88	9.10	5.27
50	9.6	4.41	3.37	3.28	3.44	2.77	7.95	5.47
75	14.5	3.49	3.04	3.05	4.12	2.96	4.76	3.29

Appendix

Table 3: Complete fixed-subset selection and estimation results on 287 held-out runs. Bold and underlining mark the best and second-best displayed values at each budget, respectively.

Panel of Figure 10 — Figure 10: Ranking fidelity for the adaptive-model comparison. Higher values are better. Both panels use the same 287 held-out evaluation runs and question budgets as Figure 5 ; random sampling is averaged over 20 seeds.

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Random	Rasch adaptive	Multidimensional 2PL adaptive
10	1.9	10.48	10.94	5.07
20	3.9	7.22	9.79	3.90
30	5.8	5.86	6.50	3.27
50	9.6	4.41	6.46	2.81
75	14.5	3.49	3.51	2.39

Appendix

Table 4: Complete adaptive-model comparison on 287 held-out runs. Bold and underlining mark the best and second-best displayed values at each budget, respectively.

Agent family	Evaluation runs
Family A	51
Family B	146
Family C	56
Family D	34
Family E	12
Total	299

Appendix

Table 5: Breakdown of transferred evaluation runs by anonymized agent family. The labels match the family-level figures and tables.

Panel of Figure 11 — Figure 11: Pooled cross-agent ranking fidelity for a difficulty-stratified gp-IRT subset calibrated on the focal family and transferred unchanged to 299 runs from five held-out families. Higher values are better. Random sampling is averaged over 20 seeds.

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Original weighted	Original gp-IRT	Transfer random	Transfer weighted	Transfer gp-IRT
10	1.9	8.10	7.57	10.96	9.83	8.40
20	3.9	6.37	5.51	7.69	6.88	6.15
30	5.8	4.63	4.35	6.19	6.26	5.85
50	9.6	3.37	3.28	4.67	4.18	4.07
75	14.5	3.04	3.05	3.68	3.37	3.32

Appendix

Table 6: Complete pooled cross-agent transfer results. Original denotes the 287 calibration-agent test runs; Transfer denotes 299 pooled runs from five additional families. Emphasis is computed separately within the original and transferred populations.

Figure 12: Cross-agent pass-rate MAE by anonymized family for the transferred difficulty-stratified gp-IRT subset and 20-seed random sampling. Lower is better. Each panel compares methods on the same held-out runs within that family.

Figure 13: Cross-agent Spearman ranking fidelity by anonymized family. Higher is better. Correlations are computed within family, and random sampling is averaged over 20 seeds.

Figure 14: Cross-agent Kendall’s

\tau

ranking fidelity by anonymized family. Higher is better. Correlations are computed within family, and random sampling is averaged over 20 seeds.

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Random	Weighted	gp-IRT
10	1.9	11.93	9.83	8.66
20	3.9	8.37	6.42	5.74
30	5.8	6.90	6.95	6.95
50	9.6	4.92	4.08	4.02
75	14.5	3.86	3.17	3.23

Appendix

Table 7: Complete transfer results for anonymized Family A. Bold and underlining mark the best and second-best displayed values at each budget, respectively.

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Random	Weighted	gp-IRT
10	1.9	10.56	9.32	7.94
20	3.9	7.46	7.46	6.45
30	5.8	5.93	6.34	5.64
50	9.6	4.56	3.66	3.53
75	14.5	3.57	3.56	3.47

Appendix

Table 8: Complete transfer results for anonymized Family B. Bold and underlining mark the best and second-best displayed values at each budget, respectively.

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Random	Weighted	gp-IRT
10	1.9	10.66	10.14	8.57
20	3.9	7.54	6.03	5.68
30	5.8	6.04	5.31	5.01
50	9.6	4.55	4.16	4.26
75	14.5	3.65	3.16	3.11

Appendix

Table 9: Complete transfer results for anonymized Family C. Bold and underlining mark the best and second-best displayed values at each budget, respectively.

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Random	Weighted	gp-IRT
10	1.9	12.09	11.20	9.69
20	3.9	8.21	7.20	7.01
30	5.8	6.58	6.79	6.96
50	9.6	5.16	5.89	5.65
75	14.5	4.03	3.27	3.21

Appendix

Table 10: Complete transfer results for anonymized Family D. Bold and underlining mark the best and second-best displayed values at each budget, respectively.

(a) Pass-rate MAE (percentage points; lower is better)
$k$	Exec. (%)	Random	Weighted	gp-IRT
10	1.9	9.88	10.61	8.38
20	3.9	6.72	4.79	4.09
30	5.8	5.80	5.33	4.40
50	9.6	4.20	6.09	5.54
75	14.5	3.30	3.32	3.05

Appendix

Table 11: Complete transfer results for anonymized Family E. Bold and underlining mark the best and second-best displayed values at each budget, respectively.

Panel of Figure 15 — Figure 15: Ranking-fidelity sensitivity of difficulty-stratified fixed subsets to nested calibration windows ending Day 28. Higher values are better. Windows vary jointly in historical range and calibration-run count; all curves use the same 287 held-out evaluation runs.

(a) Pass-rate MAE (percentage points; lower is better)
		${k=100}$		${k=200}$		${k=300}$		${k=400}$
Start	Runs	Weighted	gp-IRT	Weighted	gp-IRT	Weighted	gp-IRT	Weighted	gp-IRT
Day 1	287	2.71	2.65	1.43	1.40	1.06	0.99	0.51	0.47
Day 4	258	2.86	2.97	1.42	1.35	1.14	1.06	0.53	0.52
Day 7	222	2.71	2.69	1.50	1.49	1.29	1.35	0.50	0.45
Day 10	192	2.39	2.36	1.66	1.59	0.97	0.98	0.52	0.52

Appendix

Table 12: Complete calibration-window sensitivity results. All windows end Day 28 and use the same 287 held-out runs. MAE is in percentage points. Values are not emphasized because the windows form a sensitivity sweep rather than prespecified competing methods.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

EvalBuilder: Can LLMs Build Efficient Benchmarks?
Oct 14, 2026Huaisheng Zhu, Hangfan Zhang, Teng Xiao +5Benchmark Construction AI Agent Benchmarks
Meituan · Pennsylvania State University · Allen Institute for AI (AI2) +2
EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents
Oct 14, 2026Qianhong Guo, Wei Xie, Xiaofang Cai +3LLM Agent Evaluation AI Agent Benchmarks
College of Computer Science and Technology, National University of Defense Technology · Institute of Information Engineering, Chinese Academy of Sciences
Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Apr 12, 2026Wanyi Chen, Xiao Yang, Xu Yang +7Agentic RL AI Agent Benchmarks
Soochow University · Microsoft Research Asia · Stony Brook University +2

Oct 14, 2026J/K move · Enter open · S save

EvalBuilder: Can LLMs Build Efficient Benchmarks?

Huaisheng Zhu, Hangfan Zhang, Teng Xiao, Siyuan Xu+4

Meituan · Pennsylvania State University · Allen Institute for AI (AI2)+2

Benchmark Construction AI Agent Benchmarks Automated Evaluation LLM Agent Evaluation

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the conclusions of costly reference evaluations? We study this capability by formulating automatic benchmark development as a meta-evaluation problem, where an agent is given a task and an expensive reference benchmark and requested to construct a lower-cost proxy benchmark that faithfully distinguishes among candidate models and solutions. Based on this formulation, we introduce EvalBuilder, a framework for evaluating such benchmark-generation capabilities along three key dimensions: fidelity, efficiency, and robustness. Specifically, we assess whether generated benchmarks preserve model rankings, generalize to held-out models and freshly collected responses, and provide well-calibrated and discriminative scores while substantially reducing evaluation cost. We further provide an iterative environment in which agents can refine their benchmarks under a fixed time budget. Our framework provides a systematic testbed and benchmark for understanding when current LLM agents can develop useful evaluators and how benchmark construction itself can be increasingly automated.

Abstract

LLM Agent Evaluation

Explore similar work

EvalBuilder: Can LLMs Build Efficient Benchmarks?

EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?