FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

Huaiming Li; Can Huang; Chufan Wu; Danyu Liu; Xiaomin Yuan; Qinze Li; Yui Lo; Wenhui Bai

cs.AIOct 10, 2026

FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

Authors: Huaiming Li, Can Huang, Chufan Wu, Danyu Liu, Xiaomin Yuan, Qinze Li, Yui Lo, Wenhui Bai, +13 more

Organizations: FM Agent Team, Baidu AI Cloud · Beijing University of Posts and Telecommunications · The Chinese University of Hong Kong · Tsinghua University · Peking University · University of the Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

Abstract

Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks. We introduce FDE-Bench, the first benchmark built from real customer projects to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization. Its 49 cases span combinatorial optimization and machine learning. Clarification targets are distilled from discussions between frontline practitioners and FDEs. Agents elicit missing requirements from a simulated customer, and case-specific evaluators assess final artifacts against acceptance criteria using customer-accepted FDE solutions as quality references. Across 18 agent configurations, the highest mean delivery score is only 0.5510, compared with an FDE reference of 1.0. The highest delivery success rate (DSR), defined as the fraction of planned runs reaching this reference, is only 6.12%. These results show that reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents on underspecified real-world business tasks. The benchmark project page is available at https://baidubce.github.io/FM-Agent/FDE-Bench/.

Figures & tables

Panel of Figure 1 — Figure 1: FDE-Bench: motivation and coverage. (a) Clarifying mixing rules guides iterative warehouse optimization. (b) 49 real customer cases span two task families.

Figure 2: FDE-Bench construction pipeline. Real-project materials become agent-visible inputs, annotated requirements, case-specific evaluation, and a reviewed case package.

Figure 3: FDE-Bench evaluation framework. Agents clarify requirements and iteratively optimize deliverables. After submission, hidden evaluation extracts and scores artifacts against FDE references without online feedback; clarification coverage is assessed separately.

	Delivery $\uparrow$				Clarification $\uparrow$
Agent / model	Quality	DSR	pass@3	pass^3	Recall	Precision	Ask-F1
General scaffold: opencode
GPT-6-Astra	0.5459	5.44	91.84	81.63	83.64	36.53	48.82
GLM-5.3	0.5413	1.36	91.84	77.55	66.83	39.70	47.46
Qwen3.8-Max	0.5387	2.72	95.92	73.47	70.56	35.41	45.11
Grok-4.6	0.5221	2.72	87.76	79.59	68.19	38.30	46.42

Table 1: End-to-end evaluation. 49 cases, three runs per case, Interact-Req . Quality is case-macro; other metrics are percentages. Bold marks blockwise maxima before rounding.

Figure 4: Clarification–delivery correlations. Cells report rank correlations across 18 configurations. Stars mark Holm-adjusted two-sided $p$ -values over 24 comparisons; down arrows denote lower-is-better diagnostics.

Agent / model	Hidden	Interact-Req	Full	$\bm{\Delta_{\rm req}}$	$\bm{\Delta_{\rm remain}}$
General scaffold: opencode
opencode + GPT-6-Astra	0.1803	0.5800	0.7461	+0.3997	+0.1662
opencode + DeepSeek-V4-Pro	0.1423	0.5063	0.6821	+0.3640	+0.1758
opencode + Gemini-3.1-Pro	0.1218	0.2912	0.4886	+0.1694	+0.1974
opencode + Kimi-K3	0.1244	0.3694	0.4488	+0.2450	+0.0794
Native scaffolds

Table 2: Information ablation. Case-macro Quality on 20 paired cases (three runs/condition). Bold marks maxima. Intervals and adjusted tests are in Appendix A.6 .

Figure 5: Oracle information coverage. Codex + GPT-6-Astra receives nested answer subsets; panels (a)–(d) show four cases and (e) their macro mean.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmarks / systems	Task type	Requirements	Clarification	Execution	Reference
MIPLIB-NL [ 12 ]	Optimization modeling	✗	✗	✓	✗
OptiMUS [ 9 ]	Optimization modeling	✗	✗	✓	✗
OptiBench [ 10 ]	Optimization modeling	✗	✗	✓	✗
CO-Bench [ 13 ]	CO search	✗	✗	✓	✗
HeuriGym [ 3 ]	CO search	✗	✗	✓	✗
ALE-Bench [ 14 ]	CO search	✗	✗	✓	✗

Appendix

Table 3: Comparison with representative benchmarks and systems. ✓: primary design feature; ✗: not a primary design feature. Requirements and Reference are defined above. Clarification: evaluation of clarifying questions or the use of user feedback; Execution: task outcomes verified through code or tool execution.

Figure 6: Case map. The 49 cases by task family and subtype, numbered 01–49.

No.	Case	Fam.	Raw objective	Hard constraints	Norm. / direction	FDE $R$
01	Vehicle-drag prediction	ML	mean relative $C_{d}$ error ( $\mathrm{Cd\_MRE}$ )	coverage, unique finite rows; $0.05\leq\mathrm{Cd}\leq 1.5$	$R/x$ -1pt	0.00357527
02	City delivery routing	CO	combined route cost (distance/time/penalties)	all required stops, feasible routes and valid timestamps	$R/x$ -1pt	281497
03	Crop-yield prediction	ML	yearly mean RMSE	required predictions, finite values and complete test coverage	$R/x$ -1pt	3.77281
04	Electricity-price forecasting	ML	weighted MAE/p95/p99/high-price underprediction error	complete horizon, finite values and aligned timestamps	$R/x$ -1pt	0.48004
05	Flight-booking forecasting	ML	seat-booking forecast score	complete horizon, finite nonnegative forecasts	$x/R$ -1pt	0.160653
06	Ion-beam uniformity	ML	mean WAPE-derived uniformity score	all required rows, finite predictions and valid bounds	$x/R$ -1pt	0.64572

Appendix

Table 4: Case-level evaluator reference. Numbering follows Figure 6 .

Candidate	Decision	Reason
“Is error MAE or relative error?”	Keep as one valid point	Changes the evaluator’s scoring convention.
“Does same-city supplier precedence override cost minimization?”	Keep as one valid point	Changes a routing/fulfillment rule and can change feasibility.
“Are out-of-range predictions rejected?”	Keep as one valid point	Changes a hard constraint.
“What Python library should I use?”	Exclude	Implementation choice is not scored.
“How many rows are in the visible CSV?”	Exclude	The fact is already observable to the agent.
“Which metric and which value range are required?”	Split into two points	Metric changes the score; range changes acceptance.

Appendix

Table 5: Boundary examples for clarification points.

Agent / model	Quality	DSR	pass@3	pass^3
General scaffold: opencode
GPT-6-Astra	0.546 [0.448, 0.647]	5.4 [0.7, 12.2]	91.8 [83.7, 98.0]	81.6 [71.4, 91.8]
GLM-5.3	0.541 [0.445, 0.632]	1.4 [0.0, 3.4]	91.8 [83.7, 98.0]	77.6 [65.3, 87.8]
Qwen3.8-Max	0.539 [0.444, 0.629]	2.7 [0.0, 6.1]	95.9 [89.8, 100.0]	73.5 [61.2, 85.7]
Grok-4.6	0.522 [0.423, 0.621]	2.7 [0.7, 5.4]	87.8 [77.6, 95.9]	79.6 [67.3, 89.8]
DeepSeek-V4-Pro	0.499 [0.404, 0.593]	1.4 [0.0, 4.1]	91.8 [83.7, 98.0]	71.4 [59.2, 83.7]

Appendix

Table 6: Complete results. Estimates with 95% case-bootstrap intervals; bold marks blockwise best values.

Annotation family	Type	Points	Share	Cases
Issue type	Missing	207	77.8%	48
	Ambiguous	45	16.9%	30
	Invalidated	9	3.4%	8
	Conflicting	5	1.9%	4
Content domain	Scoring convention	78	29.3%	39
	Constraint rule	65	24.4%	33

Appendix

Table 7: Composition of the clarification-point registry. 266 points in 49 cases.

Issue type	Mean Recall	Quality $\rho$	95% CI	Holm $p$
Missing	66.3%	0.808	[0.467, 0.882]	$<0.001$
Ambiguous	60.5%	0.801	[0.423, 0.884]	$<0.001$
Invalidated	60.5%	0.513	[0.004, 0.707]	0.029
Conflicting	77.5%	0.641	[0.101, 0.705]	0.008

Appendix

Table 8: Issue types and delivery. Spearman

\rho

with case-bootstrap 95% intervals.

Agent / model	Overall	Missing	Ambiguous	Invalidated	Conflicting
General scaffold: opencode
GPT-6-Astra	83.64	87.8	75.7	64.6	95.8
GLM-5.3	66.83	69.0	63.1	56.2	75.0
Qwen3.8-Max	70.56	70.7	67.8	81.2	83.3
Grok-4.6	68.19	71.2	66.5	64.6	70.8
DeepSeek-V4-Pro	55.08	58.0	46.5	52.1	70.8

Appendix

Table 9: Type-specific Recall (%) by configuration. Bold marks blockwise maxima.

Model	Constraint	Computation	Scoring	Data	Quality
GPT-6-Astra	26.7	25.4	28.8	19.1	0.542
GLM-5.3	30.7	24.5	26.6	18.3	0.534
Qwen3.8-Max	28.1	27.4	26.3	18.2	0.523
Grok-4.6	31.5	29.7	21.7	17.2	0.519
DeepSeek-V4-Pro	33.0	27.9	20.1	19.0	0.512
Kimi-K3	31.1	26.1	24.6	18.2	0.417

Appendix

Table 10: Domain mix of matched questions by model (%). Quality is the mean over the model’s systems.

Agent / model	$\bm{\Delta_{\rm req}}$	$\bm{\Delta_{\rm remain}}$	$\bm{\Delta_{\rm total}}$
opencode + GPT-6-Astra	+0.3997 ∗ [0.2284, 0.5983]	+0.1662 [0.0369, 0.3167]	+0.5658 ∗ [0.3941, 0.7367]
opencode + DeepSeek-V4-Pro	+0.3640 ∗ [0.2367, 0.5046]	+0.1758 ∗ [0.0893, 0.2733]	+0.5398 ∗ [0.4026, 0.6741]
opencode + Gemini-3.1-Pro	+0.1694 ∗ [0.0824, 0.2684]	+0.1974 ∗ [0.0810, 0.3276]	+0.3668 ∗ [0.2266, 0.5130]
opencode + Kimi-K3	+0.2450 ∗ [0.1326, 0.3595]	+0.0794 [-0.0125, 0.1740]	+0.3244 ∗ [0.2137, 0.4438]
codex + GPT-6-Astra	+0.4000 ∗ [0.2211, 0.5751]	+0.1966 ∗ [0.0853, 0.3228]	+0.5966 ∗ [0.4348, 0.7473]
deepseek-harness + DeepSeek-V4-Pro	+0.3644 ∗ [0.2344, 0.5002]	+0.1697 ∗ [0.0689, 0.2949]	+0.5341 ∗ [0.3960, 0.6677]

Appendix

Table 11: Paired Quality differences between information conditions. ∗ : Holm-adjusted

p<0.05

Condition	Quality	Ask rate	$\bm{\#Q}$	Recall	Precision	Ask-F1
opencode + GPT-6-Astra
Interact	0.3262	100.00%	15.00	63.61%	47.72%	53.84%
Interact-Req	0.3830	100.00%	36.27	85.24%	49.04%	61.19%
opencode + DeepSeek-V4-Pro
Interact	0.4061	100.00%	10.20	37.18%	56.92%	43.73%
Interact-Req	0.5552	100.00%	21.13	49.76%	53.51%	50.62%

Appendix

Table 12: Optional versus required clarification on five cases.

Case	0%	25%	50%	75%	100%
09 Nuclear-site scheduling	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.3272 $\pm$ 0.5667	0.3206 $\pm$ 0.5553	0.9770 $\pm$ 0.0109
11 Retail-zone routing	0.0000 $\pm$ 0.0000	0.4661 $\pm$ 0.4037	0.7442 $\pm$ 0.0629	0.7935 $\pm$ 0.0670	0.8558 $\pm$ 0.0582
17 Cold-rolling sequencing	0.2347 $\pm$ 0.1189	0.1637 $\pm$ 0.0399	0.3357 $\pm$ 0.1595	0.4260 $\pm$ 0.1418	0.6782 $\pm$ 0.2787
38 Display-fab scheduling	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.3190 $\pm$ 0.5525	0.9650 $\pm$ 0.0093
Four-case mean	0.0587	0.1575	0.3518	0.4648	0.8690

Appendix

Table 13: Oracle Quality by answer coverage. Mean

\pm

SD over three repeats.

Recorded signature	Runs	Share	Interpretation
Constraint/feasibility violation	430	78.5%	Case constraints or feasibility checks are explicitly violated.
Fatal case-specific check	29	5.3%	A case-defined fatal condition is recorded, such as incomplete coverage or an out-of-horizon event.
Schema/output validation error	8	1.5%	The submitted representation fails a required schema or output check.
Evaluator/scoring execution exception	4	0.7%	The evaluator records an exception or cannot form the required score.
Quality/threshold warning	16	2.9%	A quality or threshold warning accompanies zero Quality for a valid artifact.
No explicit evaluator message	61	11.1%	The artifact is scored zero but the retained record does not identify a more specific reason.

Appendix

Table 14: Evaluator signatures of 548 zero-score runs.

Stage	Recorded evidence
Customer request	JSON replenishment plan; internal store/warehouse transfers must be exhausted before supplier purchase, and same-city stock cannot be skipped; stockout avoidance takes priority over timeliness and cost.
Agent questions	27 messages (47 scored question units) over three rounds. The agent explicitly asked about source precedence (“same-city stores and local warehouse $\rightarrow$ cross-city sources $\rightarrow$ supplier”) and about how shortages and lead times should be computed.
Key rule and coverage	The complete business information specifies receive arrivals $\rightarrow$ settle sales $\rightarrow$ dispatch transfers and purchases. The registered point “daily inventory transition order” requires this exact order. The stored majority label is covered=false , with 0.60 judge agreement (2 covered and 3 uncovered votes). No matched question was recorded for this blocker.
Submitted plan	The plan metadata nevertheless declares the same three-step order. It contains 15,987 instructions (15,515 transfers and 472 purchases), and the agent’s validator reports pass .
Executable evaluation	All five extractor/evaluator routes mark the plan invalid and return Quality $=0.0$ . The evaluator records 347 constraint violations, including eight same-city exhaustion violations for ITEM_000092 .
Reference	The customer-accepted reference has value 0.9974756. For the eight affected stores, it contains no item-092 supplier purchase and routes 10.58 units through internal transfers (seven stores receive a positive quantity), in contrast to the submitted plan.

Appendix

Table 15: Case-10 run.

Criterion	$n$	A	B	Agreement	$\bm{\kappa}$
Answer support (all labels)	1303	1250/51/2	1179/117/7	92.10%	0.385
Answer support (non-uncertain)	1170	—	—	99.57%	0.285
Cross-turn status	1303	968/320/15	965/323/15	99.00%	0.974
Run factuality (all labels)	105	79/24/2	71/28/6	82.86%	0.601
Run factuality (non-uncertain)	73	—	—	91.78%	0.218
Run consistency	105	60/36/9	60/35/10	97.14%	0.948

Appendix

Table 16: Audit results.

Agent / model	Failures / slots	Rate	No artifact
opencode
GPT-6-Astra	3/735	0.41%	0
GLM-5.3	9/721	1.25%	4
DeepSeek-V4-Pro	5/731	0.68%	2
Gemini-3.1-Pro	6/733	0.82%	0
Grok-4.6	2/733	0.27%	0

Appendix

Table 17: Extraction failures by configuration.

Explore similar work

RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
Jun 2, 2026Zongwei Lv, Zhewen Tan, Yaoming Li +7Computer-Use Agent Benchmarks AI Coding Agents
1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*
StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
Aug 18, 2026Liya Zhu, Xin Ma, Tao Liu +35AI Agent Benchmarks LLM Agent Evaluation
ByteDance Seed · Nanjing University · M-A-P
AlphaEval: Evaluating Agents in Production
Apr 14, 2026Pengrui Lu, Bingyu Xu, Wenjun Zhang +24AI Agent Evaluation AI Agent Benchmarks
SII · SJTU · GAIR +9

Jun 2, 2026J/K move · Enter open · S save

RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions

Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao+6

1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*

Computer-Use Agent Benchmarks AI Coding Agents Software Engineering Benchmarks AI Agent Benchmarks Automated Evaluation Agent Benchmarks

Open paper AskPDF arXiv

Agent benchmarks should reflect what users actually ask deployed agents to do, yet existing benchmarks often miss key realism properties of real developer-agent sessions. We introduce RealClawBench, a live benchmark framework built from real OpenClaw sessions to capture the distribution, diversity, and real-world difficulty of deployed agent use. Real user requests are challenging to benchmark because they often depend on local execution environments, involve implicit or underspecified intent, and require nontrivial verification. RealClawBench addresses these challenges with two core mechanisms: reconstructed execution environments and deterministic verifiable scorers, which together convert real sessions into reproducible, automatically scored tasks. The resulting release contains 281 executable tasks sampled from a much larger real-session pool while preserving the source distribution, with maximum final-vs-source Jensen-Shannon divergence of 0.0448. Evaluating 14 contemporary models shows that the best system solves only 65.8% of tasks, revealing substantial headroom on realistic developer-agent workloads. By turning real deployed sessions into controlled evaluation instances, RealClawBench provides a practical path toward benchmarks that better measure agent capability in actual use. Code is available at:https://anonymous.4open.science/r/real-claw-bench-582B.

Abstract

Explore similar work

RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

AlphaEval: Evaluating Agents in Production