Organizations: DeepLethe · Beijing University of Posts and Telecommunications · Tsinghua University · Shanghai Jiao Tong University · Beihang University · Xiamen University · Huawei · Harvard University · Nanjing University · Atlassian · Beijing Jiaotong University · Tianjin University · Meta · Shanghai University of Finance and Economics · Microsoft Research Asia · Stanford University · University College London · UWits Technology (Beijing) Co., Ltd. · Bopai Management Consulting (Shenzhen) Co., Ltd. · Exploration and Development Research Institute, PetroChina Southwest Oil & Gasfield Company · Cornell University · Kunlun Digital Technology Co., Ltd.
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.
Figures & tables
Figure 3: Components of an FDE-Gym task, showing the inputs, agent actions and deliverable, deployment and evaluation of the delivered system, and interpretation of the resulting score. The agent reads the task inputs and exchanges messages with customer stakeholders, who answer questions and may also initiate contact. A task is included only if the scoring anchors shown on the axis are satisfied.
Level
Information provided in the task instruction
What the agent must determine
L0
The underlying problem, applicable rules and constraints, and how the result is scored
How to design and implement an effective solution
L1
A clear statement of the underlying problem
The applicable rules and constraints, and how to design and implement a solution
L2
A vague but directionally correct request
The problem to be solved, the applicable constraints, and how to design and implement a solution
L3
The sponsor’s original request, which may target the wrong problem
Whether the request addresses the underlying problem, which constraints apply, and how to design and implement a solution
Table 1: The four task instruction levels. The workspace, stakeholders, and grading criteria remain identical across levels; only the task instruction varies.
Figure 4: A worked example: the drilling scenario ( oil_npt ), original task.
Scenario
What is delivered
Sector
Tasks
Evaluated on
bank_scam_hold
Anti-scam hold queue agent
Banking
3
523 queue calls
build_pay_cutover
Subcontract payment-ledger cut-over
Construction
4
one cut-over and its claims
campus_booking
Space-booking assistant
Education
5
82 conversations
charge_care_config
No-code driver-support agent
EV charging
4
the customer’s test suite
estate_repair_cases
Agent for fund-paid major repairs
Property mgmt.
4
145 cases over 63 days
fund_recon
Reconciliation break clearing
Asset management
8
188 breaks
Table 2: The 20 scenarios of FDE-Gym v1.0. A scenario’s tasks (the original and its variants) share the same customer environment and differ in the key facts behind the problem; there are 95 tasks in all. The last column gives the held-out data on which the original task is evaluated.
Original tasks
All 95 tasks, L3
Configuration
L0
L1
L2
L3
Score
95% CI
U
C
Resolved
< 0
Claude Opus 5.5
0.82
0.62
0.54
0.55
0.56
0.50, 0.62
0.70
0.14
14
5
Claude Sonnet 5.5
0.74
0.45
0.47
0.46
0.49
0.43, 0.55
0.67
0.18
15
7
GPT-6 Astra
0.73
0.39
0.34
0.33
0.38
0.30, 0.46
0.60
0.22
9
18
GPT-6.1 Sol
0.76
0.33
0.28
0.43
0.37
0.28, 0.44
0.57
0.20
7
17
GLM-5.3 Flash
0.68
0.29
0.38
0.24
0.22
0.11, 0.31
0.40
0.18
8
28
Table 3: Main results. Left: score by task instruction level on the 20 original tasks. Right: all 95 tasks at L3, with the 95% interval, the normalized business uplift before stakeholder contact cost, the stakeholder contact cost, and the number of tasks resolved (score ≥ 0.8, no incident) and the number of tasks finished below the do-nothing baseline (score < 0).
Figure 5: (a) Share of key stakeholder facts obtained against normalized business uplift, at L3. Leading models learn most of the facts and deliver 0.60–0.70 of the reference uplift. (b) Mean model usage cost per run against score at L3.
Figure 6: Distribution of each model’s 95 tasks at L3 by score band. Scores of 0.2 or below correspond to the cap on literal compliance with the sponsor’s task instruction.
Figure 7: Mean score at L3 on each scenario, for each model; scenarios are ordered by the best model’s score, and the highest-scoring cell in each row is outlined. Color runs from red (below the do-nothing baseline) to blue (at the reference solution).
Figure 8: Model performance on Terminal-Bench 2.1, Vibe Code Bench, Finance Agent, and FDE-Gym.
Building deployable artifacts that pass gated checks; diagnosing and mitigating incidents
Customer interaction, requirements discovery, and business outcomes
Delivery
ττ -bench [ 9 ] , Trains but Doesn’t Learn [ 10 ]
Building an agent, or a fine-tuned model, for a client and judging what was delivered after deployment
Task instructions varying from a full statement of the rules to a request that may be misdirected; uplift on a business metric minus stakeholder contact cost; many sectors and forms of deliverable
Table 4: Prior benchmarks by the delivery capability they measure, and what FDE-Gym adds to each.
Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks. We introduce FDE-Bench, the first benchmark built from real customer projects to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization. Its 49 cases span combinatorial optimization and machine learning. Clarification targets are distilled from discussions between frontline practitioners and FDEs. Agents elicit missing requirements from a simulated customer, and case-specific evaluators assess final artifacts against acceptance criteria using customer-accepted FDE solutions as quality references. Across 18 agent configurations, the highest mean delivery score is only 0.5510, compared with an FDE reference of 1.0. The highest delivery success rate (DSR), defined as the fraction of planned runs reaching this reference, is only 6.12%. These results show that reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents on underspecified real-world business tasks. The benchmark project page is available at https://baidubce.github.io/FM-Agent/FDE-Bench/.
Huaiming Li, Can Huang, Chufan Wu +18
FM Agent Team, Baidu AI Cloud · Beijing University of Posts and Telecommunications · The Chinese University of Hong Kong +4
Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduce StartupBench, an E2E agent benchmark grounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unified agent harness, even the strongest model successfully completes only approximately 30% of StartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects like complex instruction following and domain-specific expertise as major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks.
Deployment requires an agent to turn application code into a running system whose services connect, become ready, and remain observable. FDE-Bench evaluates this capability with 136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes. Agents submit declarative artifacts that are collected, rebuilt, and redeployed in a pristine environment. Four gated binary check layers measure build, readiness, behavior, and conformance to the deployment specification, using programmatic checks without an LLM judge. A four-arm release gate requires a resolving reference solution and rejects tasks solved by do-nothing, specification-transcription, or generic-stub submissions. The released check annotations expose the link between 2,145 checks and their specifications, including seven documented gaps. Three additional adversarial strategies test shortcuts in the grading signals; none resolves any of the 135 tasks they cover, while a vacuous health probe passes readiness and exposes the need for downstream checks. On the 136-task evaluation grid, seven language models from four providers use the same four-tool scaffold and resolve 52.9-75.0 percent of tasks. The three zero-intelligence floors resolve none and reach a mean Deployment Score of at most 0.44. Readiness is the largest failure stage, accounting for 110 of 313 unresolved episodes. Mean resolution rate is 30.7 percentage points higher on the repair task group than on the disjoint greenfield group, with a positive gap for every model; ten tasks resist all seven. In a 25-task case study, one practicing engineer directing Claude-Sonnet-5 resolves 92 percent against 72 percent for the autonomous baseline. FDE-Bench links deployment success and failure to artifacts that can be inspected and replayed.
Weihang Ding, Junfei Zhan, Yueting Li +1
University of California, Berkeley · Imperial College London · The Hong Kong University of Science and Technology (Guangzhou)