Organizations: DeepLethe · Beijing University of Posts and Telecommunications · Tsinghua University · Shanghai Jiao Tong University · Beihang University · Xiamen University · Huawei · Harvard University · Nanjing University · Atlassian · Beijing Jiaotong University · Tianjin University · Meta · Shanghai University of Finance and Economics · Microsoft Research Asia · Stanford University · University College London · UWits Technology (Beijing) Co., Ltd. · Bopai Management Consulting (Shenzhen) Co., Ltd. · Exploration and Development Research Institute, PetroChina Southwest Oil & Gasfield Company · Cornell University · Kunlun Digital Technology Co., Ltd.
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.
Figures & tables
Figure 3: Components of an FDE-Gym task, showing the inputs, agent actions and deliverable, deployment and evaluation of the delivered system, and interpretation of the resulting score. The agent reads the task inputs and exchanges messages with customer stakeholders, who answer questions and may also initiate contact. A task is included only if the scoring anchors shown on the axis are satisfied.
Level
Information provided in the task instruction
What the agent must determine
L0
The underlying problem, applicable rules and constraints, and how the result is scored
How to design and implement an effective solution
L1
A clear statement of the underlying problem
The applicable rules and constraints, and how to design and implement a solution
L2
A vague but directionally correct request
The problem to be solved, the applicable constraints, and how to design and implement a solution
L3
The sponsor’s original request, which may target the wrong problem
Whether the request addresses the underlying problem, which constraints apply, and how to design and implement a solution
Table 1: The four task instruction levels. The workspace, stakeholders, and grading criteria remain identical across levels; only the task instruction varies.
Figure 4: A worked example: the drilling scenario ( oil_npt ), original task.
Scenario
What is delivered
Sector
Tasks
Evaluated on
bank_scam_hold
Anti-scam hold queue agent
Banking
3
523 queue calls
build_pay_cutover
Subcontract payment-ledger cut-over
Construction
4
one cut-over and its claims
campus_booking
Space-booking assistant
Education
5
82 conversations
charge_care_config
No-code driver-support agent
EV charging
4
the customer’s test suite
estate_repair_cases
Agent for fund-paid major repairs
Property mgmt.
4
145 cases over 63 days
fund_recon
Reconciliation break clearing
Asset management
8
188 breaks
Table 2: The 20 scenarios of FDE-Gym v1.0. A scenario’s tasks (the original and its variants) share the same customer environment and differ in the key facts behind the problem; there are 95 tasks in all. The last column gives the held-out data on which the original task is evaluated.
Original tasks
All 95 tasks, L3
Configuration
L0
L1
L2
L3
Score
95% CI
U
C
Resolved
< 0
Claude Opus 5.5
0.82
0.62
0.54
0.55
0.56
0.50, 0.62
0.70
0.14
14
5
Claude Sonnet 5.5
0.74
0.45
0.47
0.46
0.49
0.43, 0.55
0.67
0.18
15
7
GPT-6 Astra
0.73
0.39
0.34
0.33
0.38
0.30, 0.46
0.60
0.22
9
18
GPT-6.1 Sol
0.76
0.33
0.28
0.43
0.37
0.28, 0.44
0.57
0.20
7
17
GLM-5.3 Flash
0.68
0.29
0.38
0.24
0.22
0.11, 0.31
0.40
0.18
8
28
Table 3: Main results. Left: score by task instruction level on the 20 original tasks. Right: all 95 tasks at L3, with the 95% interval, the normalized business uplift before stakeholder contact cost, the stakeholder contact cost, and the number of tasks resolved (score ≥ 0.8, no incident) and the number of tasks finished below the do-nothing baseline (score < 0).
Figure 5: (a) Share of key stakeholder facts obtained against normalized business uplift, at L3. Leading models learn most of the facts and deliver 0.60–0.70 of the reference uplift. (b) Mean model usage cost per run against score at L3.
Figure 6: Distribution of each model’s 95 tasks at L3 by score band. Scores of 0.2 or below correspond to the cap on literal compliance with the sponsor’s task instruction.
Figure 7: Mean score at L3 on each scenario, for each model; scenarios are ordered by the best model’s score, and the highest-scoring cell in each row is outlined. Color runs from red (below the do-nothing baseline) to blue (at the reference solution).
Figure 8: Model performance on Terminal-Bench 2.1, Vibe Code Bench, Finance Agent, and FDE-Gym.
Building deployable artifacts that pass gated checks; diagnosing and mitigating incidents
Customer interaction, requirements discovery, and business outcomes
Delivery
ττ -bench [ 9 ] , Trains but Doesn’t Learn [ 10 ]
Building an agent, or a fine-tuned model, for a client and judging what was delivered after deployment
Task instructions varying from a full statement of the rules to a request that may be misdirected; uplift on a business metric minus stakeholder contact cost; many sectors and forms of deliverable
Table 4: Prior benchmarks by the delivery capability they measure, and what FDE-Gym adds to each.