Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities. We present SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time. Each data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs. First, given a scenario and target complexity dimensions, SynthCorp synthesizes reusable event types and compiles each into a program over predefined simulation primitives. Second, SynthCorp schedules event instances on a virtual calendar, interleaving routine and disruption events so that tables, documents, and catalog entries remain consistent. Third, SynthCorp executes the calendar to materialize the data estate and records every operation in a simulation log. Fourth, SynthCorp generates analytical tasks and derives verified gold programs from the log. We use SynthCorp to generate initial evaluation benchmarks comprising seven data estates (up to 1K operational tables) and 256 tasks across five domains. The data estates pass deterministic pipeline checks (such as foreign-key violations, undeclared writes, recurrence violations, and database execution failures) with a near-zero error rate, introduce cross-asset complexities, and cover all requested dimensions. Evaluating three models across four data-context settings shows that the benchmarks separate data contexts and model tiers. Auditing candidate queries against causal provenance in the simulation log verifies gold programs without manual annotation, including on multi-event composite tasks that expose reasoning failures in state-of-the-art agents. We hope the community uses SynthCorp to generate more challenging and useful data estates and benchmarks that foster the development of enterprise data agents.
Figures & tables
Figure 1: SynthCorp builds a data estate and benchmark from business events.
Line of work
Controllable Complexity
Schema Evolution
Virtual Calendar
Assets
Data Provenance
Verified Tasks
Curated benchmarks
( Lai et al., 2026 ; Lei et al., 2025 ; Chen et al., 2024 )
✗
✗
✗
T, D
✗
✗
Enterprise simulators
( Flynt, 2026 ; Zeng et al., 2026 )
✗
✗
✓
T, D
✓
✗
Enterprise sandboxes
( Huang et al., 2025 ; Xu et al., 2026 ; Gruenbaum et al., 2026 )
∘
✗
∘
T, D
∘
∘
Table 1: Comparison of SynthCorp with related work. Assets are tables (T), documents (D), and catalog (C). ✓ full, ∘ partial, ✗ none, and New A logs only.
Figure 2: The SynthCorp pipeline. The event synthesizer (Stage 1), the compiler (Stage 2), the scheduler and the execution engine (Stage 3), and the benchmark generator (Stage 4) each pair LLM generation with human or programmatic verification and pass a validated artifact to the next stage.
Dimension
Effect on the data estate and example event
Business glossary
Domain terms or codes require definitions outside the schema, e.g., PublishGlossaryPolicyEvent defines POS status code 4 as a voided order.
Implicit business rules
Documents state rules that the schema does not enforce, e.g., ReviseDiscountPolicy writes a memo excluding staff-discounted sales from revenue.
Frozen snapshots and redundant copies remain beside authoritative tables, e.g., AnalystAuditSnapshot retains an audit snapshot.
Schema drift
Schema changes leave historical rows without new attributes, e.g., UpgradeMenuEvent freezes the snapshot and adds milk_choice to the authoritative table.
Table 2: Complexity dimensions and representative events.
Primitive
Data
Operation
Generative
CreateSchema
Ddb,C
Generates a table schema from a business description and registers it in C .
GenDocument
Ddoc,C
Generates a document, inserts it into Ddoc , and registers it and its glossary terms in C .
CreateDistribution
–
Defines a probability distribution over a specified domain.
GenValue
–
Generates a value consistent with the supplied description.
HoldConversation
Ddoc
Generates a multi-turn stakeholder dialogue and records the decision outcome.
Table 3: The twelve simulation primitives in S . marks the category whose primitives invoke an LLM.
Retail-10
Retail-100
Retail-1000
Music store
Banking
Logistics
Insurance
Complexity
SD
SD
All
IBR
BG
TR, MG
IBR, MG, SD
Horizon (days)
14
14
28
60
120
30
120
Event types (R + D)
11+2
27+1
59+31
5+5
9+3
8+5
7+4
Operators
59
299
1.3K
90
91
86
92
Event instances
65
153
834
147
152
177
585
Operator invocations
280
1.7K
15.7K
545
3.2K
1.2K
4.5K
Table 4: The seven SynthCorp data estates and benchmarks.
Table 5: Pipeline correctness on the development and held-out scenarios. The LLM judge evaluates the underlined error types, and the rule engine evaluates the rest. Parentheses give the counting unit. Each cell reports affected over total units (and the error rate when nonzero). Shading follows the error rate ( 0.01.0 ).
BIRD financial
SynthCorp Banking
Data
Static snapshot of a Czech bank (8 tables)
120 simulated days (9 base tables, 12 dated snapshots, 4 views)
History
None
Simulation log of 152 event instances
Knowledge
Column descriptions and per-question external-knowledge evidence
7 event-generated documents, no column descriptions
Catalog
–
144 glossary terms linked to source documents
Questions
Human-written
Generated from benchmark hints and detectors
Gold program
Human-written
Log-derived and debate-verified
Table 6: BIRD financial versus the SynthCorp Banking benchmark on the same Czech bank schema.
Retail-10
Retail-100
Retail-1000
Music store
Banking
Logistics
Insurance
All
#Tasks
37
31
62
28
26
19
53
256
Gemma 4 26B-A4B
S1 (Schema only)
27.9 ± 7.1
64.5 ± 8.0
22.0 ± 4.9
16.7 ± 7.0
37.2 ± 9.5
66.7 ± 9.9
27.0 ± 6.0
33.3 ± 2.8
S2 (Schema + top-1 document)
43.2 ± 7.7
67.7 ± 8.2
26.9 ± 5.3
28.6 ± 8.7
44.9 ± 9.2
47.4 ± 11.8
49.1 ± 6.6
42.3 ± 3.0
S3 (Agent + search tools)
63.1 ± 7.7
64.5 ± 8.0
34.9 ± 5.5
63.1 ± 8.1
64.1 ± 7.6
64.9 ± 9.7
52.8 ± 6.3
54.6 ± 2.8
S4 (Agent + files and CLI)
59.5 ± 7.7
60.2 ± 7.9
23.7 ± 4.7
50.0 ± 7.4
56.4 ± 7.6
59.6 ± 10.7
47.8 ± 6.3
47.1 ± 2.8
Table 7: Mean execution accuracy ( ± standard error, %) across four data-context settings on all 256 tasks over three runs. Bold marks each model’s best setting, and bold underline marks each column’s best result.
Table 8: Data realism. The top panel compares Banking with the real Czech bank data of BIRD financial . The bottom panel tests five stylized facts across all data estates against real references. Shading follows close to real departs from real .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Retail-10
Retail-100
Retail-1000
Music store
Banking
Logistics
Insurance
Total tables
33
272
1.4K
11
25
60
126
report snapshots / views
14/8
105/66
347/41
0/0
12/4
34/4
102/6
base tables
11
101
1.0K
11
9
22
18
reference
5
35
264
9
7
4
8
transaction
4
46
389
2
1
10
8
log
0
5
76
0
0
3
0
Appendix
Table 9: Breakdown of tables, rows, and documents across the seven SynthCorp data estates, extending Table 4 .
Component
Error type
Counting unit
Detection criterion and example
Compiled SP programs
Cross-column/row dependency violation
Dependencies
The program relaxes or violates a specified cross-table, cross-row, or intra-row dependency, e.g., sampling correlated columns independently or replacing a row formula with an aggregate.
Missing required output
Required outputs
The program omits the operations that produce a required table, column, document, status transition, or intra-day timing window.
Missing stochastic sampling
Value-producing operators
The program omits a required distribution draw or replaces it with a constant, e.g., a fixed customer count where the specification requires Poisson arrivals.
Unbound variable
Variables
No execution record binds the variable, so downstream operators read an empty value.
Unintended output overwrite
Operators
The program of a recurring event type writes a fixed output name, so a later event instance overwrites a table, view, or document that the specification preserves.
Foreign key violation
Tables
After execution, rows of the table reference keys absent from the parent table, e.g., an order references a nonexistent customer_id .
Appendix
Table 10: Detection criteria and counting units for the 14 pipeline error types in Table 5 , grouped by stage component. The LLM judge evaluates the underlined error types, and the deterministic rule engine checks the rest.
Metric
Unit
Real reference
Retail-10
Retail-100
Retail-1000
Music store
Banking
Logistics
Insurance
Banking vs. BIRD financial
Weekday mix, TVD
transactions
–
0.029
Amount log-shape, KS / W1
transactions
–
0.236 / 0.401
Amount median (CZK)
transactions
2,100
934
Loan amount log-shape, KS / W1
loans
–
0.212 / 0.122
Loan duration / card type / statement frequency mix, TVD
rows
–
0.104 / 0.128 / 0.176
Appendix
Table 11: Full data-realism results extending Table 8 . Each cell reports the metric value and, in parentheses, its sample size. The top block compares Banking against BIRD financial , the middle block reports stylized facts across all seven estates, and the bottom block reports domain-specific facts.
Retail-10
Retail-100
Retail-1000
Music store
Banking
Logistics
Insurance
Doc-lookup / numeric
5/13
8/10
6/12
0/10
1/6
1/6
4/21
Requested dim. / complex tasks
3/29
2/21
47/48
11/23
15/20
8/14
33/43
Distinct skeletons / gold
34/37
30/31
57/62
27/28
24/26
19/19
44/53
Join
70%
48%
48%
64%
69%
58%
57%
Grouping
70%
87%
84%
64%
92%
84%
85%
Nesting
54%
55%
56%
39%
58%
74%
66%
Appendix
Table 12: Complexity and SQL diversity of the seven benchmarks. Fractions report affected tasks over checked tasks, and Join, Grouping, and Nesting report the share of gold programs that use each SQL construct.
Tool-calling agents have become central to enterprise AI, yet training and evaluating them at scale remains severely constrained due to business and legal restrictions on enterprise systems, data, and database schemas. Tabular data synthesis offers a natural alternative, but its effectiveness is fundamentally limited by structural validity and schema availability, while procedure-based approaches yield the opposite weakness, typically lacking distributional fidelity without per-domain authoring. We introduce Synthesis Through Simulation (STS), a schema--free data synthesis paradigm in which an LLM agent generates data by executing operations against policy-enforcing APIs within simulated enterprise environments. Because data is generated through the same environment that defines what is valid, STS guarantees structural validity by construction while decoupling validity enforcement from distribution modeling, allowing each to be addressed independently. The Generalist Populator (GP), STS's domain-agnostic agent, addresses the remaining challenges of distributional fidelity and synthesis scalability: GP achieves 0.88 average marginal fidelity and 100% constraint satisfaction across all ten environments without access to DB schemas, while statistical synthesizers are inapplicable to seven due to necessary seed data requirements, and schema-privileged agents fail 82% of trajectories on airline environment's tightly coupled workflows due to brittle task composition. We open-source the full framework, all ten environments, and generated datasets at https://github.com/SAP/synthesis-through-simulation.
LLM agents for enterprise systems of record cannot be evaluated on customer production data, and no existing substitute provides ground truth. We present the Era by Eon Benchmark for evaluating LLM agents that use enterprise tools. The benchmark is built around a complete fictional company. It includes product simulators, company-specific internal databases, benchmark questions, and computed answer keys. Industry, company size, business model, application portfolio, and a seed define each company. One seeded entity graph supplies shared company data to simulators of Salesforce, Zendesk, Slack, Gong, and other products. A questionconditioned generator creates the schemas and records for internal databases. It takes shared entities, keys, and values from the same graph before generating database-specific facts. Both mechanisms therefore describe one consistent enterprise estate. Every expected answer is computed from the final records, so grading is exact. Design and answer-key checks validate the internal databases. A realism scorecard and adversarial detector validate the entity graph. Across 23 generated companies, the mean realism score rose from 61.8 to 97.0, with zero records flagged as synthetic. In the reported simulator-track comparison, nine models answered the same 33 questions three times each. Accuracy estimates ranged from 42.4% to 76.8%, and three of 36 pairwise differences remained supported after correction.
Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3
Real-world enterprise data science and analytics workflows require reasoning across dozens of tables, performing statistical analyses, and acting on the results. Established text-to-SQL benchmarks evaluate query generation alone, and audits have found their answer keys frequently wrong. Because real enterprise warehouses are too sensitive to release, these benchmarks are built on public datasets where a business event fits in a single table. We introduce Argo-Bench, an evaluation framework comprising 210 data science and analytics tasks. Drawing on public data, peer-reviewed industry literature, and regulatory filings, we simulate a food delivery platform in New York City at true scale, with 81 million orders in 2024, grounded economics, fraud patterns, and marketplace incentives. We export this world to an ERP warehouse of 235 tables and 7.5 billion rows, modeled on the Oracle E-Business Suite schema. The simulator's ground-truth state is withheld from the warehouse the agent sees, so tasks require reconstructing facts by navigating the warehouse before acting on them. Argo-Bench goes beyond text-to-SQL: the agent files actions such as banning fraudulent accounts, allocating courier incentive budgets, or issuing back pay, and the grader scores each by its consequences in the simulator. Every task has an executable reference solution that demonstrates solvability using only the warehouse. The strongest of 14 frontier and open-weight models scores 95 or higher on only 34.8% of tasks and averages 59.5 points. We hope Argo-Bench drives progress toward agents that understand, navigate, and act within real data environments.
Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing +2