SynthCorp: A Platform for Creating Benchmarks for Data Agents
Organizations: Google
Abstract
Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities. We present SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time. Each data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs. First, given a scenario and target complexity dimensions, SynthCorp synthesizes reusable event types and compiles each into a program over predefined simulation primitives. Second, SynthCorp schedules event instances on a virtual calendar, interleaving routine and disruption events so that tables, documents, and catalog entries remain consistent. Third, SynthCorp executes the calendar to materialize the data estate and records every operation in a simulation log. Fourth, SynthCorp generates analytical tasks and derives verified gold programs from the log. We use SynthCorp to generate initial evaluation benchmarks comprising seven data estates (up to 1K operational tables) and 256 tasks across five domains. The data estates pass deterministic pipeline checks (such as foreign-key violations, undeclared writes, recurrence violations, and database execution failures) with a near-zero error rate, introduce cross-asset complexities, and cover all requested dimensions. Evaluating three models across four data-context settings shows that the benchmarks separate data contexts and model tiers. Auditing candidate queries against causal provenance in the simulation log verifies gold programs without manual annotation, including on multi-event composite tasks that expose reasoning failures in state-of-the-art agents. We hope the community uses SynthCorp to generate more challenging and useful data estates and benchmarks that foster the development of enterprise data agents.
Figures & tables
| Line of work | Controllable Complexity | Schema Evolution | Virtual Calendar | Assets | Data Provenance | Verified Tasks |
|---|---|---|---|---|---|---|
| Curated benchmarks | ||||||
| ( Lai et al., 2026 ; Lei et al., 2025 ; Chen et al., 2024 ) | ✗ | ✗ | ✗ | T, D | ✗ | ✗ |
| Enterprise simulators | ||||||
| ( Flynt, 2026 ; Zeng et al., 2026 ) | ✗ | ✗ | ✓ | T, D | ✓ | ✗ |
| Enterprise sandboxes | ||||||
| ( Huang et al., 2025 ; Xu et al., 2026 ; Gruenbaum et al., 2026 ) | ✗ | T, D |
| Dimension | Effect on the data estate and example event |
|---|---|
| Business glossary | Domain terms or codes require definitions outside the schema, e.g., PublishGlossaryPolicyEvent defines POS status code 4 as a voided order. |
| Implicit business rules | Documents state rules that the schema does not enforce, e.g., ReviseDiscountPolicy writes a memo excluding staff-discounted sales from revenue. |
| Multi-granularity | Transaction rows coexist with coarser summaries, e.g., EstablishDailyRollup creates daily register summaries. |
| Table redundancy | Frozen snapshots and redundant copies remain beside authoritative tables, e.g., AnalystAuditSnapshot retains an audit snapshot. |
| Schema drift | Schema changes leave historical rows without new attributes, e.g., UpgradeMenuEvent freezes the snapshot and adds milk_choice to the authoritative table. |
| Primitive | Data | Operation |
|---|---|---|
| Generative | ||
| CreateSchema | Generates a table schema from a business description and registers it in . | |
| GenDocument | Generates a document, inserts it into , and registers it and its glossary terms in . | |
| CreateDistribution | – | Defines a probability distribution over a specified domain. |
| GenValue | – | Generates a value consistent with the supplied description. |
| HoldConversation | Generates a multi-turn stakeholder dialogue and records the decision outcome. | |
| Retail-10 | Retail-100 | Retail-1000 | Music store | Banking | Logistics | Insurance | |
| Complexity | SD | SD | All | IBR | BG | TR, MG | IBR, MG, SD |
| Horizon (days) | 14 | 14 | 28 | 60 | 120 | 30 | 120 |
| Event types (R D) | |||||||
| Operators | 59 | 299 | 1.3K | 90 | 91 | 86 | 92 |
| Event instances | 65 | 153 | 834 | 147 | 152 | 177 | 585 |
| Operator invocations | 280 | 1.7K | 15.7K | 545 | 3.2K | 1.2K | 4.5K |
| Development | Held-out | ||||||
| Error type | Retail-10 | Banking | Logistics | Retail-100 | Retail-1000 | Music store | Insurance |
| Compiled SP programs | |||||||
| Cross-column/row dependency violation (dependencies) | 3 / 105 (0.03) | 8 / 104 (0.08) | 4 / 129 (0.03) | 18 / 253 (0.07) | 186 / 1.1K (0.17) | 1 / 75 (0.01) | 14 / 109 (0.13) |
| Missing required output (required outputs) | 0 / 33 | 0 / 32 | 0 / 53 | 1 / 236 (0.00) | 25 / 1.5K (0.02) | 0 / 23 | 3 / 52 (0.06) |
| Missing stochastic sampling (value-producing operators) | 2 / 47 (0.04) | 2 / 85 (0.02) | 2 / 74 (0.03) | 3 / 257 (0.01) | 19 / 1.0K (0.02) | 0 / 84 | 0 / 83 |
| Unbound variable (variables) | 0 / 60 | 0 / 91 | 0 / 86 | 0 / 300 | 0 / 1.3K | 0 / 90 | 0 / 93 |
| BIRD financial | SynthCorp Banking | |
|---|---|---|
| Data | Static snapshot of a Czech bank (8 tables) | 120 simulated days (9 base tables, 12 dated snapshots, 4 views) |
| History | None | Simulation log of 152 event instances |
| Knowledge | Column descriptions and per-question external-knowledge evidence | 7 event-generated documents, no column descriptions |
| Catalog | – | 144 glossary terms linked to source documents |
| Questions | Human-written | Generated from benchmark hints and detectors |
| Gold program | Human-written | Log-derived and debate-verified |
| Retail-10 | Retail-100 | Retail-1000 | Music store | Banking | Logistics | Insurance | All | |
| #Tasks | 37 | 31 | 62 | 28 | 26 | 19 | 53 | 256 |
| Gemma 4 26B-A4B | ||||||||
| S1 (Schema only) | 27.9 7.1 | 64.5 8.0 | 22.0 4.9 | 16.7 7.0 | 37.2 9.5 | 66.7 9.9 | 27.0 6.0 | 33.3 2.8 |
| S2 (Schema + top-1 document) | 43.2 7.7 | 67.7 8.2 | 26.9 5.3 | 28.6 8.7 | 44.9 9.2 | 47.4 11.8 | 49.1 6.6 | 42.3 3.0 |
| S3 (Agent + search tools) | 63.1 7.7 | 64.5 8.0 | 34.9 5.5 | 63.1 8.1 | 64.1 7.6 | 64.9 9.7 | 52.8 6.3 | 54.6 2.8 |
| S4 (Agent + files and CLI) | 59.5 7.7 | 60.2 7.9 | 23.7 4.7 | 50.0 7.4 | 56.4 7.6 | 59.6 10.7 | 47.8 6.3 | 47.1 2.8 |
| Banking vs. BIRD financial | BIRD | SynthCorp |
|---|---|---|
| Weekend share of transactions | 0.291 | 0.283 |
| Share of round-hundred amounts | 0.247 | 0.227 |
| Transactions per account per 30 days (median) | 5.4 | 6.6 |
| Top-20% accounts’ share of transactions | 0.37 | 0.29 |
| Amount tail, p99/p50 | 21.6 | 15.8 |
| Transaction code mix ( type / operation / k_symbol ), TVD | – | 0.44 / 0.45 / 0.37 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Retail-10 | Retail-100 | Retail-1000 | Music store | Banking | Logistics | Insurance | |
| Total tables | 33 | 272 | 1.4K | 11 | 25 | 60 | 126 |
| report snapshots / views | 14/8 | 105/66 | 347/41 | 0/0 | 12/4 | 34/4 | 102/6 |
| base tables | 11 | 101 | 1.0K | 11 | 9 | 22 | 18 |
| reference | 5 | 35 | 264 | 9 | 7 | 4 | 8 |
| transaction | 4 | 46 | 389 | 2 | 1 | 10 | 8 |
| log | 0 | 5 | 76 | 0 | 0 | 3 | 0 |
| Component | Error type | Counting unit | Detection criterion and example |
| Compiled SP programs | Cross-column/row dependency violation | Dependencies | The program relaxes or violates a specified cross-table, cross-row, or intra-row dependency, e.g., sampling correlated columns independently or replacing a row formula with an aggregate. |
| Missing required output | Required outputs | The program omits the operations that produce a required table, column, document, status transition, or intra-day timing window. | |
| Missing stochastic sampling | Value-producing operators | The program omits a required distribution draw or replaces it with a constant, e.g., a fixed customer count where the specification requires Poisson arrivals. | |
| Unbound variable | Variables | No execution record binds the variable, so downstream operators read an empty value. | |
| Unintended output overwrite | Operators | The program of a recurring event type writes a fixed output name, so a later event instance overwrites a table, view, or document that the specification preserves. | |
| Foreign key violation | Tables | After execution, rows of the table reference keys absent from the parent table, e.g., an order references a nonexistent customer_id . |
| Metric | Unit | Real reference | Retail-10 | Retail-100 | Retail-1000 | Music store | Banking | Logistics | Insurance |
|---|---|---|---|---|---|---|---|---|---|
| Banking vs. BIRD financial | |||||||||
| Weekday mix, TVD | transactions | – | 0.029 | ||||||
| Amount log-shape, KS / | transactions | – | 0.236 / 0.401 | ||||||
| Amount median (CZK) | transactions | 2,100 | 934 | ||||||
| Loan amount log-shape, KS / | loans | – | 0.212 / 0.122 | ||||||
| Loan duration / card type / statement frequency mix, TVD | rows | – | 0.104 / 0.128 / 0.176 | ||||||
| Retail-10 | Retail-100 | Retail-1000 | Music store | Banking | Logistics | Insurance | |
|---|---|---|---|---|---|---|---|
| Doc-lookup / numeric | 5/13 | 8/10 | 6/12 | 0/10 | 1/6 | 1/6 | 4/21 |
| Requested dim. / complex tasks | 3/29 | 2/21 | 47/48 | 11/23 | 15/20 | 8/14 | 33/43 |
| Distinct skeletons / gold | 34/37 | 30/31 | 57/62 | 27/28 | 24/26 | 19/19 | 44/53 |
| Join | 70% | 48% | 48% | 64% | 69% | 58% | 57% |
| Grouping | 70% | 87% | 84% | 64% | 92% | 84% | 85% |
| Nesting | 54% | 55% | 56% | 39% | 58% | 74% | 66% |