Operating an Agent-Driven Dataset Factory: Discovery, Construction, and Public Release
Organizations: Central European University · Epoch
Abstract
Turning public data into prediction datasets requires connecting source discovery, task definition, construction, and distribution. We deployed Dataset Factory to perform these steps continuously under a standing release policy. Bounded language-model tasks interpret sources, write construction code, and review task meaning; deterministic software manages state, workers, retries, and publication. Over 81.53 hours, the factory published 14 datasets at an estimated recurring operating expenditure of about $2.74 per publication. The accounting includes failed attempts, inference, workers, and hosting. Separate agents completed prediction experiments on all twelve evaluated releases. Source review nevertheless identified a task whose construction filters contradicted its population declaration. The original reviewer accepted later-record exclusions without reconciling them with the published selection rule. This experience documents an operating source-to-release workflow and shows why successful execution and consumer use need separate checks of task meaning.
Figures & tables
| Task (domain) | Principal review finding | Release disclosure |
|---|---|---|
| Trial within 36 months (courts) | Condition: post-filing jury information; retrospective use. | Stated |
| Reliability deterioration (electricity) | Condition: lagged outcomes require restricted scoring access. | Access stated |
| Penalty reduction (pipelines) | Concern: original-notice timing is unverified. | Partly stated |
| Five-year recall (medical devices) | Concern: undated recalls can become negatives; incidence unverified. | Not found |
| Route survival (aviation) | Condition: carrier-code changes can count as entries or exits. | Exits stated; entries not found |
| Phase I to II award (research grants) | Contradiction: selection declaration; concern: award-amount timing. | Selection: conflicting statements; timing: partly stated |
| Stage | Done | Retry | Repair | Defer | Reject | Open | Total |
|---|---|---|---|---|---|---|---|
| Discover | 54 | 101 | 0 | 30 | 0 | 0 | 185 |
| Build | 37 | 260 | 0 | 1 | 0 | 1 | 299 |
| Evaluate | 14 | 4 | 21 | 0 | 2 | 0 | 41 |
| Publish | 14 | 1 | 0 | 0 | 0 | 0 | 15 |
| Total | 119 | 366 | 21 | 31 | 2 | 1 | 540 |