Agents Propose, Code Verifies, Humans Decide: Verified Agentic Ingestion of Heterogeneous Scientific Tables into Relational Databases
Authors: Bharatha Rankothge, Suzanne Hodgkins, Virginia Rich, Michael Kirby
Organizations: Dept. of Mathematics, Colorado State University, Fort Collins, CO, USA · Dept. of Microbiology, The Ohio State University, Columbus, OH, USA
Public scientific datasets rarely arrive analysis-ready. They arrive as messy folders of wide matrices, long tables, spreadsheets, and per-cohort files whose keys and encodings differ, and every reanalysis must reconnect them first. A coding agent can automate this, but its mistakes can look plausible, and without an audit trail, a mistake that corrupts data can go uncaught. We present an AI-assisted ingestion pipeline built on one rule: LLM agents propose, deterministic code verifies, and humans decide. Narrow agents see compact, citable evidence about the files, never the data, and return typed proposals. A verifier with 39 check types tests each proposal against that evidence in two loops, one for the schema's structure and one for its meaning, and a human reviewer approves the schema and the load. Code compiles PostgreSQL DDL, reshapes the data, quarantines rows it cannot place, records provenance, and reconciles every staged row against the database. On four public biomedical datasets, one held out from development, 9 of 20 first proposals failed a check and every failure was caught: the loops repaired eight, and a one-line reviewer edit fixed the ninth. All 45.1 million values loaded with zero reconciliation gap. Discovering a dataset's structure took 4-7 LLM calls and preparing its data at most one more, costing under $0.50 per dataset. In a separate test, each of nine planted data errors was quarantined with a reason or recorded in an audit log.
Figures & tables
Fig. 1: Pipeline overview: three phases, eight stages, and two human checkpoints. CP1 approves the schema before any database write; CP2 approves the load. Blue chips are LLM agents, which only propose; black and white boxes are code, which verifies or executes every proposal. Stages 3–5 are the two propose–verify loops of Fig. 2 .
Agent
Stage
Proposes
Checked by
Profiler
1
File shape label
Closed set; must cite headers in the card
Architect
3
Conceptual schema
39 verifier checks
Resolver
4
Typed edit list
Kept only if fewer checks fail
Critic
5
Recodes, objections
Must match observed values; re-verified
Interpreter
6
Plain-language summary
Cannot change the DDL
Wrangler
7
Transforms (8 ops), flags
Fixed vocabulary; projected keys must exist in the parent
TABLE I: Agents, what each proposes, and the code that checks it.
Family
Rejects, for example
Grounding
An attribute, measure, or annotation that names a source column that is not a profiled header
Keys
A key with no candidate-key support; a fact key that repeats or has a redundant column; colliding partitioned keys
Joins
A join without inclusion-dependency coverage; a join target that is not the parent key; mismatched key types
Facts
No fact for a measurement file; a melt on a non-wide file or in the wrong orientation; melted cells not matching the grid
Coverage
A column dropped without a reason; a file owned by no entity
Structure
A split that contradicts GRAIN ; a non-constant promotion; a value map that misses observed values
TABLE II: Verifier check families.
Fig. 2: Stages 3–5: two propose–verify loops that share one verifier. Loop 1 repairs the structure; Loop 2 runs only on a proposal that passes every check and confirms its meaning. Numbered badges give the order of Loop 1; recodes that fail their check are not applied and are listed for the reviewer at CP1. Dark blue: strong model; light blue: fast model; black and white: code; orange: human checkpoint.
Event
Handling
Row of the wrong length
Quarantined whole ( ragged_row )
NA token or empty cell
NULL , value_status=missing
Non-numeric measure
Quarantined ( unparseable_value )
Repeated fact grain
Quarantined ( duplicate_grain )
Fact key with no parent
Quarantined ( unmatched_parent_key )
S1 Value fails its type
NULL + audit row ( coerced_to_null )
TABLE III: How the loader handles problem rows and values: each is quarantined, recorded, or flagged.
GSE22203
COVIDome
GSE73072
breast.TCGA
Files (formats)
3 (csv)
5 (csv, xlsx)
9 (txt, csv)
4 (csv)
Measurement shape
1 wide
4 long
1 wide
3 wide
Entities / facts
3 / 1
6 / 4
4 / 1
4 / 3
Checks pass / warn / fail
75/2/0
61/1/0
100/2/0
47/0/0
LLM calls
4+1
7+0
4+1
4+0
Tokens (k)
71.1
82.5
72.1
205.2
TABLE IV: Datasets and results. Upper block: run 1 of each dataset, loaded. LLM calls are Stages 1–6 + Stage 7; tokens include thinking. Lower block: five runs of Stages 1–6 per dataset.
Fig. 3: GSE73072 run 4 through the loops of Fig. 2 ; each badge is the number of failed checks. Subject IDs repeat across the seven cohort files, so a key on the ID alone is not unique. The Resolver’s first patch fixed the key but not the join that references it, and was discarded. After one redraw, a second patch moved a column back from subject to sample because it varies within a subject. The Critic’s recodes of three 0/1 flags to Boolean were checked against the observed values and re-verified.
Planted error
Caught
Outcome
mRNA row one cell short
CP2
Row quarantined, ragged_row
NA , empty, NaN in 3 cells
load
NULL , value_status=missing
high in protein matrix
load
1 value quarantined, unparseable_value
mRNA row duplicated
load
200 values quarantined, duplicate_grain
Sample removed from labels (S3)
load
Key-only sample, stub_parent_created
Sample twice, two labels (S2)
load
First kept, duplicate_pk_conflict
TABLE V: Planted errors, one per copy (CP2: Checkpoint 2).
Fig. 4: Three datasets, three schema shapes. Top: the input files. Bottom: the schema the pipeline built, with row counts (dark headers: entities; slate headers: facts). (a) GSE22203: one melted matrix in a star schema. (b) COVIDome: four already-long tables (6.27M values) loaded as four facts sharing one sample table; proteins are keyed by aptamer and transcripts by Ensembl ID; the fact tables are named with a _fact suffix (e.g., cytokine_fact ), omitted in the figure. (c) GSE73072: a melted matrix whose subjects need a composite key.