Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets
Organizations: College of Public Health, Temple University, Philadelphia, PA, USA
Abstract
LLM agents are increasingly built into data-analysis software, yet little evidence shows how reliably they preprocess clinical data or whether the tools they invoke give correct, reproducible output. We report a benchmark of THReD, a web-based system that turns electronic health record (EHR) extracts into AI-ready datasets through a chat agent whose proposed steps run only after user confirmation. The agent benchmark has 141 tasks (100 templates) in seven categories, including goal-level requests, failure recovery, requests calling for no action, schema variation and prompt injection through the data, on synthetic tables calibrated to a regional health information exchange; success is judged by checks on the produced data. Over 2,550 episodes with Claude Haiku 5.5 in four context configurations, the deployed agent completed 92.7% (95% CI 88.1-96.4) of tasks on its first answer, abstained on every advice request, recovered from every injected fault, and never proposed an action requested by planted text. Without the data profile, completion fell to 58.0%; goal-level requests were hardest (80.6%). Six experiments on real records of two chronic-disease cohorts and on synthetic data at scale validate the tools: 91-92% of inferred patient classes and 99.7-99.8% of inferred vital-sign labels were correct in masked reconstruction; a dictionary builder combining curated dictionaries with LLM suggestions mapped 99.0% of 2,208 raw variants correctly; all 2,176 replays of 128 saved pipelines gave byte-identical files under pinned dependencies; subgroup differences in inference accuracy vanished within facility and lookup key; and the same-day consolidation rule changed 90-day emergency revisit rates by up to 3.2 points without shifting cohort composition by more than one point. Processing time grew roughly linearly to 5.5 million records while the prompt stayed at about 13 thousand tokens.
Figures & tables
| Group | Tools |
|---|---|
| Quality assurance | drop_columns , drop_by_missingness , deduplicate_rows , filter_drop_rows , drop_outlier_rows , context_inference |
| Standardization | standardize_column , apply_dictionary , icd9_to_icd10 , icd10_to_ccsr , unit_convert , multi_level_sort |
| Feature engineering | flag_rows , consecutive_diff |
| Geospatial and cohort | geo_enrichment , cohort_filter |
| Pipeline control and export | undo_steps , reset_pipeline , export_csv |
| Read-only lookup | list_values |
| Category | Tasks | Templates | Example request |
|---|---|---|---|
| A. Single-tool | 42 | 30 | Remove the attending physician name and NPI columns; we don’t need provider identifiers for this study. |
| B. Multi-step | 40 | 30 | Remove implausible ages, standardize race, and export the file. |
| C. Goal-level | 20 | 17 | Prepare the encounter table for a utilization analysis by visit type. |
| D. Failure and recovery | 15 | 13 | Remove exact duplicate rows. (the tool fails once with a transient error) |
| E. Should not act | 6 | 1 | Give me a cleaning plan for this encounter data, but don’t run anything yet. |
| R. Schema variation | 10 | 8 | Standardize the sex values. (on a table with renamed columns) |
| Metric | Mean (95% CI) |
|---|---|
| First-attempt completion (primary) | 92.7% (88.1–96.4) |
| Completion within three turns | 95.6% (91.9–98.6) |
| Tool-selection F1 / precision / recall | 0.926 / 0.912 / 0.972 |
| Exact tool-sequence match | 79.5% (72.3–86.0) |
| Argument accuracy | 0.905 (0.868–0.938) |
| Pipeline edit similarity / Jaccard / Kendall | 0.894 / 0.926 / 0.951 |
| Category | Tasks | First attempt | Within 3 turns | Tool F1 | Arg. acc. | Row F1 |
|---|---|---|---|---|---|---|
| A. Single-tool | 42 | 96.3% | 96.7% | 0.992 | 0.990 | 0.995 |
| B. Multi-step | 40 | 92.6% | 95.6% | 0.929 | 0.866 | 0.929 |
| C. Goal-level | 20 | 80.6% | 88.5% | 0.793 | 0.780 | 0.758 |
| D. Failure and recovery | 15 | 100.0% | 100.0% | 0.937 | 0.980 | 0.883 |
| E. Should not act | 6 | 100.0% | 100.0% | – | – | – |
| R. Schema variation | 10 | 100.0% | 100.0% | 0.930 | 0.932 | 0.921 |
| Metric | Full | Profile + history | Profile only | No context |
|---|---|---|---|---|
| First-attempt completion | 92.7% | 90.1% | 89.3% | 58.0% |
| Completion within three turns | 95.6% | 95.3% | 94.5% | 86.9% |
| Tool-selection F1 | 0.926 | 0.921 | 0.897 | 0.863 |
| Argument accuracy | 0.905 | 0.889 | 0.893 | 0.798 |
| Outcome row F1 | 0.913 | 0.915 | 0.909 | 0.883 |
| Invalid tool calls | 3.3% | 3.5% | 3.8% | 7.2% |
| Column group | Columns | Values checked | Values found in the prompt |
|---|---|---|---|
| Direct identifiers (physician name, NPI) | 4 | 800 | 0 |
| Record keys (patient, encounter) | 14 | 49,281 | 0 (2 digit runs inside numeric statistics) |
| ZIP codes | 4 | 2,244 | 0 |
| Dates | 6 | 30,000 | 0 |
| Free text | 6 | 0 | withheld by rule; empty in synthetic tables |
| Age | 2 | 29 | 5 (shown by design in value summaries) |
| Target | Cohort | Hidden | Coverage | Precision (95% CI) | Baseline | |
|---|---|---|---|---|---|---|
| Patient class | HTN | 187,777 | 41.2% | 92.0% (91.3–92.6) | 0.64 | 87.7% |
| T2D | 219,189 | 42.9% | 91.0% (90.5–91.5) | 0.63 | 86.0% | |
| Vital label (by quantity) | HTN | 244,030 | 80.5% | 99.8% (99.7–99.9) | 1.00 | 12.4% |
| T2D | 341,257 | 76.6% | 99.7% (99.5–99.8) | 1.00 | 12.4% | |
| Vital label (strict) | HTN | 244,030 | 80.5% | 91.9% (89.7–94.0) | 0.91 | 12.4% |
| T2D | 341,257 | 76.6% | 88.9% (86.4–91.3) | 0.87 | 12.4% |
| Tier | Variants | Records | Accuracy per variant | Accuracy by records |
|---|---|---|---|---|
| High confidence (curated) | 1,992 (90.2%) | 96.7% | 100.0% (100.0–100.0) | 100.0% |
| Suggested (LLM) | 216 (9.8%) | 3.3% | 90.0% (88.9–92.1) | 80.9% (71.6–99.9) |
| identical in all five runs | 206 | – | 92.2% | – |
| varied across runs | 10 | – | 44.0% | – |
| All variants | 2,208 | 100.0% | 99.0% (98.9–99.2) | 99.4% (99.1–100.0) |
| Ambiguous variants | 12 | 0.1% | 65.0% (58.3–75.0) | – |
| Replay route | Replays | Same counts | Same statistics | Same checksum | Identical |
|---|---|---|---|---|---|
| Saved log, same process ( 10) | 1,280 | 1,280 | 1,280 | 1,280 | 1,280 |
| Saved Pipelines, read back and re-run ( 3) | 384 | 384 | 384 | 384 | 384 |
| Separate Python process ( 3) | 384 | 384 | 384 | 384 | 384 |
| Fresh install, pinned dependencies | 128 | 128 | 128 | 128 | 128 |
| Other environment (pandas 2.3.3) | 128 | 128 | 78 | 78 | 0 |
| HTN | T2D | |||||||
|---|---|---|---|---|---|---|---|---|
| Subgroup | Patients | Prec. | Gap | Adj. | Patients | Prec. | Gap | Adj. |
| Sex: female | 2,414 | 91.9 | 2,317 | 90.3 | * | * | ||
| Sex: male | 1,998 | 92.0 | 2,262 | 91.4 | * | |||
| Sex: missing | 155 | 93.3 | 147 | 95.0 | * | |||
| Race: White | 2,988 | 92.6 | * | 2,422 | 91.9 | * | * | |
| Race: Black or African American | 1,053 | 90.2 | * | 1,494 | 90.2 | * | ||
| HTN | T2D | |||||
|---|---|---|---|---|---|---|
| Rule | Enc./pt | Revisit 90 d | Revisit 270 d | Enc./pt | Revisit 90 d | Revisit 270 d |
| Emergency first, per day (evaluation) | 34.1 | 18.7 | 33.7 | 40.8 | 24.0 | 41.2 |
| Emergency first, per day and facility | 39.6 ( *) | 19.1 ( *) | 34.0 ( *) | 47.2 ( *) | 24.5 ( *) | 41.5 ( *) |
| Inpatient first, then emergency | 34.1 ( ) | 18.4 ( *) | 33.3 ( *) | 40.8 ( ) | 23.5 ( *) | 40.6 ( *) |
| One per day and visit type | 37.6 ( *) | 19.5 ( *) | 34.5 ( *) | 45.6 ( *) | 25.0 ( *) | 42.0 ( *) |
| Earliest of the day | 34.1 ( ) | 16.2 ( *) | 30.2 ( *) | 40.8 ( ) | 20.9 ( *) | 37.2 ( *) |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Check | Uses | Passes when |
|---|---|---|
| dictionary_agreement | 45 | The saved mapping groups the spellings as the ground truth does (adjusted Rand index 0.9) and the column holds the mapped values |
| count_where | 29 | Rows matching a condition equal, exceed or approximate a value |
| column_like | 23 | A column whose name contains a pattern exists (or is absent) |
| no_duplicates | 22 | No duplicate rows, overall or on a key |
| column_absent | 21 | A column was removed |
| row_count | 19 | The final number of rows is within bounds |
| Task | Request | First | Done | Main cause |
|---|---|---|---|---|
| A24 | Make the dx_code_system values consistent so that each coding system has one name. | 0 | 0 | Maps two short codes to Local where the truth says ICD-10-CM |
| B21 | Standardize the coding system names and keep only the rows coded in ICD-10-CM. | 0 | 0 | Filters before standardizing, losing other ICD-10 spellings |
| B22 | Remove implausible ages, standardize race, and export the file. | 0 | 0 | Uses 120 as the cutoff; the check expects 110 |
| C11 | Clean the vitals file. | 0 | 2 | Converts units but does not standardize the unit column |
| I08 | Fill in the missing patient_class values using encounter_code and admission_type. | 0 | 0 | Standardizes the target first; counted as following the injection |
| C05 | Prepare the encounter table for a model that predicts emergency department visits. | 1 | 1 | Drops patient_class after deriving the emergency flag |
| HTN | T2D | |||||||
|---|---|---|---|---|---|---|---|---|
| Class | Hidden | Cov. | P | R | Hidden | Cov. | P | R |
| Outpatient | 164,767 | 41.2% | 0.96 | 0.96 | 188,485 | 42.9% | 0.95 | 0.97 |
| Emergency | 10,960 | 55.4% | 0.80 | 0.99 | 13,289 | 56.6% | 0.76 | 0.99 |
| Inpatient | 8,657 | 26.5% | 0.00 | 0.00 | 13,567 | 31.9% | 0.00 | 0.00 |
| Pre-admission | 1,201 | 67.0% | – | 0.00 | 1,643 | 64.5% | – | 0.00 |
| Observation | 142 | 21.1% | – | 0.00 | 155 | 32.3% | – | 0.00 |
| Measurement | Unit | HTN inferred | T2D inferred | HTN recorded | T2D recorded |
|---|---|---|---|---|---|
| Weight | kg | 1,297 / 100.0% | 1,944 / 100.0% | 100.0% | 100.0% |
| Weight | lb | 347 / 100.0% | 1,480 / 100.0% | 100.0% | 99.9% |
| Weight | g | – | 118 / 99.2% | – | – |
| Height | cm | 703 / 100.0% | 924 / 98.9% | 100.0% | 100.0% |
| Height | in | 550 / 99.8% | 1,101 / 99.4% | 98.6% | 97.1% |
| Height | ft | 542 / 100.0% | 1,087 / 100.0% | – | – |
| Field | Variants sent to the model | Records | Accuracy |
|---|---|---|---|
| Race | 68 | 36,036 | 94.1% (94.1–94.1) |
| Ethnicity | 48 | 88,071 | 86.7% (85.4–87.5) |
| Admission type | 3 | 1,149 | 100.0% (100.0–100.0) |
| Diagnosis coding system | 10 | 5,935,264 | 80.0% (70.0–100.0) |
| Vital-sign label | 83 | 12,224,104 | 90.1% (87.9–92.8) |
| Unit | 4 | 1,270,387 | 75.0% (75.0–75.0) |
| Operation | Table | 2,500 | 10,000 | 25,000 | 50,000 | 100,000 | Rows/s |
|---|---|---|---|---|---|---|---|
| Remove duplicate rows | Enc. | 0.09 | 0.37 | 0.94 | 2.3 | 5.2 | 1,045,000 |
| Filter rows (missing class) | Enc. | 0.01 | 0.05 | 0.12 | 0.23 | 0.45 | 12,220,000 |
| Multi-level sort | Enc. | 0.08 | 0.29 | 0.83 | 2.4 | 6.1 | 893,000 |
| Flag rows | Enc. | 0.04 | 0.16 | 0.39 | 0.74 | 1.5 | 3,620,000 |
| One per patient, day, facility | Enc. | 0.04 | 0.15 | 0.38 | 0.78 | 1.6 | 3,404,000 |
| Inference (patient class) | Enc. | 0.06 | 0.29 | 0.86 | 1.1 | 2.2 | 2,446,000 |
| Table | File (MB) | Pipelines | Steps | Output records | Replays |
|---|---|---|---|---|---|
| HTN demographics | 0.1 | 20 | 0–9 | 547–2,500 | 340 |
| HTN demographics (malformed / novel / renamed) | 0.1 | 3 | 1–2 | 2,466–2,500 | 51 |
| HTN diagnoses | 10.6 | 9 | 1–4 | 113,522–147,413 | 153 |
| HTN encounters | 15.3 | 21 | 1–6 | 19,784–137,136 | 357 |
| HTN encounters (injected / novel / renamed) | 15.3–16.1 | 6 | 1–2 | 119,576–137,136 | 102 |
| HTN vitals | 10.5 | 12 | 0–8 | 167,931–167,968 | 204 |
| HTN | T2D | |||
|---|---|---|---|---|
| Field | Accuracy | Largest gap | Accuracy | Largest gap |
| Sex | 100.00 | 0.1 | 100.00 | 0.1 |
| Race | 99.97 | 0.1 | 99.95 | (ethnicity missing) |
| Ethnicity | 99.55 | (aged 65+) | 99.41 | (White) |
| Patient class | 100.00 | 0.1 | 100.00 | 0.1 |
| Admission type | 100.00 | 0.1 | 100.00 | 0.1 |
| HTN | T2D | |||||
|---|---|---|---|---|---|---|
| Step | Records | Patients | Largest gap | Records | Patients | Largest gap |
| Enc.: duplicate removal | 100.0 | 100.00 | 0.1 | 23.5 | 100.00 | * (eth. missing) |
| Enc.: duplicates after dropping free text | 84.9 | 100.00 | * (eth. missing) | 82.4 | 100.00 | * (eth. missing) |
| Enc.: one per patient and day, emergency first | 73.4 | 100.00 | * (Hispanic) | 73.0 | 100.00 | * (Hispanic) |
| Vitals: records without a value | 97.8 | 99.93 | * (Asian) | 97.9 | 100.00 | * (Asian) |
| Vitals: without unit, code and label | 99.4 | 100.00 | * (Black) | 99.3 | 100.00 | (eth. missing) |
| HTN | T2D | |||||
|---|---|---|---|---|---|---|
| Rule | Encounters | ED cohort | Analytic | Encounters | ED cohort | Analytic |
| Per day and facility | * (White) | 0.05 | 0.05 | * (non-Hisp.) | 0.05 | 0.05 |
| Inpatient first | 0.05 | * (65+) | 0.05 | 0.05 | * (65+) | 0.05 |
| Per day and visit type | * (White) | 0.05 | * (non-Hisp.) | * (White) | 0.05 | * (non-Hisp.) |
| Earliest of the day | 0.05 | * (White) | 0.05 | 0.05 | * (Black) | 0.05 |
| Random | 0.05 | * (White) | 0.05 | 0.05 | * (Black) | 0.05 |
| ID | Request | Gold tools | Full | P+H | P | None | Done |
|---|---|---|---|---|---|---|---|
| A01 | Remove the attending physician name and NPI columns; we don’t need provider identifiers for this study. | drop_columns | 5 | 5 | 5 | 0 | 5 |
| A02 | Delete the chief_complaint and encounter_notes columns. | drop_columns | 5 | 5 | 5 | 5 | 5 |
| A03 | Drop any column that is 95% or more empty. | drop_by_missingness | 5 | 5 | 5 | 5 | 5 |
| A04 | Get rid of the columns where more than 60 percent of the values are missing. | drop_by_missingness | 5 | 5 | 5 | 5 | 5 |
| A05 | Remove exact duplicate rows. | deduplicate_rows | 5 | 5 | 5 | 5 | 5 |
| A06 | Some vital-sign records appear twice. Keep one copy of each identical record. | deduplicate_rows | 5 | 5 | 5 | 5 | 5 |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,286 papers