cs.AINov 2, 2026

Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets

Authors: Javad Mohammad Alizadeh, Huanmei Wu

Organizations: College of Public Health, Temple University, Philadelphia, PA, USA

Abstract

LLM agents are increasingly built into data-analysis software, yet little evidence shows how reliably they preprocess clinical data or whether the tools they invoke give correct, reproducible output. We report a benchmark of THReD, a web-based system that turns electronic health record (EHR) extracts into AI-ready datasets through a chat agent whose proposed steps run only after user confirmation. The agent benchmark has 141 tasks (100 templates) in seven categories, including goal-level requests, failure recovery, requests calling for no action, schema variation and prompt injection through the data, on synthetic tables calibrated to a regional health information exchange; success is judged by checks on the produced data. Over 2,550 episodes with Claude Haiku 5.5 in four context configurations, the deployed agent completed 92.7% (95% CI 88.1-96.4) of tasks on its first answer, abstained on every advice request, recovered from every injected fault, and never proposed an action requested by planted text. Without the data profile, completion fell to 58.0%; goal-level requests were hardest (80.6%). Six experiments on real records of two chronic-disease cohorts and on synthetic data at scale validate the tools: 91-92% of inferred patient classes and 99.7-99.8% of inferred vital-sign labels were correct in masked reconstruction; a dictionary builder combining curated dictionaries with LLM suggestions mapped 99.0% of 2,208 raw variants correctly; all 2,176 replays of 128 saved pipelines gave byte-identical files under pinned dependencies; subgroup differences in inference accuracy vanished within facility and lookup key; and the same-day consolidation rule changed 90-day emergency revisit rates by up to 3.2 points without shifting cohort composition by more than one point. Processing time grew roughly linearly to 5.5 million records while the prompt stayed at about 13 thousand tokens.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

    May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10AI Agent BenchmarksElectronic Health Records

  2. A Living Benchmark for Information Retrieval from Electronic Health Records

    Sep 24, 2026Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani +23Synthetic Benchmark GenerationClinical QA

  3. EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs

    May 28, 2026Yuzhang Xie, Keqi Han, Yunpeng Xiao +7HealthcareClinical Decision-Making