TimeTox: An LLM-Based Pipeline for Automated Extraction of Time Toxicity from Clinical Trial Protocols
Authors: Saketh Vinjamuri, Marielle Fis Loperena, Marie C. Spezia, Ramez Kouzy
Organizations: Fairview Hospital, Cleveland Clinic Foundation, Cleveland, OH, USA · The George Washington University School of Medicine and Health Sciences, Washington, DC, USA · University of Missouri-Columbia School of Medicine, Columbia, MO, USA · The University of Texas MD Anderson Cancer Center, Houston, TX, USA
Time toxicity, the cumulative healthcare contact days from clinical trial participation, is an important but labor-intensive metric to extract from protocol documents. We developed TimeTox, an LLM-based pipeline for automated extraction of time toxicity from Schedule of Assessments tables. TimeTox uses Google's Gemini models in three stages: summary extraction from full-length protocol PDFs, time toxicity quantification at six cumulative timepoints for each treatment arm, and multi-run consensus via position-based arm matching. We validated against 20 synthetic schedules (240 comparisons) and assessed reproducibility on 644 real-world oncology protocols. Two architectures were compared: single-pass (vanilla) and two-stage (structure-then-count). The two-stage pipeline achieved 100% clinically acceptable accuracy (±3 days) on synthetic data (MAE 0.81 days) versus 41.5% for vanilla (MAE 9.0 days). However, on real-world protocols, the vanilla pipeline showed superior reproducibility: 95.3% clinically acceptable accuracy (IQR ≤ 3 days) across 3 runs on 644 protocols, with 82.0% perfect stability (IQR = 0). The production pipeline extracted time toxicity for 1,288 treatment arms across multiple disease sites. Extraction stability on real-world data, rather than accuracy on synthetic benchmarks, is the decisive factor for production LLM deployment.
Figures & tables
Figure 1: Representative SoA table from a complex synthetic breast oncology protocol (BRST-2025-01) showing two treatment arms with three visit days per cycle.
Figure 2: Step-by-step pipeline for processing protocol PDFs via the Gemini API to extract relevant schedules and generate a consolidated summary document.
Characteristic
Value
Total protocols collected
649
Protocols with successful summary extraction
644
Summary extraction success rate
99.5%
Summary extraction model
Gemini 2.5 Flash
Average processing time per protocol
2 to 3 min
Average SoA pages identified per protocol
5 to 15
Table 1: Dataset Overview
Characteristic
Value
Total schedules
20
Total arms
40 (20 intervention, 20 control)
Total ground truth comparisons
240
Disease categories
8
Complexity: Simple / Moderate / Complex
5 / 10 / 5
Cycle lengths
7, 21, 28, 35 days
Table 2: Synthetic Schedule Characteristics
Pipeline
Exact Match
Clinically Acceptable Accuracy ( ± 3d)
MAE (days)
Vanilla (single-pass)
0.3%
41.5%
9.0
Two-Stage (structure → count)
29.2%
100.0%
0.81
Table 3: Validation Results on 20 Synthetic Schedules (240 Comparisons)
Time Window
Median Error
Mean Error
MAE
% Overcount
% Undercount
% Exact
Screening
− 1.0
− 0.2
1.6
10.0%
90.0%
0.0%
1 Month
+2.0
+1.9
2.2
77.5%
20.0%
2.5%
3 Months
+5.5
+4.9
5.8
80.0%
20.0%
0.0%
6 Months
+10.0
+9.1
11.3
75.0%
25.0%
0.0%
9 Months
+11.5
+11.9
14.1
75.0%
25.0%
0.0%
12 Months
+11.5
+14.0
16.3
75.0%
25.0%
0.0%
Table 4: Vanilla Pipeline Error Distribution on Synthetic Data by Time Window (240 Comparisons)
Figure 3: Vanilla pipeline signed error distribution by time window on 20 synthetic schedules (240 comparisons). Each dot represents one arm-timepoint comparison. Red: overcount > 3 days; blue: undercount > 3 days; green: within ± 3 days. Solid line: mean error; dashed lines: mean ± 1.96 SD (limits of agreement).
Figure 4: Mean absolute error (days) by schedule complexity and time window for the vanilla pipeline on synthetic data. Errors are lowest at screening and increase with both time window duration and schedule complexity.
The reliance on unstructured free text for documenting clinical trial protocols creates a significant barrier to automated reasoning, cohort discovery, and trial simulation. The lack of formal structure obscures critical temporal phenotypes, such as dynamic eligibility criteria and event timing constraints. Although Temporal Ensemble Logic (TEL) offers an expressive framework for modeling these elements, manual encoding remains a prohibitive bottleneck. We introduce the CT-TEL workflow: a scalable pipeline leveraging Large Language Models (LLMs) to translate narrative clinical protocols into TEL formulas. We applied CT-TEL to generate logical models for 23 real-world trials from ClinicalTrials.gov. We evaluated translation fidelity via a back-translation approach, using LLMs to convert TEL formulas back into natural language and measuring semantic similarity against source texts. The resulting semantic retention suggests that LLMs may offer a pathway for mapping informal protocols to computable logic, providing preliminary evidence toward scalable clinical trial emulation within the emerging "Symbolic Biomedicine" paradigm championed by the corresponding author.
Yan Huang, Xubing Hao, Xiaojin Li +4
McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, Texas, USA · McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA
Existing pipelines for clinical timeline extraction from case reports are evaluated using an expert reference and are limited by imperfect reference annotations and imprecise event alignment. We developed GAVEL, an LLM judge protocol that compares two timelines with the case report and returns a discrepancy type, verdict, and report passage for each difference. We evaluated the event matcher, reviewed 2,738 findings from GPT5.6sol and DeepSeek V3.2, ranked six LLM extractors and two human annotators, and tested GAVEL guided merging. True match rates were 60% immediately below and 48% immediately above the 0.10 cutoff. Manual review confirmed 89.4% and 88.6% of findings. Across 126 reports, merged timelines were preferred in 77.0% of comparisons (95% CI, 69.8 to 84.1%) and reduced discrepancies attributed to the evaluated timeline from 7.63 to 0.85 per report. GAVEL supports report-based comparison and revision without treating either timeline as ground truth.
Jack Cummins, Sayantan Kumar, Ketan Tamirisa +1
Princeton University, Princeton, New Jersey, USA · National Library of Medicine, Bethesda, Maryland, USA · Washington University in St. Louis, St. Louis, Missouri, USA
This study presents OpenExtract, an open-source pipeline for automated data extraction in large-scale systematic literature reviews. The pipeline queries large language models (LLMs) to predict data entries based on relevant sections of scientific articles. To test the efficacy of OpenExtract, we apply it to a systematic literature review in digital health and compare its outputs with those of human researchers. OpenExtract achieves precision and recall scores of > 0.8 in this task, indicating that it can be effective at extracting data automatically and efficiently. OpenExtract: https://github.com/JimAchterbergLUMC/OpenExtract.
Jim Achterberg, Bram Van Dijk, Jing Meng +8
Leiden University Medical Center, The Netherlands · Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands · University of Warwick, UK +2