TimeTox: An LLM-Based Pipeline for Automated Extraction of Time Toxicity from Clinical Trial Protocols
Authors: Saketh Vinjamuri, Marielle Fis Loperena, Marie C. Spezia, Ramez Kouzy
Organizations: Fairview Hospital, Cleveland Clinic Foundation, Cleveland, OH, USA · The George Washington University School of Medicine and Health Sciences, Washington, DC, USA · University of Missouri-Columbia School of Medicine, Columbia, MO, USA · The University of Texas MD Anderson Cancer Center, Houston, TX, USA
Time toxicity, the cumulative healthcare contact days from clinical trial participation, is an important but labor-intensive metric to extract from protocol documents. We developed TimeTox, an LLM-based pipeline for automated extraction of time toxicity from Schedule of Assessments tables. TimeTox uses Google's Gemini models in three stages: summary extraction from full-length protocol PDFs, time toxicity quantification at six cumulative timepoints for each treatment arm, and multi-run consensus via position-based arm matching. We validated against 20 synthetic schedules (240 comparisons) and assessed reproducibility on 644 real-world oncology protocols. Two architectures were compared: single-pass (vanilla) and two-stage (structure-then-count). The two-stage pipeline achieved 100% clinically acceptable accuracy (±3 days) on synthetic data (MAE 0.81 days) versus 41.5% for vanilla (MAE 9.0 days). However, on real-world protocols, the vanilla pipeline showed superior reproducibility: 95.3% clinically acceptable accuracy (IQR ≤ 3 days) across 3 runs on 644 protocols, with 82.0% perfect stability (IQR = 0). The production pipeline extracted time toxicity for 1,288 treatment arms across multiple disease sites. Extraction stability on real-world data, rather than accuracy on synthetic benchmarks, is the decisive factor for production LLM deployment.
Figures & tables
Figure 1: Representative SoA table from a complex synthetic breast oncology protocol (BRST-2025-01) showing two treatment arms with three visit days per cycle.
Figure 2: Step-by-step pipeline for processing protocol PDFs via the Gemini API to extract relevant schedules and generate a consolidated summary document.
Characteristic
Value
Total protocols collected
649
Protocols with successful summary extraction
644
Summary extraction success rate
99.5%
Summary extraction model
Gemini 2.5 Flash
Average processing time per protocol
2 to 3 min
Average SoA pages identified per protocol
5 to 15
Table 1: Dataset Overview
Characteristic
Value
Total schedules
20
Total arms
40 (20 intervention, 20 control)
Total ground truth comparisons
240
Disease categories
8
Complexity: Simple / Moderate / Complex
5 / 10 / 5
Cycle lengths
7, 21, 28, 35 days
Table 2: Synthetic Schedule Characteristics
Pipeline
Exact Match
Clinically Acceptable Accuracy ( ± 3d)
MAE (days)
Vanilla (single-pass)
0.3%
41.5%
9.0
Two-Stage (structure → count)
29.2%
100.0%
0.81
Table 3: Validation Results on 20 Synthetic Schedules (240 Comparisons)
Time Window
Median Error
Mean Error
MAE
% Overcount
% Undercount
% Exact
Screening
− 1.0
− 0.2
1.6
10.0%
90.0%
0.0%
1 Month
+2.0
+1.9
2.2
77.5%
20.0%
2.5%
3 Months
+5.5
+4.9
5.8
80.0%
20.0%
0.0%
6 Months
+10.0
+9.1
11.3
75.0%
25.0%
0.0%
9 Months
+11.5
+11.9
14.1
75.0%
25.0%
0.0%
12 Months
+11.5
+14.0
16.3
75.0%
25.0%
0.0%
Table 4: Vanilla Pipeline Error Distribution on Synthetic Data by Time Window (240 Comparisons)
Figure 3: Vanilla pipeline signed error distribution by time window on 20 synthetic schedules (240 comparisons). Each dot represents one arm-timepoint comparison. Red: overcount > 3 days; blue: undercount > 3 days; green: within ± 3 days. Solid line: mean error; dashed lines: mean ± 1.96 SD (limits of agreement).
Figure 4: Mean absolute error (days) by schedule complexity and time window for the vanilla pipeline on synthetic data. Errors are lowest at screening and increase with both time window duration and schedule complexity.
McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, Texas, USA · McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA
Princeton University, Princeton, New Jersey, USA · National Library of Medicine, Bethesda, Maryland, USA · Washington University in St. Louis, St. Louis, Missouri, USA
Leiden University Medical Center, The Netherlands · Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands · University of Warwick, UK +2