The Last Mile of Coding Agents for Long-Document Tasks: An Empirical Study on Clinical Trial Protocol Digitization

Jiacheng Lin; Geoffrey Low; Nishit Agarwal; Rahul Jain; Afrah Shafquat; Mandis Beigi; Jimeng Sun

cs.CLOct 13, 2026

The Last Mile of Coding Agents for Long-Document Tasks: An Empirical Study on Clinical Trial Protocol Digitization

Authors: Jiacheng Lin, Geoffrey Low, Nishit Agarwal, Rahul Jain, Afrah Shafquat, Mandis Beigi, Jimeng Sun

Organizations: University of Illinois Urbana-Champaign · Medidata Solutions

Abstract

Clinical trial protocols serve as the source of truth for study design, but are mostly written and used as unstructured PDF documents. Digitizing protocols turns static content into structured, machine-readable data that clinical software can analyze, validate, and execute. Coding agents, which select tools, execute code, and revise their approach as they work, are increasingly applied to such long-document tasks, yet it remains unclear how reliably they turn complete documents into accurate structured outputs. We study this question using protocol digitization as a testbed. We introduce TrialPD-Bench, an expert-annotated benchmark of 150 publicly available protocol PDFs spanning four task categories. Beyond final-answer performance, our evaluation measures evidence recall from agent traces, agent behavior, and cost. Across nine proprietary and open-weight models in five coding-agent harnesses, even the strongest agents do not fully solve protocol digitization. Our analysis further shows that agents usually locate the required evidence but fail in the last mile of transcribing and structuring it. We further develop TrialScribe, a domain-specific harness that equips coding agents with task skills and a verification loop. TrialScribe improves a locally served open-weight model by 6.2-59.4 points across eight metrics, enabling it to match or surpass proprietary models such as GPT-5.6-Sol.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of this study. (a–c) Coding agents must localize evidence, interpret complex layouts, and link information across long protocols; output quality varies across tasks. (d) Overall score on TrialPD - Bench against average cost per run. (e) TrialScribe with Qwen3.8-27B versus the base model, Opus 5, and GPT-5.6-Sol.

Figure 2: Overview of TrialPD - Bench . (a) Source documents and the four task categories. The corpus comprises 150 unique studies, using one representative document per NCT identifier. (b) Coverage of therapeutic areas. (c) Document-length distribution, with a median of 122 pages marked by the dashed line. (d) Study counts across 106 sponsors.

Task	Samples	Mean	Median	p90	Reference annotations	Main metrics
		Evidence pages
Disease indication	40	1.00	1	1	41 indication descriptions; 1-2 per sample	Indication F1
Eligibility criteria	134	4.32	4	7	3,974 criteria: 1,337 inclusion and 2,637 exclusion	Eligibility criteria F1
Objectives and endpoints	89	3.17	3	5	665 objectives and 1,484 endpoints; 60 objectives without endpoints	Objective, endpoint, and objective-linked endpoint F1
SoA tables	37	1.89	1	4	70 tables; 10 protocols with multiple SoA pages	Activity-Visit and sub/sup F1
* Structured timelines	3	6.33	8	8	14 epochs; 69 encounters; 115 activities	Epoch accuracy; Activity and Encounter recall

Table 1: Task-level sample sizes and annotation statistics on the test split, excluding the development protocols of Appendix H.1 . For every task, the input is the full protocol PDF. Evidence pages is the number of PDF pages per sample that contain the evidence needed for the reference annotations; p90 is the 90th percentile.

Panel of Table 2 — Table 2: Main results (%) of coding agents on TrialPD - Bench . Each task reports its final-answer metrics followed by evidence recall (ER) , the recall of the required evidence pages visited in the agent trace. Obj/End/Link: objective, endpoint, and objective-linked endpoint F1; Sched. and Sub/sup: Activity-Visit F1 (scheduling marks) and Sub/sup F1; Epoch: epoch type positional accuracy; Enc.: encounter recall; Act.: activity recall.

Panel of Table 3 — Table 3: Agent behavior per task: average number of tool execution errors and agent response turns per run. Ind.: indication; Crit.: eligibility criteria; O&E: objectives and endpoints; SoA: SoA tables; TL: structured timeline.

Panel of Table 4 — Table 4: Vision use (%): the share of vision-required cases in which the agent obtains an image or OCR of a relevant hard page. Structured timeline has no vision-required protocol. Task abbreviations as in Table 3 .

Panel of Table 5 — Table 5: Average cost per run in USD. Task abbreviations as in Table 3 .

Figure 3: SoA Activity-Visit F1 on complete protocols (Table 2 ) against model release date. Lines connect models of the same family; hollow markers denote open-weight models.

Figure 4: Failure analysis across protocol digitization tasks. (a) Each pie shows the distribution of failure categories among failing runs within one task; labels indicate shares of at least 8%. (b) Illustrative examples of the five main failure categories, with incorrect or missing predictions highlighted in red.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Input	Size (unit)	Disease	Eligibility	Objectives &	Schedule of	Open
			indication	criteria	endpoints	activities	source
Chia ( Kury et al., 2020 )	Registry text	1,000 trials	✗	✓	✗	✗	✓
LCT ( Dobbins et al., 2022 )	Registry text	1,006 trials	✗	✓	✗	✗	✓
AutoCriteria ( Datta et al., 2024 )	Registry text/XML	180 trials	✗	✓	✗	✗	✗
Kumar et al. ( Kumar et al., 2024 )	Protocol PDFs	91 protocols	✗	✗	✗	$\triangle$	✗
SPIRIT-CONSORT-TM ( Jiang et al., 2025 )	Publication text	200 publications	$\triangle$	$\triangle$	$\triangle$	$\triangle$	✓

Appendix

Table 6: Comparison with prior datasets and systems for clinical trial protocol digitization. ✓ indicates explicit coverage,

\triangle

partial or related-task coverage, and ✗ no reported coverage.

General (all tasks)
Step	Section	Action	Notes
1	Title page	Extract titles	At least the brief title
2	Title page	Extract identifiers	Internal and NCT identifiers
3	Title page	Read the therapeutic area	Preferably codeable
4	Title page	Read the interventions
Indication

Appendix

Table 7: Annotation steps. The general steps were performed for every protocol, before the steps of the task being annotated.

Model	Model identifier	Harness
GPT-4o	gpt-4o	Codex
GPT-5.6-Sol	gpt-5.6-sol	Codex
GPT-6-Astra	gpt-6-astra	Codex
Opus 4.6	claude-opus-4-6	Claude Code
Opus 4.8	claude-opus-4-8	Claude Code
Opus 5	claude-opus-5	Claude Code

Appendix

Table 8: Evaluated models and coding-agent harnesses. GPT-4o is reported only in the appendix.

Figure 5: Page-level SoA extraction, comparing coding agents with OCR and document-understanding baselines using (a) Activity-Visit F1 and (b) Sub/sup F1. Activity-Visit F1 measures the recovery of scheduling associations between activities and visits, usually represented by X marks in SoA table cells. Sub/sup F1 measures the recovery of subscript and superscript footnote markers.

Figure 6: Comparison of coding agents and the RAG baseline across (a) disease indication, (b) eligibility criteria, and (c) objectives and endpoints. Bars show F1 scores (%), and annotations report the difference between the coding agent and RAG for each model in percentage points (

\Delta=\mathrm{Agent}-\mathrm{RAG}

). Positive values favor coding agents.

	Inclusion F1		Exclusion F1
Model	Agent	RAG	Agent	RAG
GPT-4o	9.4	58.6	4.7	64.2
GPT-5.6-Sol	81.8	67.6	90.9	77.1
GPT-6-Astra	81.7	29.8	91.0	34.5
Opus 4.6	81.9	53.8	89.5	62.4
Opus 4.8	82.1	52.1	90.0	60.7

Appendix

Table 9: Eligibility criteria F1 (%) by category for coding agents and RAG.

Agent	Indication F1
	All $n=40$	Text $n=33$	Vision $n=7$
GPT-4o	41.7	44.4	28.6
GPT-5.6-Sol	80.0	84.8	57.1
GPT-6-Astra	82.5	87.9	57.1
Opus 4.6	67.5	69.7	57.1
Opus 4.8	72.5	72.7	71.4

Appendix

Table 10: Study indication. Agent results (%); higher is better. Each metric reports all PDFs, text-sufficient PDFs, and vision-required PDFs.

Agent	Criterion F1
	All $n=134$	Text $n=94$	Vision $n=40$
GPT-4o	6.2	7.7	2.7
GPT-5.6-Sol	87.1	88.2	84.4
GPT-6-Astra	87.8	89.1	84.6
Opus 4.6	86.8	88.5	82.9
Opus 4.8	87.2	88.9	83.0

Appendix

Table 11: Eligibility criteria. Agent results (%); higher is better. Each metric reports all PDFs, text-sufficient PDFs, and vision-required PDFs.

Agent	Objective F1			Endpoint F1			Objective-linked endpoint F1
	All $n=89$	Text $n=61$	Vision $n=28$	All $n=89$	Text $n=61$	Vision $n=28$	All $n=89$	Text $n=61$	Vision $n=28$
GPT-4o	11.4	16.6	0.0	5.6	8.0	0.4	2.7	3.9	0.0
GPT-5.6-Sol	80.4	82.1	76.6	63.6	64.4	61.7	54.7	56.6	50.6
GPT-6-Astra	82.2	84.5	77.0	65.0	67.7	59.2	58.1	61.4	50.8
Opus 4.6	80.6	82.7	75.9	60.6	61.7	58.2	49.6	50.1	48.7
Opus 4.8	82.7	86.2	75.1	66.9	67.1	66.4	58.5	62.2	50.5

Appendix

Table 12: Objectives and endpoints. Agent results (%); higher is better. Each metric reports all PDFs, text-sufficient PDFs, and vision-required PDFs.

Agent	Activity–Visit F1			Sub/sup F1
	All $n=86$	Text $n=61$	Vision $n=25$	All $n=86$	Text $n=61$	Vision $n=25$
GPT-4o	0.5	0.0	1.9	8.1	10.5	2.4
GPT-5.6-Sol	88.2	86.9	91.3	84.0	80.7	92.0
GPT-6-Astra	85.1	83.2	89.8	83.7	80.3	92.2
Opus 4.6	52.4	47.5	64.3	52.9	47.6	65.9
Opus 4.8	86.3	84.7	90.4	81.0	78.9	86.0

Appendix

Table 13: SoA page extraction. Agent results (%); higher is better. Each metric reports all PDFs, text-sufficient PDFs, and vision-required PDFs.

Agent	Activity–Visit F1			Sub/sup F1
	All $n=37$	Text $n=29$	Vision $n=8$	All $n=37$	Text $n=29$	Vision $n=8$
GPT-4o	2.1	2.7	0.0	3.1	2.3	5.9
GPT-5.6-Sol	83.7	84.6	80.4	81.0	79.5	86.3
GPT-6-Astra	86.8	88.0	82.4	84.5	83.7	87.1
Opus 4.6	54.7	56.1	49.6	53.4	54.4	49.6
Opus 4.8	81.4	82.0	79.2	78.0	78.1	77.7

Appendix

Table 14: SoA document extraction. Agent results (%); higher is better. Each metric reports all PDFs, text-sufficient PDFs, and vision-required PDFs.

Agent	Epoch accuracy			Activity recall			Encounter recall
	All $n=3$	Text $n=3$	Vision $n=0$	All $n=3$	Text $n=3$	Vision $n=0$	All $n=3$	Text $n=3$	Vision $n=0$
GPT-4o	0.0	0.0	—	0.0	0.0	—	0.0	0.0	—
GPT-5.6-Sol	61.1	61.1	—	88.1	88.1	—	23.2	23.2	—
GPT-6-Astra	73.3	73.3	—	91.1	91.1	—	20.3	20.3	—
Opus 4.6	61.1	61.1	—	85.3	85.3	—	0.0	0.0	—
Opus 4.8	64.4	64.4	—	82.5	82.5	—	0.0	0.0	—

Appendix

Table 15: Structured timeline. Agent results (%); higher is better. Each metric reports all PDFs, text-sufficient PDFs, and vision-required PDFs.

	Indication F1		Criterion F1		Objective F1		Endpoint F1		Linked endpoint F1
Model	Agent	RAG	Agent	RAG	Agent	RAG	Agent	RAG	Agent	RAG
GPT-4o	41.7	44.5	6.2	61.9	11.4	48.3	5.6	36.4	2.7	20.0
GPT-5.6-Sol	80.0	74.2	87.1	72.8	80.4	70.2	63.6	50.3	54.7	45.0
GPT-6-Astra	82.5	65.4	87.8	31.9	82.2	63.3	65.0	46.9	58.1	41.7
Opus 4.6	67.5	67.5	86.8	58.4	80.6	67.7	60.6	49.5	49.6	37.3
Opus 4.8	72.5	57.5	87.2	56.7	82.7	62.8	66.9	42.1	58.5	33.4

Appendix

Table 16: Coding-agent and RAG results (%) on disease indication, eligibility criteria, and objectives and endpoints, corresponding to Figure 6 . Linked endpoint F1 is the objective-linked endpoint F1 defined in Appendix D.3 ; it is reported here but not plotted.

Coding agents
Method	Activity-Visit F1	Sub/sup F1
GPT-4o	0.5	8.1
GPT-5.6-Sol	88.2	84.0
GPT-6-Astra	85.1	83.7
Opus 4.6	52.4	52.9
Opus 4.8	86.3	81.0

Appendix

Table 17: Page-level SoA extraction results (%) on the 86 single-page inputs, corresponding to Figure 5 a-b.

	Activity-Visit F1
Model	Agent	RAG
GPT-4o	2.1	4.7
GPT-5.6-Sol	83.7	44.0
GPT-6-Astra	86.8	51.5
Opus 4.6	54.7	33.5
Opus 4.8	81.4	19.8

Appendix

Table 18: Document-level SoA extraction results (%) on the 37 protocols. Activity-Visit F1 averages all annotated SoA pages within each protocol, with missing pages scored zero, and then across protocols.

	Epoch accuracy		Activity recall		Encounter recall
Model	Agent	RAG	Agent	RAG	Agent	RAG
GPT-4o	0.0	6.7	0.0	3.7	0.0	0.0
GPT-5.6-Sol	61.1	61.1	88.1	54.9	23.2	40.6
GPT-6-Astra	73.3	67.8	91.1	46.0	20.3	0.0
Opus 4.6	61.1	49.4	85.3	83.3	0.0	0.0
Opus 4.8	64.4	61.1	82.5	80.3	0.0	0.0

Appendix

Table 19: Structured timeline extraction results (%) on the three official example protocols. Encounter recall counts a matched visit as recovered only when its type and contact modes are also correct (Appendix D.5 ).

Figure 7: Failure profiles of ten coding agents, the nine main-text agents and GPT-4o, across six extraction tasks. Each bar shows the distribution of dominant failure categories among failing runs for a model and task, normalized to 100%. These profiles describe failure composition rather than failure rates.

Category	Task	Reference	Prediction
Transcription fidelity	Criteria	Disease duration of more than 10 years in participants with an EDSS $\leq$ 2.0 at screening	Disease duration of more than 10 years in participants with an EDSS $<$ 2.0 at screening
Transcription fidelity	Endpoints	The cumulative change in EDSS scores measured in area under the concentration-time curve (AUC) by Week 96	The cumulative change in EDSS scores, measured in AUC by Week 96
Localization and coverage	Objectives	To evaluate progression-free survival (PFS)	Missing; 9 of the 11 reference objectives are omitted
Relational structure	Objectives	Endpoint “The change in low-contrast visual acuity from baseline to Weeks 48 and 96” under the exploratory objective	The same endpoint attached to a different objective
Domain grounding	Indication	Relapsed/Refractory Hematologic Malignancies	relapsed/refractory Multiple Myeloma (MM)
Execution reliability	Criteria	–	No output file written

Appendix

Table 20: Additional failure examples.

--- name: indication-extraction description: Use when extracting or auditing the treated or prevented indication of a clinical trial protocol, especially when wording, population restrictions or multiple diseases are ambiguous. --- Extract the indication, not the population sentence Write /app/output/indications.json as {"descriptions":["source disease phrase"]} . Use only this protocol. The source helper is python3 /tools/pdf.py : search ’indication|disease|study population’ , pages 1 3 , or render 18 . […] Resolve the source phrase Collect the disease wording from (a) the current full title, (b) an explicit Indication/disease field in the synopsis, and (c) the purpose/primary objective. Do this before deciding. Save the competing source phrases, pages and your selection rationale in /app/output/source_map.json . […] Resolve the three source locations by what each states, not by a fixed title-first or longest-phrase rule. A dedicated Indication field containing a disease noun phrase is the preferred literal span when it agrees with the actual trial purpose. A shorter title must not erase the field’s resectability, stage, resistance or other disease-defining qualifier. A field written as a treatment/population sentence needs parsing: separate the disease from intervention, symptom endpoints, age and enrolment boilerplate. […] Return the disease noun phrase. Remove “treatment of”, “patients/subjects with”, age-group wording and other enrolment boilerplate. Retain disease-defining stage, severity, histology, recurrence/resistance and biomarkers when the indication itself states them. […] Copy the selected phrase verbatim, including its case, hyphenation, spelling and printed abbreviation. Do not expand acronyms or modernize terminology from memory. If OCR appears corrupted, inspect the page and corroborate in another source location. Do not silently change a medical word into another plausible word. Preserve a source’s combined disease phrase when it is presented as one indication for a common diagnostic population. The presence of "or", two names or two abbreviations alone does not establish that separate output entries are required. Look for explicit separate indications, diagnostic cohorts or primary anatomical sites before splitting. […] Audit and repair Independently compare candidate against all three source locations. Check that it preserves the chosen source span, has not dropped disease qualifiers in favor of a shorter title, has not split a combined population without source evidence, is the target disease, is neither an overlong population sentence nor an unsupported short label, retains required qualifiers, and has the right number of distinct conditions. […]

Appendix

Table 21: The TrialScribe skill for indication extraction (abridged).

--- name: criteria-extraction description: Use when extracting or auditing inclusion and exclusion criteria from a full clinical trial protocol, including numbered lists, cross-page continuations, cohorts and laboratory tables. --- Complete criteria with source-preserving text Output /app/output/criteria.json using the original task schema. Do not invent criteria from medical knowledge. /app/pages.json is a one-based physical-page array when addressed as page N -> index N-1. […] Locate, inventory, extract 1. Find the authoritative detailed eligibility section, not just the synopsis or table of contents. Identify inclusion start/end, exclusion start/end, continuation pages, and separate cohort/substudy eligibility sections. […] 2. Save /app/output/source_map.json : section page ranges; printed top-level numbering and counts by category/cohort; any gaps or intentionally omitted duplicate synopsis. Counts must come from the source, not from your output. 3. One output item per source top-level numbered criterion. Subletters, roman numerals, qualifications, exceptions and tables remain inside their parent. Separate genuinely separate numbered criteria even if topics overlap. Preserve additional cohort criteria without duplicating a repeated summary of the same list. 4. Prefer constructing strings from exact source spans in Python instead of retyping/paraphrasing them. Join wrapped lines only within the same paragraph. Strip running headers/footers and top-level list numbers; preserve actual content, units, inequalities, exceptions, references, punctuation and abbreviation spelling. […] 5. Encode paragraphs as , sublists as <ul> / <ol> containing <li>…</li> , and actual tabular criteria as <table> . Escape literal < , > and & in text. Preserve meaningful printed dash/label text inside subpoints; don’t convert a clinical minus sign into a bullet or discard words near numbering. In particular, when the printed subitem begins with a literal dash, retain that dash in the subitem text even inside <li>...</li> . […] 6. Use Inclusion/Exclusion exactly, IN01… / EX01… by category, and save valid JSON. Compare output category counts and list boundaries to the source inventory. Audit the entire list Use source numbering as a checklist. Verify every source top-level item has exactly one output counterpart and vice versa; inspect the final item and cross-page joins. Check missing tails, split subcriteria, omitted tables, negation, decimal thresholds, units and category inversions. For a large list, audit it in bounded page batches; do not inspect only the first few examples and declare the whole list correct. […] An empty list is valid only after locating and checking where eligibility would be stated, including scanned relevant pages. Readability failure means uncertainty, not a supported empty answer. Report concrete missing/incorrect rows with source pages and suggested minimal corrections. […]

Appendix

Table 22: The TrialScribe skill for eligibility criteria extraction (abridged).

--- name: objectives-extraction description: Use when extracting or auditing clinical trial objectives and their attached endpoints, including separate endpoint sections, mixed hierarchy levels and objective-endpoint tables. --- Recover objectives and endpoint attachments Output /app/output/objectives.json in the original task schema. Use only the source. Use python3 /tools/pdf.py search ’objective|endpoint|efficacy variable|outcome|estimand’ to locate sections, pages N M for bounded text, and render N followed by read for tables or damaged text. Do not confuse physical PDF pages with printed page numbers. Build two inventories before linking them Find the detailed objectives and all endpoint/efficacy/safety variable definitions, which can appear in different sections. Compare the synopsis with the detailed definitions; short summaries may merge separately defined endpoints or omit secondary and exploratory endpoints. […] Write /app/output/source_map.json with source pages, each objective’s printed level label and text, and each endpoint’s own label/text plus proposed parent and attachment evidence. Use one row per explicit objective, not one per clause or drug arm. […] Close each source list at a verified section boundary A page ending is not an endpoint-list ending. For each endpoint subsection, read through every continuation page until the next same-level or higher-level heading. If the last retrieved page contains endpoint bullets, read the following page even when it has no repeated endpoint heading. […] In source_map.json, record an endpoint_sections inventory: subsection heading, physical start/end pages, top-level item count, first/last item, and the following heading that proves the list ended. A clipped tool response or a fixed page window does not establish that boundary. […] Preserve each field independently • objectiveText : copy the complete objective sentence or printed list item; do not shorten a “To assess …” sentence into its topic or combine separate objectives. • objectiveLevel : PRIMARY, SECONDARY or EXPLORATORY (including tertiary/other). • levelVerbatim : the level label actually used in the source at that objective. Prefer the direct adjacent level cell/label when present (e.g. “Primary”); do not append “Objectives”, numeric section prefixes or invented capitalization. Preserve meaningful phase/cohort distinctions when they are part of the level label. Check this field explicitly; correct objective text with the wrong label is incomplete. For a plain category heading consisting only of Primary, Secondary, Exploratory, Tertiary, Safety or Other followed by Objective(s), use the category word itself as the level-label span, preserving its printed capitalization. […] • endpointText : the detailed printed endpoint sentence/list item, including its timepoint, definition and units. Separate explicitly enumerated/co-primary endpoints instead of replacing them by an earlier summary sentence. […] • endpointLevel : read the endpoint’s own heading. Do not inherit the objective level automatically. A primary efficacy objective may have secondary/exploratory efficacy endpoints; they still belong to that primary efficacy objective. Attach by the question measured, not by the hierarchy label Prefer explicit table rows or cross-references. Otherwise compare the measurement, clinical domain, population, intervention comparison and time frame to the question posed by each objective. […] The same endpoint may legitimately occur under more than one objective if the source states this; do not globally deduplicate away valid attachments. An objective may have no endpoint: use [] when none is supplied. […] Audit and minimal fix Independently check objective count, every verbatim level label, endpoint inventory, endpoint levels and each attachment. Count source endpoints before comparing output. For every unassigned source endpoint, either identify its objective or explain the source ambiguity; for every emitted endpoint, identify source evidence. Inspect cross-page table continuations and detailed definitions, not just the synopsis. Independently open the page after each claimed endpoint-list ending and verify the recorded closing heading. […]

Appendix

Table 23: The TrialScribe skill for objectives and endpoints extraction (abridged).

Reading an SoA page out of a PDF The job is to reproduce one page of a clinical trial protocol — a Schedule of Activities table — as HTML. There are two ways to read the page, and which one works is a property of the file, not a preference. Text extraction on a page whose text layer is broken produces confident, wrong output. […] Step 1 — Triage (always first, costs one command) python3 /app/triage_pdf.py /app/inputs/protocol.pdf It prints one of three routes and the numbers behind it. Follow the route it gives you. The thresholds sit in wide empty gaps in the measured distribution, so a borderline page is rare — if you get one, prefer VISION. route what it means go to TEXT clean text layer, ruling lines present Step 2a VISION text layer exists but is unusable Step 2b VISION_ONLY no text layer at all — a scan or a picture Step 2c Do not skip this and do not argue with it. "The text extraction looked plausible" is exactly how the failures happen: a broken encoding yields well-formed rows full of wrong characters. Step 2a — Route TEXT The text layer is usable. Parse it. python3 -c "import pdfplumber; print(pdfplumber.open(’/app/inputs/protocol.pdf’).pages[0].extract_table() is not None)" pdfplumber and PyMuPDF are already installed — do not spend turns on pip install . Follow .claude/skills/pdf/SKILL.md , section SoA Table Extraction , for the patterns that matter here: None vs ’’ semantics, column groups, normalize_row_v2 , section-header merging, superscript handling, colspan validation, X-cell variants. If extract_table() returns None or an empty table, stop and switch to Step 2b. That is the one case triage can miss when PyMuPDF is unavailable and the ruling-line check could not run. Then go to Step 3. Step 2b — Route VISION (text layer present but unusable) Two different things are broken under this route, and the triage output tells you which: "decodes to junk" — some fraction of the characters are wrong. The geometry is still sound, so you may use pdfplumber to find the grid: how many columns, where the row boundaries are, which cells are spanned. But every character you emit must be read off the rendered image , not copied from the extracted text. […] "no ruling lines" — extract_table() will return nothing, because it finds tables by their lines. There is no grid to recover from the text layer. Build the table from the image. Either way, render it: pdftoppm -png -r 200 -f 1 -l 1 /app/inputs/protocol.pdf /tmp/page Re-render a region at higher DPI (300, 400) whenever a character is doubtful. That is cheap, and it is the only way to settle it. If the render shows no schedule table on the page at all — just body text — say so plainly in your final message and write what the page does contain. Do not invent a table to fill the file. Then go to Step 3. Step 2c — Route VISION_ONLY (no text layer) Same as Step 2b, with one thing to be explicit about: Do not install or use pytesseract . The tesseract binary is not in this image, so OCR cannot run — the import will succeed and the call will fail. Your own vision is the extraction method here, not a fallback for it. pdftoppm -png -r 200 -f 1 -l 1 /app/inputs/protocol.pdf /tmp/page Then go to Step 3.

Appendix

Table 24: The TrialScribe skill file for SoA table extraction (part 1 of 2): triage and reading routes.

Step 3 — The fix loop Applies to every route. A table that reads plausibly can still attach every mark to the wrong visit, and that scores zero. /app/verify_soa.py reads only your own HTML — there is no answer key in this container — and reports structural problems. # fix until the structural report is clean while ! python3 /app/verify_soa.py /app/output/soa_table.html; do : read the findings, edit the HTML, run it again done Then read your own column names back against the page: python3 /app/verify_soa.py /app/output/soa_table.html --keys --keys prints, for every visit column, the full header path a reader would follow to name it: col 1: Screening Assessments | Within 4 weeks (Day -28 to 0) col 2: Screening Assessments | Within 1 week (Day -7 to 0) col 3: Cycle 1 | 1 (1) Put that beside the rendered page and walk the columns left to right: 1. The count. As many printed columns as visit columns on the page. 2. The order. Column n names the n -th visit column on the page. An off-by-one here is the single most common way to score zero: every mark lands one visit over, so nothing matches even though the table looks right and the checker is happy. 3. The levels. Each key carries every level the page shows for that column — period, visit, visit number — and no level belonging to a neighbour. If column 1’s key contains the row-label heading ("Study Procedure", "Visit Number", "Assessment/Procedure"), the label has leaked into the column header and every visit name is shifted by one. That is a shift, not a typo: fix the corner cell, do not patch the names. Do not rewrite a header that is already right Rewriting headers that did not need it can cancel out everything the loop gained elsewhere. The reason: a column’s identity is its entire header path, including any footnote marker. Drop one a from a cell with colspan="15" and all 15 columns under it stop matching. So: • If the structural report is clean and the rendered page agrees with your --keys output, you are done. Submit it. • Change a header cell only when you can point at the specific thing on the page that disagrees with it. • When you do change one, copy the cell’s text exactly, superscripts included. What the checker cannot tell you It has no access to the page. It cannot see a table that is internally perfect but uniformly shifted, or a cell whose text is misread. Many ways of failing this task are invisible to it — which is why the read-back against the render is not optional, and also why a clean report is not permission to start editing. Output Write the finished table to /app/output/soa_table.html . Detailed extraction technique: .claude/skills/pdf/SKILL.md .

Appendix

Table 25: The TrialScribe skill file for SoA table extraction (part 2 of 2): the fix loop.

--- name: timeline-extraction description: Extract and audit USDM epochs, encounters and activity definitions from a protocol with source inventories, explicit attribute evidence and compatible timing output. --- Source-grounded USDM components Read USDM_REFERENCE.md , the supplied instruction and schema. The schema is a four-array projection, not the complete USDM graph. Preserve its field names. Never read gold, study-specific reference workbooks or same-study structured examples. A source plan and earlier audit are fallible model outputs. Extract with a source inventory 1. Locate study design, schedule tables, all continuation pages, visit descriptions, activity definitions and relevant footnotes. Inspect page images for ambiguous spans, composite rows and damaged text. Track physical PDF pages. 2. Inventory source stages, contacts and activity rows/groups before deciding output counts. Retain source_id, page, quotation, scope and output row indices in source_map.json. Explain exclusions. Prioritize a usable timeline.json before elaborating the sidecar; update both when fixing output. 3. For every row, keep a concise source-readable name and label. Put qualifications in description/notes, not synthetic ID-like names or long identity labels. 4. Explicitly review the fields below; missing evidence means null, not a guessed default. An omitted field must not pass unnoticed merely because schema allows it. Epoch: stage, classification and sequence Record source-supported sequential stages in study order. Classification belongs in type, e.g. SCREENING, RUN-IN, TREATMENT, WASHOUT, FOLLOW-UP when appropriate. These examples are not an exhaustive ontology. Keep period numbers in name/label; use the same stage class for distinct treatment periods when the source defines them. Do not emit run_in , generic Study phase , an arm, or a residence setting as a stage classification. Never automatically split a stage at every dose or site change. Never merge genuinely distinct source stages solely because their types coincide. […] Encounter: identity, kind, and mode 1. Locate the schedule of activities AND its footnotes. Read the relevant visit/contact prose. Render a schedule page when merged headers make text ambiguous. Use a compact working inventory of visit/day columns, not a long planning document. 2. An encounter is a planned participant contact, not an epoch or an individual procedure. Preserve distinct visit numbers and period/day identities. A continuous inpatient period contains daily encounters when daily clinical assessments, blood sampling, or staff-administered treatment are scheduled: expand an explicit daily range into day-specific records. […] Distinguish conditional/unscheduled contacts from routine contacts. 3. Read footnotes for telephone contacts tied to specific visits. If a numbered visit comprises an onsite component and a later telephone assessment, retain its identity and both contact modes, describe their different timing. […] 4. Use type "Visit" for clinical study visits; put screening, inpatient, unscheduled, early termination, and follow-up qualifiers in name/label/description, not invented type values such as "Visit (unscheduled)". 5. contactModes is an array of display terms, e.g. ["In Person"], ["Telephone Call"], or both. […] Use null only when the source and scheduled procedures do not establish a mode. 6. Use short identifiable names/labels containing period and day or visit number. […] Required final consistency check: if your source evidence states a telephone component is attached to a numbered visit, that visit must include Telephone Call as well as its onsite mode. […] Keep encounter evidence in the shared source_map.json, alongside epoch, activity and timing evidence. […] Activity: definition, group and scheduled occurrence Inventory exact schedule row designations and explicit presentation group headings. Read prose/footnotes for additional source-stated procedures. Preserve a composite row as a composite unless the source defines independent activities. Reuse an activity definition across visits; occurrence scheduling does not create new copies. USDM permits grouping activities. […] Compact audit and repair Review the actual source, not only candidate counts. Report specific errors with short literal quotes and physical page numbers. Uncertainty is a warning, not an automatic repair or deletion. […] Repair only explicitly identified errors after an independent source check; preserve correct values. […] For timing, FROM occurs BEFORE/AFTER TO, and start/end relationships and windows retain their original meaning. Use readable source event names and period context rather than anonymous TP IDs. A zero offset can legitimately relate distinct coincident events. Do not infer elapsed durations by subtracting nominal day labels. Reference existing encounter names when the endpoint is that encounter; a within-visit procedure endpoint must identify its own source event and period.

Appendix

Table 26: The TrialScribe skill for structured timeline extraction (abridged; omitted passages are marked […]).

Panel of Figure 8 — Figure 8: SoA table extraction case study: source pages and the TrialScribe output.

Panel of Figure 9 — Figure 9: SoA table extraction case study: OCR and document-understanding baselines on the same pages, annotated against the reference.

Panel of Figure 10 — Figure 10: SoA table extraction case study of TrialScribe .

Table type	What it describes
PK sampling	PK samples relative to dosing
PD sampling/assessment	Pharmacodynamic measurements
Laboratory assessment	Hematology, chemistry, urinalysis, etc.
Safety assessment	AEs, vitals, ECG, physical exam, etc.
Efficacy assessment	Tumor assessments, disease scores, etc.
Study treatment/dosing	Dose, schedule, route, cycles

Appendix

Table 28: Other common protocol table types beyond those covered in TrialPD - Bench .

Coding Agents

Coding agents use language models, tools, and feedback to modify software and complete engineering tasks.

See all 139 papers

May 10, 2026

CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

Clinical reasoning agents based on large language models (LLMs) aim to automate tasks such as intensive care unit (ICU) monitoring and patient state tracking from electronic health records (EHRs). Existing systems typically rely on manually curated clinical tools or skills for concepts such as sepsis detection and organ failure assessment. However, maintaining these tool libraries requires substantial expert effort, while zero-shot querying or code generation often produces inefficient and unreliable reasoning chains, especially under institution-specific clinical policies. We introduce CodeClinic, a benchmark built on MIMIC-IV for evaluating whether LLM agents can synthesize and compose reusable clinical skills instead of relying on fixed toolboxes. The benchmark contains two complementary tasks: longitudinal ICU surveillance and compositional information seeking. The longitudinal setting simulates monitoring patient trajectories with structured decisions every four hours across 25 findings and eight clinical families, while the compositional setting spans 63k instances across 259 tasks in nine domains and is stratified by compositional dependency depth to evaluate increasingly complex multi-step reasoning. We further propose an offline autoformalization pipeline that converts natural-language clinical guidelines into reusable and verified Python skill libraries through iterative LLM refinement. Compared with zero-shot code generation, the resulting libraries improve consistency while reducing per-query token usage by up to 40%.

Timothy Ossowski, Xinchi Liu, Danyal Maqbool +6

University of Wisconsin–Madison · 2Microsoft Research

Healthcare LLM Agent Skill Learning Clinical Reasoning Autoformalization LLM Code Generation Agent Benchmarks

Ask PDF

Sep 25, 2026

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity. Using the benchmark as an optimization signal, we discover a description-writing prompt that reaches full fidelity and generalizes to unseen files. We then test the hypothesis that motivated the work: that better documentation helps an agent resolve real repository issues. Across two model families and ten repositories, and against a positive control confirming that our evaluation can detect a genuine improvement, we find that it does not. When the source is present, neither static compact documentation nor retrieved context beats the issue alone. We report this negative result together with the benchmark and the optimizer, and we characterize the boundary at which documentation helps.

Md Shohel Arman, Igor Molybog

Daffodil International University, Dhaka, Bangladesh · HawAII, University of Hawai‘i at M¯anoa, Honolulu, HI, USA

Software Engineering AI Coding Agents Code Generation Benchmarks Text Summarization

Ask PDF

Jul 28, 2026

HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let that document govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document constrains its behavior over an extended tool-use horizon. We present HANDBOOK_md, a benchmark of 65 agentic tasks modeled on how employees follow company handbooks. Each task places an agent in a self-contained company environment (a file workspace with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol) and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20-124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and 10 fictional companies. To resist memorization, every task modifies one of 10 base handbooks, altering the specific rules and thresholds on which grading depends, so no two tasks share the same set of policies. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the strongest evaluated model passes 36.2% of trials, and most frontier models remain below 25%. Failures follow consistent patterns: agents let a plausible but unauthorized in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release the tasks, environments, and evaluation harness.

Liudas Panavas, Sebastian Minus, Bradley Monton +4

Surge AI*

Computer-Use Agent Benchmarks AI Agent Reliability Tool-Augmented Language Model Agents Long-Horizon Agent Evaluation AI Agent Benchmarks Instruction Following+1

Ask PDF

Abstract

Coding Agents

Explore similar work

CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following