Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses

Chengyang He; Wenlong Zhang; Koduvayur Subbalakshmi; Ping Wang

cs.CLOct 29, 2026

Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses

Authors: Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi, Ping Wang

Organizations: Stevens Institute of Technology, Hoboken, NJ, USA

Abstract

Ask a language model whether sales tend to dip after a product recall, or whether one side effect tends to follow another after a vaccine, and it answers in a fluent sentence that reads like a finding, with no test behind it. Assertions about how events unfold across many cases fill the Web, and each is a temporal hypothesis that only a significance test over the right records can settle. The verifier must specify the null, select the test and decide which records bear on the hypothesis, and any of these choices can go wrong while the conclusion still comes out right. Yet benchmarks for language-model data analysis grade only the conclusion, with the relevant data handed over. We study the problem in health, where a wrong verdict is costly, and present STRATA, a benchmark that grades every decision on the way to a verdict: 8,735 hypotheses over the clinical timelines of vaccine adverse-event reports, with gold plans, evidence sets, p-values and verdicts derived under a released codebook and reproduced by a deterministic oracle. We also propose ART (articulate, retrieve, test), a staged paradigm that exposes each decision as an inspectable output. Extensive experiments with five model configurations show that systems fail mostly at choosing which records count, a verdict-evidence gap that accuracy alone conceals. ART closes most of this gap and brings a 31B open model to within 0.01 of the frontier models under the same paradigm; the frontier models succeed once given the codebook.

Figures & tables

Figure 1. A worked example. (1) Four Moderna reports, one per category; the bar sorts the 30 reports with both events into the numerator $N$ ( $k=18$ ), the rest of the denominator $D$ ( $n-k=4$ ) and the 8 reports outside $D$ , where the two events share a stage. (2) A single-query baseline and (3) Art , with the same open model.

Figure 2. Overview of the Strata benchmark and the Art pipeline: (1) construction, (2) verification, (3) the measure that scores each stage against its gold.

Work	Null	Test choice	Test run	Temporal hypotheses	Clinical	Stage gold
StatQA ( Zhu et al., 2024 )	❍	✓	✗	✗	✗	✗
StatFormBench ( Wang et al., 2026 )	❍	❍	✗	✗	✗	❍
StatABench ( Zhu et al., 2026 )	❍	✓	❍	✗	✗	✗
P-Bench ( Miao et al., 2026 )	✗	❍	❍	❍	❍	❍
DiscoveryBench ( Majumder et al., 2025 )	❍	❍	❍	✗	✗	❍
EHRSQL ( Lee et al., 2022 )	✗	✗	✗	✗	✓	❍

Table 1. Comparison of representative related work on what each work asks and scores (full version: Appendix A ). Test choice : chosen and scored; Test run : the verdict follows from the test result; Stage gold : gold beyond the answer. ✓ yes, ❍ partly, ✗ no, – n/a.

Test set (Medium / Hard)	1,500 (1,418 / 82)
Distinct facts	861
Single-report case study (Easy)	180
Development set (4–10 hypotheses per group)	184
Groups / families / difficulty levels	26 / 7 / 3
Registers per hypothesis	4
Pool: hypotheses (texts)	8,735 (34,940)

Table 2. Basic data statistics. Per-group sizes are in the hypothesis catalogue (Appendix B ); Easy hypotheses have no insufficient class.

Methods	Qwen2.5-32B pair				qwen3.8-27B				gemma4-31B				Opus 5.5				GPT-5.6
	Acc	P	R	F1	Acc	P	R	F1	Acc	P	R	F1	Acc	P	R	F1	Acc	P	R	F1
EHRSQL	.528	.552	.522	.516	.593	.611	.588	.585	.634	.647	.630	.627	.867	.877	.866	.868	.872	.872	.871	.871
EHRSQL-C	.577	.606	.571	.571	.744	.763	.740	.743	.857	.887	.854	.860	1.000	1.000	1.000	1.000	.984	.984	.983	.983
EHRAgent	.596	.689	.602	.592	.838	.858	.836	.842	.733	.738	.732	.734	—				.796	.809	.795	.798
DiscoveryBench	n/a				.650	.823	.647	.716	.674	.763	.671	.706	—				—
Art (Ours)	.806	.878	.801	.822	.869	.893	.867	.878	.981	.983	.981	.982	.991	.991	.990	.991	.981	.982	.981	.981

Table 3. Main results on the test set: verdict accuracy (Acc) and macro P, R and F1 over the three verdicts; leading zeros dropped. Rows below Art remove one of its components. Under the Qwen2.5 pair, the single-query baselines and the agents run Qwen2.5-Coder alone. 95% intervals on Acc are at most

\pm 0.03

. Bold/underline: best/second best among the adapted baselines and Art . Dash: run not made, to bound API cost; n/a: the agent does not run with this model.

Figure 3. Evidence errors of Art on the test set, by hypothesis family and model (Section 5.3 ); 0 is perfect in every panel. The colour scale is square-root.

Figure 4. Art on the test set: (a) verdict accuracy by hypothesis family; (b) accuracy by paraphrase register, open models; (c) change in verdict accuracy when the cap on re-asks per stage rises from

T-1

T

( Art as reported uses

T=2

		Qwen2.5	qwen3.8	gemma4	Opus	GPT
Stage	Methods	32B pair	27B	31B	5.5	5.6
S1	Art : Articulate (Ours)	.952	.915	.992	.991	.991
	Plain prompt	.871	.856	.919	—	—
	Plain prompt + CoT	.923	.859	.909	—	—
S2	Art : Retrieve (Ours)	.833	.943	.989	1.000	.991
S2	OmniSQL-32B ‡	.740	n/a	n/a	n/a	n/a

Table 4. Stage baselines on the test set: verdict accuracy with one stage filled by Art ’s component or a baseline, the others gold or deterministic. CoT: zero-shot chain-of-thought. ‡ Fine-tuned from Qwen2.5-Coder-32B. Bold: best per column within a stage where a baseline exists. Dash: run not made.

Figure 5. Verdict accuracy on one hypothesis shape, A precedes B , by difficulty: Easy, one report (P-R1, 20 hypotheses); Medium, one cohort (P-V1, 95); Hard, two cohorts (P-M2, 82). One panel per model, Art against EHRSQL-C; bands are 95% bootstrap intervals over facts.

	Qwen2.5	qwen3.8	gemma4	Opus	GPT
Methods	32B pair	27B	31B	5.5	5.6
EHRSQL	.433	.989	.744	—	—
EHRSQL-C	.750	1.000	1.000	1.000	1.000
Art	1.000	1.000	1.000	.983	1.000

Table 5. Verdict accuracy on the 180 single-report hypotheses (20 per group). Bold: best per model; a dash marks a run not made.

		Qwen2.5	qwen3.8	gemma4	Opus	GPT
Metric	Methods	32B pair	27B	31B	5.5	5.6
Acc	EHRSQL	.528	.593	.634	.867	.872
	EHRSQL-C	.577	.744	.857	1.000	.984
	Art	.806	.869	.981	.991	.981
Ev. F1	EHRSQL	.547	.588	.692	.880	.899
	EHRSQL-C	.643	.751	.836	1.000	1.000

Table 6. Evidence behind the verdicts on the test set, for the end-to-end systems that return evidence sets (Section 5.3 ); Acc repeats Table 3 ; Ev. F1: Evidence F1; Faithful: Faithfulness. Bold: best per metric and model.

Figure 6. The insufficient class on the test set, by system and model. Left: overclaim by gold sub-type. Right: underclaim by cause; grey where the cause is not observable. Numbers are row totals.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Null	Test selection	Test execution	Temporal hypotheses	Clinical data	Gold released
Benchmarks
StatQA ( Zhu et al., 2024 )	❍ applicability only	✓ model picks from a catalogue, scored	✗ none run	✗ no	✗ no	✗ final answer
StatFormBench ( Wang et al., 2026 )	❍ test family	❍ model names the family	✗ none run	✗ no	✗ textbook	❍ formulation
StatABench ( Zhu et al., 2026 )	❍ test choice	✓ model picks, scored	❍ fixed tool functions	✗ no	✗ no	✗ final answer
P-Bench / Fisher-R1 ( Miao et al., 2026 )	✗ given	❍ model picks, unscored	❍ model-written R	❍ survival only	❍ partly	❍ reference $p$
DiscoveryBench / BLADE ( Majumder et al., 2025 ; Gu et al., 2024 )	❍ open-ended	❍ agent, rubric-scored	❍ agent code	✗ no	✗ no	❍ rubric

Appendix

Table 7. Positioning against related work by what each work asks and scores. Null : who states the null and the population it is tested over. Test selection : who chooses the test and whether that choice is scored. Test execution : what computes the statistic the verdict rests on, and with which tests. Temporal hypotheses : hypotheses about the order or timing of events within a report. Gold released : what a benchmark scores besides the final answer; systems release no gold of their own. Marks: ✓ has it, ❍ partly, ✗ lacks it, – not applicable.

Group	Hypothesis template ( Original register)	Reports counted	Null	Test	Hyp.	Med. $n$
Easy: read off one report
P-R1	In report $r$ , $A$ occurred before $B$ .	the named report	—	none	400	—
P-R2	In report $r$ , $A$ came first, before $B$ .	the named report	—	none	400	—
C-R1	In report $r$ , $A$ and $B$ co-occur.	the named report	—	none	400	—
S-R1	In report $r$ , $A$ , $B$ , $C$ occur in the order $A$ $\rightarrow$ $B$ $\rightarrow$ $C$ .	the named report	—	none	400	—
F-R1	In report $r$ , $A$ occurs first among $A$ , $B$ , $C$ .	the named report	—	none	400	—

Appendix

Table 8. Hypothesis catalogue: the 26 template groups by difficulty level.

A

–

D

are event terms,

v

a vaccine cohort and

r

a report; for example, P-V1 is realised as “After Pfizer, Headache typically occurs before Fatigue”. Reports counted : the denominator rule. Null :

0.5

is the symmetric null; runner-up tests the named candidate against its strongest competitor; two rates equal compares two arms; data-derived computes chance from the denominator reports (a report’s share of stages in the earlier half, or the share of its distinct-stage event pairs that are adjacent); a bar in text is read from the wording. ∗ A report in both arms has its two outcomes exchanged (Appendix C ). Hyp. : released hypotheses; Med.

n

: median denominator size. Single-report hypotheses are read off one timeline with no test. Cohort groups ship near-equal thirds of the three verdicts, except C-V3 and Y-V3 (0.27 and 0.24 per decisive class), whose stricter tests leave more hypotheses insufficient.

System	Model	Seconds	Tokens	API cost (USD)
Art	Opus 5.5	7.1	9,150	0.014
Art	GPT-5.6	6.2	5,600	0.012
Art	Qwen2.5-32B pair	12.3	5,000	0
Art	qwen3.8-27B	10.1	5,150	0
Art	gemma4-31B	14.1	5,300	0
EHRSQL-C	Opus 5.5	5.6	4,850	0.013

Appendix

Table 9. Cost and latency per hypothesis (test set for the local models, usage counts from the test runs for the frontier models). Art rows are the two base calls without repair; tokens are prompt plus completion, and GPT-5.6’s completion count includes its hidden reasoning tokens. Local latencies are measured on one RTX A5000 with the model fully resident; the Qwen2.5 pair’s latency includes swapping the two models between stages.

Scientific Claim Verification

Scientific claim verification checks whether research statements are supported, contradicted, or left uncertain by relevant evidence.

See all 57 papers

CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives
Aug 13, 2026Chengyang He, Tahreem Arif, Marko Zivkovic +3Temporal Reasoning Clinical Reasoning
Stevens Institute of Technology, Hoboken, NJ, 07030, USA · Genesis Research Group, Hoboken, NJ, 07030,USA · New Jersey Institute of Technology, Newark, NJ, 07102, USA
All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting
Feb 19, 2026Zeyu Zhang, Ryan Chen, Bradly C. StadieData Contamination in Language Models Temporal Data Leakage
Department of Statistics and Data Science, Northwestern University · Bridgewater AIA Labs
Hindsight Bias in Clinical Temporal Reasoning: How Future Data Exposure Affects Large Language Model Judgment
Sep 11, 2026Misaki Matsuura, Sayantan Kumar, Ojas Kadam +1Temporal Reasoning in Language Models Clinical Reasoning
Case School of Engineering, Case Western Reserve University, USA · National Library of Medicine, National Institutes of Health, USA · Rice University University, USA

Aug 13, 2026J/K move · Enter open · S save

CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives

Chengyang He, Tahreem Arif, Marko Zivkovic, Lijing Wang+2

Stevens Institute of Technology, Hoboken, NJ, 07030, USA · Genesis Research Group, Hoboken, NJ, 07030,USA · New Jersey Institute of Technology, Newark, NJ, 07102, USA

Temporal Reasoning Clinical Reasoning Inference-Time Guidance

Open paper AskPDF arXiv

Understanding the temporal progression of symptoms in clinical narratives is critical for disease monitoring, safety surveillance, and causality assessment. Clinical narratives, however, rarely provide explicit temporal anchors. Current approaches to temporal information reasoning focus predominantly on pairwise relation classification across multi-visit and timestamp-rich records, leaving the reconstruction of structured symptom trajectories from individual anchor-sparse reports largely unaddressed. We propose CRAFT, an LLM framework that pairs a generator with a constraint-based verifier to iteratively produce and refine stage-wise symptom timelines through targeted feedback. We conduct evaluation on MedTempo, a new benchmark of 5,347 vaccine adverse-event narratives spanning three COVID-19 vaccine types, with expert-validated temporal stage annotations for 3,166 reports. Experiments across four LLM backbones demonstrate that CRAFT consistently improves temporal ordering accuracy, with ablation analysis isolating the contribution of generator and verifier components across model capability levels.

Abstract

Scientific Claim Verification

Explore similar work

CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives

All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting

Hindsight Bias in Clinical Temporal Reasoning: How Future Data Exposure Affects Large Language Model Judgment