ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts

Zhaoyu Fan; Bowen Han; Jincheng Ou; Yanwei Ren; Zehua He; Kaiyu Huang; Pengcheng He; Shanshan Ni

cs.AIOct 12, 2026

ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts

Authors: Zhaoyu Fan, Bowen Han, Jincheng Ou, Yanwei Ren, Zehua He, Kaiyu Huang, Pengcheng He, Shanshan Ni,

Organizations: Tongji University · Shanghai Hengfang Health Technology Co., Ltd. · University of Electronic Science and Technology of China · Shenzhen Research Institute of Big Data

Abstract

Real-world studies (RWS) analyze routinely collected healthcare data, such as electronic health records and insurance claims, to estimate treatment effects, monitor safety, and identify risk factors outside randomized trials. Automating RWS with large language model agents is appealing but brittle: valid analyses depend on dataset-specific clinical semantics, including index dates, follow-up windows, patient/event-level units, and outcome boundaries that are often implicit in observational data. This leads to schema mismatches, temporal errors, inconsistent analytical granularity, and unsupported clinical claims. We present ProCARE, a profile-grounded framework that reformulates RWS automation as constrained compilation. ProCARE converts heterogeneous data profiles into Evidence Contracts specifying available variables, temporal attributes, distributions, and evidentiary limits, and compiles study objectives into RWS-IR, a strongly typed intermediate representation for variables, time relations, analysis units, and statistical tasks. Deterministic validation and localized repair then check and revise generated workflows across schema mapping, cohort construction, variable derivation, analysis, and interpretation. On 36 EHR-based benchmark tasks, ProCARE outperforms strong LLM-agent baselines in code executability, semantic consistency, and conclusion traceability, with automated violation diagnoses showing moderate-to-substantial agreement with blinded expert review.

Figures & tables

Figure 1: ProCARE architecture for contract-valid RWS automation. A heterogeneous RWS profile constrains RWS-IR compilation, violation-guided execution, and evidence-certified DAG reporting as a profile-grounded evidence contract.

	High-level profile (dataset/task)	Low-level profile (field/instance)
Structural	$\Phi^{\mathrm{H,S}}$ Clinical form registry; subject/visit/table linkage; cohort-level missingness.	$\Phi^{\mathrm{L,S}}$ Field type, unit, range, date fields, temporal anchors, and window-specific missingness.
Knowledge	$\Phi^{\mathrm{H,K}}$ Cohort and index-date schema; exposure, outcome, censoring, confounder, and claim-scope roles.	$\Phi^{\mathrm{L,K}}$ Ontology mapping; unit/value validation; temporal-role validation; field-level proxy and confounder roles.

Table 1: Heterogeneous data profile

\Phi

organized by profile level and abstraction.

Dimension	Metrics and scoring
IR	GA : LLM-rubric goal alignment; PF : schema/IR feasibility with LLM adjudication.
Execution	XE : sandbox pass rate $n_{\mathrm{pass}}/N$ ; CQ : LLM-rubric code quality; VRR : resolved/detected contract violations.
Visualization	VC : LLM-rubric visual clarity; CF : chart–result consistency.
Reporting	RS : report schema and section coverage; SL : LLM-rubric scientific language.
Claims	CTC : post-hoc claim trace coverage; UCR : unsupported factual-claim rate; CLV : language-support violation rate.

Table 2: Evaluation metrics for end-to-end RWS automation.

Method	GA $\uparrow$	PF $\uparrow$	XE $\uparrow$	CQ $\uparrow$	VC $\uparrow$	CF $\uparrow$	RS $\uparrow$	SL $\uparrow$	VRR $\uparrow$	CTC $\uparrow$	UCR $\downarrow$	CLV $\downarrow$
ReAct ‡§	0.78	0.62	0.38	0.47	0.28	0.48	0.63	0.59	0.34	0.48	0.24	0.17
ToolLLM ‡§	0.73	0.68	0.00	N/A	N/A	N/A	0.56	0.24	0.00	0.21	0.33	0.24
TableGPT-Agent ‡§	0.61	0.59	N/A	N/A	N/A	N/A	0.40	0.43	N/A	0.18	0.36	0.22
ChatExcel-Max †§	N/A	N/A	N/A	N/A	N/A	N/A	0.52	0.27	N/A	0.12	0.39	0.28
Gemini-3-Pro direct †	0.75	0.51	0.13	0.45	0.28	0.48	0.61	0.53	0.18	0.36	0.27	0.15
TableGPT2 ‡	0.41	0.45	0.00	0.25	N/A	N/A	N/A	N/A	0.00	N/A	N/A	N/A

Table 3: Baseline comparison on 36 HCC RWS tasks. GA–SL report task performance, while VRR–CLV report contract diagnostics following Table 2 ; N/A indicates no executable artifact bundle. † Proprietary; ‡ Open-source; § Agent-based.

Baseline	Fairness scaffold	PF $\uparrow$	XE $\uparrow$	CQ $\uparrow$	VRR $\uparrow$	CTC $\uparrow$	UCR $\downarrow$	CLV $\downarrow$
ReAct + scaffold	$\Phi$ summary + bundle tmpl. + val. feedback + 3 repairs	$0.79$	$0.74$	$0.69$	$0.62$	$0.70$	$0.13$	$0.10$
ToolLLM + scaffold	Same scaffold on the tool-use baseline	$0.80$	$0.68$	$0.65$	$0.58$	$0.66$	$0.15$	$0.12$
G3P + profile/template	$\Phi$ + manifests; no validator loop	$0.73$	$0.66$	$0.62$	$0.31$	$0.55$	$0.20$	$0.15$
G3P + validator repair	$\Phi$ + manifests + val. feedback + repairs	$0.78$	$0.79$	$0.72$	$0.68$	$0.65$	$0.15$	$0.11$
G3P + study-spec schema	Emits study-spec artifact; no IR compiler	$0.82$	$0.70$	$0.68$	$0.43$	$0.52$	$0.21$	$0.16$
G3P + claim-trace schema	Emits claim-trace artifact; no cert. gate	$0.74$	$0.63$	$0.61$	$0.35$	$0.74$	$0.11$	$0.08$

Table 4: Scaffolded RQ1 controls. Direct rows use Gemini-3-Pro with the same decoding settings as the default ProCARE backbone. Non-ProCARE rows are independent baseline adaptations; ProCARE modules are not transplanted into them.

Variant	PF $\uparrow$	XE $\uparrow$	VRR $\uparrow$	CTC $\uparrow$	UCR $\downarrow$	SL $\uparrow$	Dominant failure
No profile $\Phi$	0.49	0.18	0.35	0.58	0.28	0.42	Field hallucination; invalid joins.
No RWS-IR	N/A	0.34	0.44	0.63	0.22	0.69	Missing grain, anchors, and claim policy.
No repair	0.84	0.69	0.43	0.80	0.13	0.70	Unresolved runtime and artifact violations.
No certificates	0.84	0.92	0.81	N/A	0.26	0.62	Unsupported causal or comparative claims.
No DAG ordering	0.84	0.92	0.81	0.84	0.11	0.64	Dependency-agnostic report ordering.
Full ProCARE	0.84	0.92	0.81	0.90	0.06	0.79	Localized repair; traceable final claims.

Table 5: Mechanism ablation over 36 RWS instances. N/A indicates no typed RWS-IR.

Prompt	Claims	CTC $\uparrow$	UCR $\downarrow$	CLV $\downarrow$	Main downgrade reason
Goal	$5.8$	$0.87$	$0.09$	$0.05$	Unsupported causal/comparative wording.
Goal+Approach	$6.4$	$0.90$	$0.05$	$0.03$	Proxy endpoint beyond evidence level.
Goal+Method	$6.9$	$0.94$	$0.03$	$0.02$	Sparse endpoint or subgroup evidence.
Overall	6.4	0.90	0.06	0.03	Downgrades mainly concern causal strength.

Table 6: ProCARE claim diagnostics. Values are averaged over 12 instances per prompt level; Overall aggregates all 36 instances.

Stage	Reviewers	Metric(s)	Auto	Expert	$\rho$
RWS-IR	3 clinical + 2 stats/data	$(GA+PF)/2$	0.82	4.38 $\pm$ 0.48	0.82
Code	2 stats/data	$CQ$	0.88	4.42 $\pm$ 0.21	0.74
Visualization	3 clinical + 2 stats/data	$(VC+CF)/2$	0.66	4.50 $\pm$ 0.53	0.75
Reporting	3 clinical + 2 stats/data	$SL$	0.76	4.13 $\pm$ 0.64	0.71
All Spearman correlations are computed across the 36 primary HCC instances; all $p<0.001$ .

Table 7: Expert consistency of automated metrics. Expert scores are mean

\pm

SD on a 5-point scale; CI and reliability details are reported in the appendix.

Figure 2: Performance across four RWS tasks and prompt-specificity levels. Cell labels report raw metric values; UCR and CLV are lower-is-better rates with reversed color intensity.

Dataset	Method	XE $\uparrow$	VRR $\uparrow$	CTC $\uparrow$	UCR $\downarrow$	CLV $\downarrow$
UCI Diabetes	Claude Code	0.60	0.61	0.69	0.12	0.09
	Scaffolded G3P	0.57	0.59	0.66	0.13	0.10
	ProCARE	0.64	0.72	0.87	0.07	0.04
NHANES	Claude Code	0.52	0.54	0.63	0.15	0.11
	Scaffolded G3P	0.50	0.52	0.60	0.16	0.12
	ProCARE	0.58	0.66	0.84	0.09	0.06

Table 8: Public-RWS-18 transfer results. Values average six tasks per dataset; baseline claim traces are computed post hoc.

Family	Model	Performance								Contract diagnostics
		GA $\uparrow$	PF $\uparrow$	XE $\uparrow$	CQ $\uparrow$	VC $\uparrow$	CF $\uparrow$	RS $\uparrow$	SL $\uparrow$	VRR $\uparrow$	CTC $\uparrow$	UCR $\downarrow$	CLV $\downarrow$
GPT	GPT-5.2 †	0.84	0.82	0.94	0.84	0.68	0.70	0.78	0.77	0.78	0.88	0.06	0.04
	GPT-4 Turbo †	0.81	0.78	0.89	0.80	0.64	0.66	0.75	0.74	0.74	0.85	0.08	0.06
Claude	Claude Opus 4.5 †	0.85	0.83	0.97	0.86	0.72	0.74	0.78	0.77	0.80	0.89	0.06	0.04
	Claude Sonnet 4.5 †	0.83	0.81	0.94	0.84	0.70	0.72	0.76	0.75	0.78	0.87	0.07	0.05
Gemini	Gemini-3-Pro †	0.86	0.84	0.92	0.88	0.72	0.74	0.80	0.79	0.81	0.90	0.06	0.03

Table 9: Backbone sensitivity with fixed ProCARE workflow. Metrics follow Table 2 ; model access details and version identifiers are reported in the appendix. † Proprietary models; ‡ Open-source models.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Appendix section	Contents	Purpose in the supplement
App. A	Roadmap, complete navigation, and main-paper crosswalk.	Orients readers to both the full supplement and the subset of material that directly supports main-paper claims.
App. B	HCC benchmark scope, release boundary, task construction, Public-RWS-18, claim policies, and worksheet inventory.	Documents the datasets, task instances, and schema constraints needed to reproduce or audit the benchmark.
App. C	Profile schema, concrete profile instance, RWS-IR checks, evidence certificates, validators, artifact bundle, coding harness, and DAG reporting.	Specifies the executable contract objects and implementation checks behind ProCARE.
App. D	Execution environment, cost accounting, baseline adaptation, scaffolded fairness controls, ablations, model access, and repository commands.	Records experimental controls and reproducibility details.
App. E	Claim-diagnostic interpretation and backbone-sensitivity interpretation.	Explains claim-policy downgrades and performance differences across backbones.
App. F	Dashboard, profile panel, RWS-IR view, coding loop, report review, and section-wise report generation screenshots.	Supplies workflow/interface evidence showing where artifacts, traces, logs, and evidence-linked report sections are inspected.

Appendix

Table 10: Complete appendix navigation. The map covers all supplement sections, including audit and interface material that is not tied to a specific main-paper table or figure.

Main-paper location	Claim or reference	Supplementary support
Sec. 3.1	The heterogeneous profile records structural and knowledge-driven constraints, with a concrete serialization and de-identified profile instance.	App. C.1 and App. C.2 .
Sec. 3.2	RWS-IR compilation and execution enforce schema, semantic-role, temporal, grain, method, and artifact constraints.	App. C.3 , App. C.5 , and App. C.10 .
Sec. 3.3	Evidence certificates and claim policies gate factual report claims, while a DAG orders supported report sections.	App. C.6 , App. C.9 , and App. C.11 .
Sec. 4	The primary benchmark contains 36 HCC tasks over four objectives and a 54-sheet longitudinal eCRF workbook.	App. B and Table 23 .
Sec. 4 – 5	Public-RWS-18 uses three public datasets, dataset-specific claim policies, and the same validators and rubrics.	App. B.6 .
Main Table 3	LLM-agent, table-analysis, and code-agent baselines are evaluated under a unified prompt, budget, artifact contract, and post-hoc checker.	App. D.2 .

Appendix

Table 11: Main-paper claims and corresponding supplementary support. This crosswalk is not the full appendix map; complete section navigation is provided in Table 10 .

Release item	Included material	Privacy and reproducibility treatment
De-identified HCC tables	54 linked eCRF sheets covering the 1,294-subject benchmark cohort.	Direct participant identifiers are removed; random benchmark IDs preserve subject-level and visit-level linkage required for rerunning all tasks.
Task and prompt files	The 36 primary HCC benchmark prompts and task metadata.	Prompts refer only to de-identified table and field names and do not expose participant identities.
Schema/profile artifacts	Field dictionaries, join keys, temporal roles, missingness summaries, and profile objects.	Contains schema and aggregate metadata used by validators; no direct identity fields are present.
Evaluation assets	Validators, judge prompts, artifact templates, scoring scripts, and expected output schema.	Enables independent reruns against the released de-identified tables.
Release metadata	Data dictionary, checksums, and responsible-use terms.	Documents provenance, file integrity, de-identification scope, and restrictions against re-identification or clinical use.

Appendix

Table 12: HCC benchmark release contents and privacy treatment.

Group	Representative sheets	Main contents	Use in benchmark/profile
System metadata	SYSVAR, ECRF, DATA_RANGE	Form catalog, site allocation, variable descriptions, and codebook fields.	Schema parsing, worksheet indexing, and field-name grounding.
Baseline	DM, PH, CP, BCLC, ECOG	Demographics, history, liver function, staging, and performance status.	Cohort definition, baseline table construction, and confounder checks.
Longitudinal labs	LB_AFP, LB_BB, LB_BR, LB_TT	Tumor markers, biochemistry, complete blood count, and coagulation tests.	Safety proxy signals and follow-up burden measures.
Treatment	TACE, TART, IMU, PATE, CM	Procedures, systemic therapy, retreatment, and concomitant medication records.	Exposure definition, treatment intensity, and resource-utilization tasks.
Outcomes	VT, AE_HQ, SAE_HQ, SF	Recurrence and survival proxies, AE/SAE screening, and quality-of-life forms.	Endpoint derivation and limitation checks.

Appendix

Table 13: Functional worksheet groups used by profiling and benchmark construction.

Level	Information provided	Expected behavior
GOAL	High-level study intent only.	Select feasible variables and disclose assumptions.
GOAL+APPROACH	Study intent plus an analysis strategy.	Follow the strategy when the data support it; otherwise report the deviation.
GOAL+METHOD	Specific variables, time windows, or statistical method.	Stop or report infeasibility when required data are missing.

Appendix

Table 14: Prompt specificity levels in the benchmark.

Objective	ID	Core research question	Primary data sources
Cohort	C1	Characterize the analyzable HCC cohort, including enrollment, baseline characteristics, and clinical staging.	CO , DM , IE , SG , BCLC , CP , ECOG
Cohort	C2	Assess longitudinal follow-up completeness and data density.	CPH_FT , VT , LB_AFP , LB_BB , LB_BR , ECOG
Cohort	C3	Assess data readiness for downstream endpoint analyses.	DM , SG , BCLC , CP , ECOG , VT , treatment sheets
Safety	S1	Summarize AE/SAE screening rates and the availability of detailed event records.	AE_HQ , SAE_HQ , AE , SAE
Safety	S2	Describe postoperative laboratory safety signals from liver function, hematology, and coagulation panels.	LB_BB , LB_BR , LB_TT , SG
Safety	S3	Compare safety proxy signals after major treatment exposures.	TACE , TART , IMU , PATE , AE_HQ , SAE_HQ , lab sheets

Appendix

Table 15: Core research questions used to instantiate benchmark prompts.

ID	Prompt text	Required or expected evidence
C1-G	Characterize the baseline hepatocellular carcinoma cohort available in the workbook. Deliver a cohort summary suitable for a real-world study report.	Discover DM , SG , BCLC , CP , ECOG , and subject identifiers.
C1-A	Characterize the baseline HCC cohort. Build a STROBE-style cohort table covering demographics, surgery, liver function, tumor stage, and performance status.	Use patient-level baseline records, report missingness, and avoid duplicate subject counting.
C1-M	Produce a baseline cohort table for subjects with valid Subject_Id . Join DM , SG , BCLC , CP , and baseline ECOG ; summarize AGE , SEX , HEIGHT , WEIGHT , SGDAT , SGTYP , BCLC , CPRES , CPGRA , and ECOG .	Use one baseline row per subject; report variable-level missingness; do not impute incomplete AGE .
C2-G	Assess whether the dataset has sufficient longitudinal follow-up for real-world evidence generation.	Identify visit-level forms and repeated measurements.
C2-A	Assess longitudinal follow-up completeness across follow-up forms, laboratory panels, tumor-marker tests, ECOG assessments, and efficacy follow-up records.	Use CPH_FT , VT , laboratory sheets, and ECOG ; produce visit-level summaries.
C2-M	Quantify longitudinal data completeness using CPH_FT , VT , LB_AFP , LB_BB , LB_BR , and ECOG ; count distinct visits by Subject_Id , tabulate availability by Visit_No , and estimate follow-up duration from SG.SGDAT to the latest VT.VTDAT1 or VT.VTDAT2 .	Use Subject_Id and Visit_No ; report subjects with no valid follow-up date.

Appendix

Table 16: Cohort benchmark instances. Suffixes denote prompt specificity: G = GOAL, A = GOAL+APPROACH, and M = GOAL+METHOD.

ID	Prompt text	Required or expected evidence
S1-G	Analyze the adverse-event and serious-adverse-event safety profile in the cohort.	Discover that AE/SAE screening pages exist but detailed event tables are empty in the current workbook.
S1-A	Analyze AE and SAE safety reporting. Start with screening pages, then check whether detailed AE/SAE records are available for term, onset date, severity, causality, outcome, and seriousness.	Document infeasibility of detailed AE/SAE term analysis when detail tables have no valid records.
S1-M	Summarize AE/SAE reporting completeness using AE_HQ.AEYN and SAE_HQ.SAEYN ; inspect AE.AETERM , AE.AESTDAT , AE.AESEV , AE.AEREL , AE.AESAE , SAE.SAESTDAT , and SAE.SAESEV .	Compute screening prevalence by subject; terminate detail-level analysis if AE and SAE contain zero valid records.
S2-G	Identify laboratory safety signals after surgery.	Use longitudinal laboratory sheets rather than relying only on AE tables.
S2-A	Identify post-surgical laboratory safety signals by summarizing liver function, hematology, and coagulation markers over follow-up visits.	Use lab value/result-judgment pairs and visit dates; report abnormal-result flags when available.
S2-M	Generate a post-surgical laboratory safety profile anchored at SG.SGDAT ; use LB_BB.LBBBDAT , ALTRES , ASTRES , TBILRES , ALBRES ; LB_BR.BRDAT , WBCRES , HBRES , PLTRES ; and LB_TT.LBTTDAT , PTRES , INRRES , DDIMRES .	Summarize baseline and postoperative windows; use available-case denominators and _SIG flags when numeric thresholds are unavailable.

Appendix

Table 17: Safety benchmark instances.

ID	Prompt text	Required or expected evidence
E1-G	Analyze recurrence-related outcomes after surgery.	Locate surgery dates and recurrence dates in VT .
E1-A	Analyze recurrence-free outcomes using surgery date as time origin, recurrence events from follow-up records, and clinically relevant baseline risk strata.	Handle multiple follow-up visits and competing or missing dates.
E1-M	Estimate recurrence-free survival proxies using SG.SGDAT as index date; define recurrence as the earliest non-missing VT.PADHDAT or VT.PADRDAT ; censor at latest VT.VTDAT1 or VT.VTDAT2 ; stratify by BCLC.BCLC , CP.CPGRA , and exposure from TACE , TART , IMU , or PATE .	Use earliest valid recurrence date and valid time ordering; generate Kaplan–Meier curves only when dates are sufficient.
E2-G	Analyze overall survival patterns in the cohort.	Identify death dates and follow-up dates.
E2-A	Analyze overall survival by deriving time from surgery to death or last follow-up and comparing clinically meaningful strata.	Use DEDAT1 / DEDAT2 and VT follow-up dates; disclose death-date limitations.
E2-M	Estimate overall-survival proxies using SG.SGDAT as index date; define death date as earliest non-missing VT.DEDAT1 or VT.DEDAT2 ; censor at latest valid VT.VTDAT1 or VT.VTDAT2 ; compare treatment exposures and adjust or stratify descriptively by DM.SEX , DM.AGE , BCLC.BCLC , CP.CPGRA , and ECOG.ECOG when available.	Note that DEDAT2 is unavailable in the current data; avoid over-adjusted models when events are limited.

Appendix

Table 18: Efficacy benchmark instances.

ID	Prompt text	Required or expected evidence
H1-G	Analyze postoperative treatment resource utilization in the cohort.	Recognize that the task concerns utilization rather than direct cost.
H1-A	Analyze postoperative treatment resource utilization by quantifying use and timing of TACE, targeted therapy, immunotherapy, postoperative retreatment, and concomitant medications.	Use procedure and drug counts with dates as utilization proxies.
H1-M	Quantify treatment-utilization intensity by deriving per-subject counts and time-to-first-use for TACE.TACEDAT , TART.TARSDAT , IMU.IMUSDAT , PATE.PATEYN / PATETR* , and CM.CMCAT / CMSTDAT ; summarize utilization distributions and stratify by BCLC.BCLC and recurrence status when feasible.	Explicitly state that no direct cost or billing variables are present.
H2-G	Estimate follow-up and testing burden associated with the RWS cohort.	Use visit and form/test counts as burden proxies.
H2-A	Estimate follow-up and testing burden by counting follow-up visits, efficacy assessments, laboratory panels, tumor-marker checks, coagulation tests, ECOG assessments, and QoL forms per patient.	Summarize burden by patient and visit.
H2-M	Build a follow-up resource-burden profile by counting records in CPH_FT , VT , LB_AFP , LB_BB , LB_BR , LB_TT , ECOG , and SF ; compute total follow-up forms/tests, median counts per subject, and distribution by Visit_No ; use SG.SGDAT and latest VT.VTDAT1 / VT.VTDAT2 to contextualize burden over time.	Frame the result as utilization burden, not economic cost.

Appendix

Table 19: Health economics and resource-use benchmark instances.

Dataset	Structure	Task focus	Boundary
UCI Diabetes 130-US Hospitals	101,766 encounters, 50 columns; the department-split workbook is used only as a sheet-navigation variant.	Readmission proxy, medication process, utilization, coded variables, missingness.	Encounter-level data with limited longitudinal structure.
NHANES 2017–2018 subset	10 linked XPT/CSV tables with 3,036–19,643 rows per table, linked by SEQN .	Survey cohort construction, labs, medications, conditions, demographic covariates.	Cross-sectional survey data, not EHR.
TCGA-LIHC clinical metadata	377 cases, 137 clinical metadata columns.	HCC metadata, vital status, treatment fields, follow-up proxy fields.	Cancer registry-like clinical metadata, not routine-care EHR.

Appendix

Table 20: Public-RWS external validation datasets. These datasets test cross-dataset transfer and contract diagnostics, not additional full HCC-style longitudinal RWS validity.

Dataset	Objective	Contract stressor
UCI Diabetes	Readmission proxy profiling	Outcome binding, coded variables, missingness, conservative claim scope.
UCI Diabetes	Medication and utilization analysis	Exposure/proxy roles, subgroup aggregation, artifact traceability.
NHANES	Diabetes/cardiometabolic cohort analysis	Multi-table joins by SEQN , lab-defined variables, survey-table grounding.
NHANES	Medication and condition association analysis	Questionnaire, laboratory, and prescription-medication grounding under a cross-sectional claim policy.
TCGA-LIHC	HCC case and vital-status profiling	Domain transfer, oncology metadata grounding, missingness disclosure.
TCGA-LIHC	Treatment/follow-up metadata analysis	Sparse fields, proxy endpoint handling, limitation reporting.

Appendix

Table 21: Task objectives in Public-RWS-18. Each objective is evaluated under low-, medium-, and high-specificity prompts.

Dataset	Allowed claims	Forbidden claims
UCI Diabetes	Descriptive and readmission-proxy association claims.	Causal treatment effect, safety conclusion, longitudinal endpoint claim.
NHANES	Descriptive and cross-sectional association claims.	Temporal effect, incidence claim, causal effect.
TCGA-LIHC	Descriptive oncology metadata and follow-up-proxy claims.	Treatment efficacy, causal survival effect, routine-care EHR claim.

Appendix

Table 22: Dataset-specific claim policies for Public-RWS-18.

Prefix	#Fields	#Rows	Sheet Description	Differential field examples
SYSVAR	2	27	System variables and metadata.	Variable , Description
DATA_RANGE	2	4	Data range / center mapping and allocation notes.	Center , RangeCode
ECRF	8	54	eCRF form catalog and design attributes.	FormCode , FormName , FormType
CPH_FT	22	11614	Header/footer records for visit-/form-level.	VISTNM , SITNO , SUBNO
CO	21	1296	Study cover page and subject identification fields.	SITENO , SUBJNO , SUBINI
DS	19	1296	Informed consent date and related tracking fields.	DSDAT

Appendix

Table 23: Standardized worksheet inventory in the Excel dataset.

Symbol	Serialized key	Contents	Static admissibility checks
$D$	study_design	Study type, estimand boundary, and whether the requested output is descriptive, associative, or inferential.	Must be compatible with available tables, observed design, and claim policy.
$G$	analysis_grain	Subject-, visit-, encounter-, field-, or record-level analysis unit.	Must match key multiplicities and downstream aggregation.
$J$	join_graph	Tables, join keys, cardinalities, and permitted join paths.	All tables and keys must exist in $\Phi_{L,S}$ ; declared cardinality must match profiled relationships.
$K$	entity_keys	Subject, visit, encounter, treatment, and artifact identifiers.	Required keys must be present and nondegenerate under the declared grain.
$T$	temporal_anchors	Index date, event dates, censoring dates, windows, and ordering rules.	Date fields must be parseable or declared as partial/proxy dates; impossible orderings are disallowed.
$V$	variable_roles	Exposure, outcome, censoring, covariate, subgroup, proxy, and reporting variables.	Clinical roles must be supported by $\Phi_{L,K}$ , a validated proxy flag, or a recorded assumption.

Appendix

Table 24: Machine-readable fields in the RWS-IR and static checks before execution.

Violation type	Trigger	Allowed repair
Missing schema reference	A table, field, or key in $Z_{0}$ is absent from $\Phi_{L,S}$ .	Replace with a profiled field, remove the variable, or declare infeasibility for Goal+Method .
Unsupported semantic role	Exposure, endpoint, censoring, or confounder role is not validated in $\Phi_{L,K}$ .	Use a validated proxy with limitation, downgrade to descriptive support, or stop.
Invalid join or grain	Join cardinality or aggregation conflicts with the declared analysis grain.	Change grain, revise the join graph, or aggregate before merging.
Invalid temporal anchor	Index, event, censoring, or follow-up dates are missing, partial, nonparseable, or ordered incorrectly.	Use a supported anchor, restrict to analyzable records, or report feasibility only.
Method infeasibility	The requested method requires unsupported design assumptions, insufficient events, or unavailable covariates.	Switch to a supported fallback method or emit an infeasibility statement.
Claim-policy excess	$\Pi_{q}$ allows causal, safety-critical, economic-cost, or generalization claims unsupported by the data.	Restrict $\Pi_{q}$ to descriptive, feasibility, or proxy-association claims.

Appendix

Table 25: Static violation types emitted by IRCheckstatic .

Validator	Check	Acceptance criterion
Entry-point validator	Runs analysis.py or main.py in the sandbox.	Exit code is zero and no failure sentinel such as TASK_FAILED appears.
Schema validator	Parses code and generated outputs for table, field, and key usage.	Every referenced table, field, join key, and output column is traceable to $\Phi_{L,S}$ or a derived artifact.
Grain and join validator	Checks duplicate inflation, one-to-many joins, and final denominator consistency.	Output denominators match the declared analysis grain or disclose aggregation rules.
Temporal validator	Recomputes index, event, censoring, and follow-up windows.	Invalid date strings and impossible orderings are filtered, logged, or reported as limitations.
Method validator	Checks cohort filters, variables, denominators, covariates, and statistical choices against $Z.M$ .	Method matches the admitted RWS-IR or a declared fallback.
Artifact validator	Checks result and figure manifests, file existence, and generated figure formats.	Required tables, figures, manifests, and logs exist and are nonempty when applicable.

Appendix

Table 26: Execution validators used before accepting an artifact bundle.

Claim type	Allowed support	Examples of blocked wording
Descriptive	Validated table, count, summary statistic, figure, or profile entry.	Claims that imply generalization beyond the analyzed cohort.
Feasibility or data-quality	Profile entries, missingness summaries, temporal-validity logs, or execution artifacts.	Treating missingness or feasibility as clinical outcome evidence.
Proxy association	Valid proxy flag, artifact support, and explicit limitation.	Causal treatment-effect language or safety conclusion language.
Causal or comparative effect	Requires design assumptions, adjustment plan, and method support declared in $Z$ .	Causal language when only descriptive or proxy support is available.
Safety-critical or economic-cost	Requires detailed safety records or direct cost/billing fields.	Detailed AE/SAE term claims when only screening pages exist; cost-effectiveness claims without cost variables.

Appendix

Table 27: Claim types used by the certificate gate.

State role	Stored artifacts	Example contents	Accessed by
Execution trace	exec_logs.json , exec_status.json , opencode_live.log , violation_set.json , audit records.	execution traces, contract violations, validation outcomes, disclosed assumptions.	violation-guided execution, report generator, frontend status view.
Working context	excel_info.json , rws_ir.json , execution_spec.md , selected sheet payloads.	selected sheets, required fields, join keys, variable roles, temporal anchors, method specification, claim policy.	IR compiler, execution workspace, report context builder.
Profile semantics	$\Phi^{H,K}$ , $\Phi^{L,K}$ , ontology mappings, reporting prompts.	exposure/outcome/covariate roles, plausibility checks, clinical terminology, section semantics.	IR compiler, code generator, report DAG nodes.
Certified outputs	result manifests, figure manifests, evidence certificates, report sections, disclosed assumptions.	validated tables and figures, claim traces, support levels, limitations, section drafts.	claim checker, report DAG, final report renderer.
Operational policy	IR compilation, profiling, coding, figure, validation, and report-generation policies.	coding rules, artifact schemas, figure manifest schema, section templates, validation gates, claim policies.	execution workspace, report builder.

Appendix

Table 28: Operational mapping between profile-grounded state roles and implementation artifacts.

Artifact	Typical file name	Evaluation use
RWS-IR	rws_ir.json	PF, method validation, temporal and grain checks, claim policy.
Executable entry point	analysis.py or main.py	XE, CQ, artifact generation, code-region traces.
Execution logs	exec_logs.json , exec_status.json	Runtime pass/fail, violation localization, evidence reconstruction.
Results manifest	results_manifest.json	Result-table existence, denominators, generated statistics.
Figure manifest	figure_manifest.json	Figure existence, caption/axis metadata, chart-fidelity checks.
Violation set	violation_set.json	VRR and repair diagnostics.

Appendix

Table 29: Required artifact-bundle schema.

Harness component	Concrete artifact	Purpose
Workspace $W$	Isolated job directory with generated scripts, plots, logs, and report fragments.	Keeps one RWS task reproducible and separates generated files from source code.
Context $X$	rws_ir.json , excel_info.json , and execution_spec.md .	Supplies objective, selected sheets, required fields, join keys, temporal anchors, method choices, and claim policy.
Policies $\Pi$	Local coding instructions, figure policy, publication_plotting.py , and artifact schema.	Converts profile and reporting requirements into file-level rules that the coding agent must follow.
Tools $\mathcal{T}$	File editing, Python execution, local skill use, and sandbox validation.	Allows edit-run-fix behavior inside the controlled workspace.
Validators $\mathcal{V}$	Entry-point execution, TASK_FAILED sentinel check, figure_manifest.json , and PNG/PDF/SVG existence checks.	Turns soft generation instructions into completion gates for executable artifacts.
Traces $L$	opencode_live.log , exec_logs.json , exec_status.json , and violation_set.json .	Records evidence for violation-guided repair, failure reporting, and later report grounding.

Appendix

Table 30: Concrete artifacts used by the profile-constrained coding harness.

Symbol	Meaning
$\mathrm{prod}(t)$	Artifacts produced by task $t$ , such as tables, figures, statistics, or text.
$\mathrm{req}(t)$	Prerequisites used by task $t$ , such as variables, cohorts, temporal anchors, or method assumptions.

Appendix

Table 31: Task metadata used in the reporting DAG.

Signal	Definition
$S_{\mathrm{var}}$	Variable availability in the analysis-ready cohort, with missingness penalties. One instantiation is $\frac{1}{\|\mathcal{V}\|}\sum_{v\in\mathcal{V}}\max(0,1-\mathrm{miss}(v))$ .
$S_{\mathrm{temp}}$	Temporal validity, defined as $1-r_{\mathrm{invalid}}$ , where $r_{\mathrm{invalid}}$ is the fraction of missing, invalid, or impossible temporal prerequisites.
$S_{\mathrm{size}}$	Sample adequacy, defined as $\min(1,N_{\mathrm{eff}}/30)$ after filtering and joins.
$S_{\mathrm{cons}}$	Method-design consistency, with partial credit only when execution is feasible under recorded assumptions.

Appendix

Table 32: Support signals for reporting-task prioritization.

Panel of Figure 3 — Figure 3: Instantiated reporting DAG for the recurrence-free survival task. Solid edges denote feasible artifact dependencies and show selected support scores; the dashed edge records the blocked AE-detail analysis, which is routed to limitations rather than promoted as a safety result.

Node	Task	Required inputs	Produced artifacts	Report role
T0	Objective binding	User prompt, heterogeneous profile, report schema	Objective, endpoint definition, required section plan	Study overview
T1	Cohort and attrition	SG.Subject_Id , SG.SGDAT	Analysis cohort, attrition table, index-date assumptions	Overview / Attrition
T2	Baseline covariates	T1, DM , BCLC , CP , ECOG	Table 1, missingness table, covariate map	Baseline characteristics
T3	Endpoint and censoring	T1, VT.PADHDAT , VT.PADRDAT , VT.VTDAT1 , VT.VTDAT2	RFS dataset, event/censor flags, temporal validity log	Primary analysis
T4	Exposure matrix	T1, TACE , TART , IMU , PATE	Treatment exposure matrix, dated exposure availability	Primary / Subgroup
T5	Method feasibility gate	T2, T3, T4, procedural policy	KM feasibility decision and caveats	Methods

Appendix

Table 33: Instantiated report-task DAG for the recurrence-free survival benchmark instance. Each node declares required inputs and produced artifacts. Edges encode artifact prerequisites; infeasible edges are not selected and instead propagate limitations.

Edge	Dependency rationale	$\mathbf{s_{ij}}$	$\mathbf{c_{ij}}$
T0 $\rightarrow$ T1	Cohort construction requires the objective, index date, and output contract.	1	1.00
T1 $\rightarrow$ T2	Baseline summaries must be computed within the index cohort; BCLC and Child–Pugh coverage is high, while age missingness must be disclosed.	1	0.89
T1 $\rightarrow$ T3	RFS endpoints and censoring must be aligned to surgery date; 1,215 subjects have usable follow-up and one temporal anomaly is flagged.	1	0.98
T1 $\rightarrow$ T4	Treatment exposure grouping must be aligned to the index cohort; treatment proxies cover 1,104 subjects but dated exposures are incomplete.	1	0.85
T5 $\rightarrow$ T6	The primary RFS analysis is selected only after the method gate passes; the combined gate has $S_{\mathrm{var}}=0.828$ , $S_{\mathrm{temp}}=0.999$ , $S_{\mathrm{size}}=1.000$ , and $S_{\mathrm{cons}}=1.000$ .	1	0.931
T8 $\rightarrow$ detailed AE profiling	Detailed AE/SAE profiling requires non-empty detail tables; both detail tables contain no valid records.	0	0.00

Appendix

Table 34: Selected dependency edges and support scores for the instantiated DAG. The feasibility gate

s_{ij}

blocks hard-infeasible dependencies. The support score

c_{ij}

summarizes deterministic evidence from variable availability, temporal validity, sample adequacy, and method-design consistency.

Section	Expected content
Study Overview	Cohort definition and objective statement.
Attrition Analysis	Patient disposition and eligibility filtering steps.
Baseline Characteristics	Demographics, clinical history, baseline vitals, and baseline labs.
Primary Analysis	Main efficacy or exposure-outcome results, such as KM curves or hazard ratios.
Safety Profiling	Adverse-event screening summaries and available safety proxies.
Sensitivity/Subgroup Analysis	Checks across predefined strata.

Appendix

Table 35: Required RWS report sections used by the RS metric.

Method family	Run protocol	Scoring rule
ReAct and ToolLLM	Agentic tool-use baselines are asked to inspect the tables, write executable analysis code, generate artifacts, and produce a report under the same prompt.	Deterministic validators execute the final entry point when provided; reports are checked post hoc for evidence support.
TableGPT-Agent and ChatExcel-Max	Table-analysis agents are asked to generate the requested analysis and report.	If no executable artifact bundle is produced, execution-dependent metrics are N/A; report and claim metrics are computed when possible.
Gemini-3-Pro direct baseline	Gemini-3-Pro receives the task and data-access instructions and is asked to return code, artifacts, and report content.	Generated code is executed in the same sandbox and evaluated by the same artifact and claim validators.
TableGPT2	The table model is evaluated on table-analysis outputs and executable artifacts when available.	Metrics requiring a complete bundle are N/A when the method does not emit the required artifact.
Claude Code and Codex	Native coding-agent baselines receive the benchmark prompt, data-access path, artifact contract, and sandbox budget through their repository editing interfaces. They may create, edit, execute, and repair files within the job workspace.	The final workspace is re-executed from a clean scorer sandbox. Execution, artifact, clinical/statistical language, and claim-trace metrics are computed post hoc from the submitted files and report.
ProCARE	Full profile–IR–execution–certificate–DAG workflow.	All deterministic and judge metrics are computed on the accepted artifact bundle.

Appendix

Table 36: Baseline adaptation and scoring rules.

Condition	Information given to model	Output and repair control	Purpose
Profile/template baseline scaffold	Task prompt, raw data access, $\Phi$ summary, table/field dictionary, join keys, temporal-role candidates, and artifact template.	One initial generation; no validator feedback loop beyond final scoring.	Controls for missing schema and output-format information.
Profile/template/repair baseline scaffold	Same as above.	Three edit-run-repair turns with standardized validator feedback and identical execution budget.	Controls for unequal repair opportunity.
Direct Gemini-3-Pro + validator interface	Gemini-3-Pro receives $\Phi$ , artifact template, sandbox logs, and validator feedback.	Three repair turns; no typed IR compiler or certificate admission gate.	Controls for deterministic validation feedback without ProCARE planning.
Direct Gemini-3-Pro + study-spec schema	Gemini-3-Pro receives $\Phi$ and must emit study_spec.json before writing code and report.	No ProCARE IR compiler; validator feedback is limited to schema/artifact execution errors.	Controls for evaluator-compatible study-spec output.
Direct Gemini-3-Pro + claim-trace schema	Gemini-3-Pro receives $\Phi$ , artifact template, and claim-trace file schema.	No certificate admission gate or certified DAG ordering.	Controls for evaluator-compatible claim-trace output.
ProCARE reference	Full profile, IR compiler, violation-guided execution, evidence certificates, and certified report DAG.	Same three repair turns and final checker.	Reports the integrated contract workflow for comparison.

Appendix

Table 37: Inputs and controls used in the scaffolded-baseline fairness evaluation. These are baseline adaptations, not ProCARE component ablations.

Baseline condition	Performance								Contract diagnostics
	GA $\uparrow$	PF $\uparrow$	XE $\uparrow$	CQ $\uparrow$	VC $\uparrow$	CF $\uparrow$	RS $\uparrow$	SL $\uparrow$	VRR $\uparrow$	CTC $\uparrow$	UCR $\downarrow$	CLV $\downarrow$
ReAct + fairness scaffold	0.82	0.79	0.74	0.69	0.62	0.65	0.75	0.70	0.62	0.70	0.13	0.10
ToolLLM + fairness scaffold	0.79	0.80	0.68	0.65	0.58	0.62	0.72	0.67	0.58	0.66	0.15	0.12
Direct Gemini-3-Pro + profile/template	0.77	0.73	0.66	0.62	0.54	0.58	0.73	0.65	0.31	0.55	0.20	0.15
Direct Gemini-3-Pro + validator repair	0.80	0.78	0.79	0.72	0.60	0.64	0.75	0.69	0.68	0.65	0.15	0.11
Direct Gemini-3-Pro + study-spec schema	0.82	0.82	0.70	0.68	0.57	0.62	0.71	0.68	0.43	0.52	0.21	0.16

Appendix

Table 38: Full metric results for fairness-controlled scaffolded baselines. Higher is better except UCR and CLV. Rows are baseline adaptations, not ProCARE ablations; values are final checker outputs.

Variant	Removed component	What remains fixed
w/o profile $\Phi$	The model does not receive the heterogeneous profile as a contract object. It may inspect raw table metadata and generated summaries, but generation lacks profile-grounded repair hints.	Prompt, budget, execution sandbox, reporting request, and post-hoc evaluator.
w/o RWS-IR	The model writes code and a report without producing the typed RWS-IR.	Raw data access, profile summaries when available, execution sandbox, and post-hoc claim diagnostics. PF is N/A because no typed IR exists.
w/o violation-guided repair	Runtime validators detect violations, but localized violation sets are not fed back into the next edit.	Profile, RWS-IR, execution sandbox, artifact validators, certificates, and report policy.
w/o evidence certificates	The report is generated without explicit claim certificates and without the claim-admission gate.	Profile, RWS-IR, execution repair, artifacts, and DAG section ordering. CTC for emitted explicit certificates is zero; post-hoc support can still be inspected separately.
w/o certified DAG ordering	Evidence certificates remain available, but report sections are generated in a fixed or unconstrained order without dependency-aware traversal.	Profile, RWS-IR, execution repair, artifact validators, and claim admission.
Full ProCARE	No contract component is removed.	Complete profile–IR–repair–certificate–DAG workflow.

Appendix

Table 39: Ablation definitions for contract mechanisms.

Role	Paper-level model identifier	Access mode	Use in evaluation
Backbone	GPT-5.2	Proprietary API	Backbone-sensitivity run under the fixed ProCARE workflow.
Backbone	GPT-4 Turbo	Proprietary API	Backbone-sensitivity run under the fixed ProCARE workflow.
Backbone	Claude Opus 4.5	Proprietary API	Backbone-sensitivity run under the fixed ProCARE workflow.
Backbone	Claude Sonnet 4.5	Proprietary API	Backbone-sensitivity run under the fixed ProCARE workflow.
Backbone	Claude-3 Opus	Proprietary API	Backbone-sensitivity run under the fixed ProCARE workflow.
Backbone, direct baseline, and scaffolded control	Gemini-3-Pro	Proprietary API	Default ProCARE backbone; also used for the direct baseline and all Direct Gemini-3-Pro scaffolded fairness-control rows.

Appendix

Table 40: Model access, version metadata, and evaluation roles.

Path	Contents
paper/	Manuscript, supplement, bibliography, NeurIPS style file, build script, and paper figures.
code/backend/	Profiling, planning, code-generation, execution, report-generation, and API services.
code/backend/tests/	Local tests for profile construction, planning, code execution, OpenCode integration, and report generation.
code/frontend/	Interface for project setup, profile inspection, RWS-IR review, execution monitoring, and report review.
code/output/knowledge_base/profiles/	Cached profile artifacts used by the implementation.
rws_hcc_benchmark/	Fully de-identified HCC benchmark tables, data dictionary, checksums, task metadata, and release terms for reproducing the 36 primary tasks.

Appendix

Table 41: Repository components used for reproduction.

Panel of Figure 4 — Figure 4: ProCARE interface states for project setup, profile inspection, RWS-IR compilation, and coding execution.

Panel of Figure 5 — Figure 5: Report-facing interface states for reviewing generated artifacts, evidence links, and section-wise report content.

Metric	Type	Scoring rule
RWS-IR Feasibility (PF)	Deterministic + judge	Validates the RWS-IR against $\Phi^{S}$ and penalizes non-existent tables, columns, invalid joins, unsupported temporal anchors, or incompatible analysis grains. Residual ambiguity is checked by the judge.
Code Executability (XE)	Deterministic	Sandbox execution success rate, $n_{\mathrm{pass}}/N$ ; runtime errors and invalid output files count as failures.
Violation Resolution Rate (VRR)	Deterministic	Fraction of detected execution-time contract violations resolved before the accepted run.
Report Structure (RS)	Deterministic	Fraction of required report sections in Table 35 covered with non-trivial content.
Claim Trace Coverage (CTC)	Deterministic	Fraction of final factual claims linked to valid evidence certificates.
Unsupported Claim Rate (UCR)	Deterministic	Fraction of final factual claims without admissible evidence support.

Appendix

Table 42: Metrics used in the appendix evaluation protocol.

Column	Allowed values or format	Definition
review_id	stable string	Unique row identifier.
benchmark_id	HCC-001–HCC-036	Primary benchmark instance identifier.
objective	cohort, safety-proxy, efficacy, resource-use	Benchmark objective family.
prompt_level	goal, goal+approach, goal+method	Prompt-specificity level.
method_id	submitted method identifier	Method label masked during review and restored only for analysis.
stage	rws_ir, code, visualization, reporting	Workflow stage being scored.

Appendix

Table 43: Raw expert-rating data format used before aggregation. One row stores one reviewer–instance–stage score.

Stage	Eligible reviewers	Primary review focus	Raw records
RWS-IR	C1–C3, S1–S2	Clinical goal, endpoint validity, data linkage, analysis grain, and statistical design feasibility.	180
Code	S1–S2	Reproducibility, executable implementation, statistical method use, artifact writing, and result consistency.	72
Visualization	C1–C3, S1–S2	Clinical interpretability, sample-size labeling, figure-result agreement, and avoidance of unsupported visual claims.	180
Reporting	C1–C3, S1–S2	Clinical language, limitation disclosure, statistical wording, and claim-to-evidence support.	180

Appendix

Table 44: Expert panel responsibilities and raw scoring volume.

Stage	Instance means	Raw records	Mean $\pm$ SD	95% CI	Median [IQR]
RWS-IR	36	180	4.38 $\pm$ 0.48	4.22–4.54	4.33 [4.00, 4.67]
Code	36	72	4.42 $\pm$ 0.21	4.35–4.49	4.42 [4.25, 4.58]
Visualization	36	180	4.50 $\pm$ 0.53	4.32–4.68	4.67 [4.00, 5.00]
Reporting	36	180	4.13 $\pm$ 0.64	3.91–4.35	4.17 [3.67, 4.67]

Appendix

Table 45: Stage-level expert score summary on the 36 primary HCC benchmark instances. CI is the 95% confidence interval for the instance-level mean.

Stage	Panel	Statistic	Estimate	95% CI	Interpretation
RWS-IR	C1–C3, S1–S2	ICC(2,k)	0.84	0.72–0.91	Strong agreement on specification validity.
Code	S1–S2	$\kappa_{w}$	0.76	0.61–0.88	Substantial agreement on reproducibility and statistical implementation.
Visualization	C1–C3, S1–S2	ICC(2,k)	0.79	0.66–0.88	Strong agreement on figure clarity and consistency.
Reporting	C1–C3, S1–S2	ICC(2,k)	0.77	0.63–0.87	Substantial-to-strong agreement on language and claim scope.

Appendix

Table 46: Inter-rater reliability for the human expert panel. Stages scored by five reviewers use two-way random-effects ICC for averaged ratings; the two-reviewer code stage uses weighted Cohen’s

\kappa_{w}

Stage	Expert aggregation	Automated score	$\rho$	95% CI	$p$	$q$
RWS-IR	Mean of C1–C3, S1–S2	$(GA+PF)/2$	0.82	0.67–0.90	$<0.001$	$<0.001$
Code	Mean of S1–S2	$CQ$	0.74	0.54–0.86	$<0.001$	$<0.001$
Visualization	Mean of C1–C3, S1–S2	$(VC+CF)/2$	0.75	0.55–0.86	$<0.001$	$<0.001$
Reporting	Mean of C1–C3, S1–S2	$SL$	0.71	0.49–0.84	$<0.001$	$<0.001$

Appendix

Table 47: Correlation between automated metrics and aggregated expert scores.

q

is the Benjamini–Hochberg adjusted

p

value across the four stage-wise correlations.

Metric	Score 5	Score 3	Score 1
Goal Alignment	Addresses the clinical objective, target population, and endpoints with explicit limitation disclosure.	Partially addresses the task with minor errors in inclusion criteria or endpoints.	Gives an irrelevant analysis or misunderstands the clinical goal.
Code Quality	Uses sound pandas or SQL logic, handles missing data and types correctly, and keeps the implementation modular.	Runs but remains fragile, often because of hardcoded values or inefficient joins.	Uses an incorrect method, applies the wrong statistical test, or contradicts the plan.
Visual Clarity	Produces figures with clear axes, legends, labels, and titles that support the reported results.	Produces legible charts with missing units, overlapping labels, or default styling.	Produces misleading, illegible, or unrelated visualizations.
Scientific Language	Uses clinical language with appropriate hedging and clear limitation disclosure.	Uses a general scientific tone but has verbose phrasing or imprecise medical terminology.	Uses casual language or makes unsupported claims.

Appendix

Table 48: Semantic rubric anchors for LLM judges.

Scientific Workflow Automation

Scientific workflow automation uses software, AI agents, and laboratory systems to plan, execute, validate, and reproduce research tasks.

See all 161 papers

Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases
Mar 24, 2026Dubai Li, Yuxiang He, Yan Hu +2LLM Agent Evaluation AI Agent Benchmarks
Zhejiang University, Hangzhou, China
FastOMOP: A Foundational Architecture for Reliable Agentic Real-World Evidence Generation on OMOP CDM data
Apr 27, 2026Niko Moeller-Grell, Shihao Shenzhang, Zhangshu Joshua Jiang +2Multi-Agent Orchestration Healthcare
EPSRC DRIVE-Health, Department of Biostatistics & Health Informatics, King’s College London, UK. · Lancashire Teaching Hospitals NHS Foundation Trust, UK. · Institute for Health Informatics, University College London, UK. +5
CLAIRE: A Schema-Grounded Hybrid Workflow for Healthcare Administrative Form Completion
Sep 26, 2026Garapati Keerthana, Manik GuptaWorkflow Automation Human-in-the-Loop AI
Department of Computer Science and Information Systems, Birla Institute of Technology and Science, Pilani, Hyderabad, India

Mar 24, 2026J/K move · Enter open · S save

Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases

Dubai Li, Yuxiang He, Yan Hu, Yu Tian+1

Zhejiang University, Hangzhou, China

LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Observational studies can yield clinically actionable evidence at scale, but executing them on real-world databases is open-ended and requires coherent decisions across cohort construction, analysis, and reporting. Prior evaluations of LLM agents emphasize isolated steps or single answers, missing the integrity and internal structure of the resulting evidence bundle. To address this gap, we introduce RWE-bench, a benchmark grounded in MIMIC-IV and derived from peer-reviewed observational studies. Each task provides the corresponding study protocol as the reference standard, requiring agents to execute experiments in a real database and iteratively generate tree-structured evidence bundles. We evaluate six LLMs (three open-source, three closed-source) under three agent scaffolds using both question-level correctness and end-to-end task metrics. Across 162 tasks, task success is low: the best agent reaches 39.9%, and the best open-source model reaches 30.4%. Agent scaffolds also matter substantially, causing over 30% variation in performance metrics. Furthermore, we implement an automated cohort evaluation method to rapidly localize errors and identify agent failure modes. Overall, the results highlight persistent limitations in agents' ability to produce end-to-end evidence bundles, and efficient validation remains an important direction for future work. Code and data are available at https://github.com/somewordstoolate/RWE-bench.

Abstract

Scientific Workflow Automation

Explore similar work

Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases

FastOMOP: A Foundational Architecture for Reliable Agentic Real-World Evidence Generation on OMOP CDM data

CLAIRE: A Schema-Grounded Hybrid Workflow for Healthcare Administrative Form Completion