Kurate: Scalable Scientific Quality Analysis

Matthew J. Vowels; Jamie Cummins

cs.CLOct 5, 2026

Kurate: Scalable Scientific Quality Analysis

Authors: Matthew J. Vowels, Jamie Cummins

Organizations: Kivira Health, U.S.A. · University of Bern, Bern, Switzerland

Abstract

Scientific search systems can find papers that are relevant to a question, but they generally do not assess the quality of the evidence that those papers provide. We present Kurate, a system that uses large language models (LLMs) to assess the quality of published studies. Kurate uses both the paper and its related documents (e.g., the study's trial registration and protocol), and links each of its judgments to the passage of text on which that judgment is based. We applied Kurate to a corpus of 4,347 papers (3,913 of which report randomized trials) and scored each paper on 8 dimensions of study design and reporting: specifically, statistical power, causal identification, preregistration, selective reporting, measurement validity, analysis prespecification, reporting transparency, and conflict of interest and funding. Across the corpus, we found that papers most often exhibited issues with statistical power, selective reporting, and analysis prespecification, although average quality differed between clinical areas. When compared against expert annotations of 60 held-out clinical-trial documents, the information Kurate extracted matched the expert label in 221/242 protocol scorepoints and 294/370 results-publication scorepoints, with AC1 0.94 and 0.81, respectively. Using a well-reputed, high quality clinical trial as a worked example, we show how a single paper's overall grade breaks down into separate judgments, with each linked to specific evidence from the trial's registration, protocol, and published report. Together, these results show that large-scale quality assessment of this kind is feasible, and that it can be used to address meta-scientific research questions.

Figures & tables

Figure 2: The Kurate pipeline. Stage names in bold correspond to the boxes in the figure. Ingestion/source capture collects the published report of a study together with its linked documents, and extracts the bibliographic information, full text, tables, figures, registry and protocol identifiers, funding statements, and a record of any problems encountered during parsing. Deterministic Knowledge Graph uses fixed rules (i.e., without LLMs) to build a knowledge graph (KG), which is a database of linked records connecting papers, related documents, authors, funders, registrations, text passages, tables, and figures. KG swarm semantic extraction uses a set of LLM-based agents to extract information which requires interpretation of the text. This includes which outcomes were primary or secondary, what the treatment and comparison conditions consisted of, which disorders were studied, evidence on sample size and statistical power, the flow of participants through the study, whether the paper agrees with its registration and protocol, and information on funding and conflicts of interest. Rating scores this information against Kurate’s quality dimensions (Appendix C ). This produces a score for each dimension (positive where the evidence supports the study’s quality, and negative where it indicates a problem), an overall score scaled to lie between 0 and 1 (the normalized score), and a letter grade. Hard eval checks for serious problems which can rule out the use of a paper as evidence regardless of its grade. Examples of such issues can include a document which is not an empirical report, the absence of randomization, conflicts between the registration and the published report, large amounts of missing data, and primary claims which the results do not support. Executive assessment/adjudication combines the scores, the results of these checks, and any uncertainties or caveats into a final decision about whether the paper should be used as evidence. Reports produces a report for each paper which shows the evidence, the document each piece of evidence came from, the scores, the caveats, and the reasons for the final decision. Derived Database surfaces updates the summary tables used for searching and for corpus-level analysis (e.g., summaries by topic, medication, and funder). Validation/promotion checks each paper’s record for completeness and consistency, and adds fully processed papers to the curated database used for browsing, searching, and meta-scientific analysis.

Figure 3: The evidence summary which Kurate returns for an example query about esketamine.

Figure 4: Mean normalized Kurate scores by publication era and clinical domain. Points are shown where a domain has at least five papers in that era; endpoint labels identify trajectories and latest observed-era scores.

Study design and population
Information Kurate extracts	Trial protocols	Results publications
Who was eligible to take part	30/30 (1.00)	—
What the intervention was	30/30 (1.00)	29/30 (0.97)
What it was compared against	30/30 (1.00)	25/30 (0.83)
How many participants	30/30 (1.00)	38/46 (0.83)
How the outcome was defined	24/30 (0.80)	26/29 (0.90)

Table 1: Accuracy of Kurate’s information extraction, compared with expert annotators and a published automated system. Upper block: for each reporting item, the number of held-out documents in which the value used by Kurate matched the expert label, with the proportion in parentheses. “—” indicates that the item is not annotated for that type of document (results publications do not state who was planned to be eligible, and protocols contain no findings to report). Lower block: Gwet’s AC1 ( Gwet, 2008 ) measures agreement after correcting for the agreement expected by chance. It is the measure reported by the benchmark’s authors, and is therefore the only measure for which human and baseline figures are available. The 95% CI row gives Kurate’s confidence interval for mean AC1. The benchmark reports the human and baseline figures only as overall means across both document types, so each figure appears in both columns. They are also means over all 119 benchmark questions, whereas Kurate’s figures are means over only the 9 (protocols) and 15 (results publications) questions for which Kurate extracts information. See Appendix A .

Figure 5: Number of papers by publication year in the current Kurate corpus among papers with a known publication year (4,344 of 4,347 papers).

Grade	Include	Flag	Retained	Exclude	Total
A	71	137	208	72	280
B	100	456	556	378	934
C	44	606	650	775	1,425
D	10	570	580	891	1,471
E	0	65	65	171	236
F	0	1	1	0	1

Table 2: Kurate quality grade by final evidence-use decision in the targeted 4,347-paper corpus. Counts are papers. Retained is the sum of include and flag ; exclude papers remain in the corpus but are not used as supporting evidence in downstream synthesis.

Domain	$N$	Mean score	A/B/C	Excluded
Overall corpus	4,347	0.564	60.7%	52.6%
Infectious disease / HIV	349	0.602	72.5%	51.3%
Neurology / pain	138	0.593	63.0%	44.9%
Cardiometabolic	1,122	0.592	68.2%	51.2%
Cancer	843	0.580	65.5%	63.0%
Resp. / allergy / immune	152	0.569	60.5%	52.0%

Table 3: Quality results for each clinical domain in the targeted 4,347-paper corpus, sorted from highest to lowest mean normalized score (range 0–1). Only domains with at least 25 papers are shown. A/B/C is the percentage of papers graded A, B, or C; Excluded is the percentage of papers for which Kurate’s evidence-use decision was “exclude”. All percentages describe Kurate’s outputs, not independently verified properties of the studies.

Dealbreaker criterion	Papers	Excluded papers
Control group	124	5.4%
Comparator validity	444	19.4%
Randomization	712	31.1%
Blinding and outcome susceptibility	398	17.4%
Analysis aligned with research question	183	8.0%
ITT or justified alternative	320	14.0%

Table 4: A breakdown for the papers which failed on the individual dealbreaker criteria in our current sample, specifically the 2,287 papers with a final evidence-use decision of exclude .

Dimension	Score	Planned or external evidence	Reported evidence	Kurate’s assessment
Statistical power	1.00	[registry] planned enrollment “9,250 participants”	[report] “88.7% power …20% effect”; “randomly assigned 9361 persons”	Strong support: the paper reports a specific power calculation, and the number of participants randomized exceeded the planned target.
Causal identification	0.67	[registry] intensive vs standard systolic blood pressure target groups; comparison of two active treatments; limited detail on blinding	[report] “We randomly assigned 9361 persons …” to “intensive” vs “standard” treatment	Partial support: the paper clearly reports random assignment, but evidence on allocation concealment (i.e., preventing those enrolling participants from knowing the upcoming assignment) and on blinding is not fully explicit in the paper.
Preregistration	1.00	[registry] first submitted 2010-09-20, before study start on 2010-10-01; NCT01206062	[report] “ClinicalTrials.gov number, NCT01206062”; the same trial is described in the article	Strong support: Kurate links the article to a registration which was submitted before the trial began, and verifies the dates.
Selective reporting	1.00	[registry] primary outcome “MI …ACS …Stroke …HF …CVD Death”; secondary outcome “All-cause Mortality”	[abstract] same composite primary outcome; “All-cause mortality was also significantly lower …”	Strong support: the main reported outcomes match the preregistered outcomes, and the early stopping of the trial is reported openly.
Measurement validity	1.00	[registry] primary outcome is a composite of major cardiovascular events	[report] “primary composite outcome was myocardial infarction …stroke …heart failure …”	Strong support: the outcome is a standard clinical measure and is directly relevant to the question the trial addresses.
Analysis prespecification	1.00	[protocol / registry] registered in advance; protocol available; planned “time to the first occurrence …” analysis using a Cox model (a standard model for time-to-event data) on randomized participants	[results] “stopped early …owing to” lower primary-event rates; the change from the plan is stated explicitly	Strong support: Kurate finds an analysis plan specified in advance, and the major change to that plan (stopping the trial early) is reported explicitly rather than made without disclosure.

Table 5: Kurate’s assessment of the SPRINT trial on each scored dimension. Scores are normalized to lie between 0 and 1. The two middle columns show the evidence Kurate compared, with the source document given in square brackets: what was planned or expected (mainly from the registration or protocol), and what was reported.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Protocols (SPIRIT)		Results (CONSORT)
	Train	Held-out	Train	Held-out
Documents	70	30	70	30
Scored items $n$	566	242	906	370
Questions $k$	9	9	15	15
Candidate extraction	0.921	0.934	0.846	0.838
95% CI [ Wilson, 1927 ]	—	[0.895, 0.959]	—	[0.797, 0.872]

Appendix

Table 6: Full results for the training and held-out sets. The held-out columns use the benchmark’s own test and validation sets combined, scored once for each document type.

k

is the number of benchmark questions for which Kurate has a corresponding field. AC1 is the mean (

\pm

SD) of the coefficients calculated separately for each question, which matches the method used by [ Jiang et al., 2026 ] .

Benchmark item	Kurate field	Candidate extraction	Final value
text10a_participants_inclusion	eligibility criteria	30/30	30/30
text11a_intervention_description	intervention name	30/30	30/30
text11a_intervention_description	comparator name	30/30	30/30
text14b_sample_calculation	sample size	30/30	30/30
text20a_statistical_methods_outc	planned analysis methods	29/29	29/29
text31d_sharing_data	data-sharing statement	5/5	5/5

Appendix

Table 7: Protocols, held-out set (

n=242

Benchmark item	Kurate field	Candidate extraction	Final value
text14b_sample_calculation	sample size	23/23	20/23
text37a_analysis_numbers	sample size	23/23	18/23
text38a_outcome_results	$p$ -value	22/22	21/22
text11a_intervention_description	intervention name	29/30	29/30
text38a_outcome_results	summary statistic	28/30	23/30
text12a_outcomes_definitions	endpoint name	26/29	26/29

Appendix

Table 8: Results publications, held-out set (

n=370

Figure 6: Screenshot of the summary section at the top of Kurate’s report for the SPRINT trial.

Dimension	Short description	Range	Weight
Statistical power	Whether the study was large enough for its main claim, given the planned and analyzed sample sizes.	$[-3,3]$	1.35
Causal identification	Whether the design supports a causal comparison, including randomization, allocation safeguards, comparator choice, and blinding where feasible.	$[-3,3]$	1.35
Preregistration	Whether the main claim was specified before the relevant data were collected or analyzed.	$[-3,3]$	1.20
Selective reporting	Whether the reported outcomes, analyses, and emphasis match the plan made in advance.	$[-3,3]$	1.20
Measurement validity	Whether the outcomes are appropriate for the construct, population, intervention, and timepoint.	$[-3,3]$	1.00
Analysis prespecification	Whether the main analysis was planned in advance and answers the stated research question.	$[-3,3]$	1.00

Appendix

Table 9: Scored quality dimensions used to assign the Kurate grade.

Scoring element	Mapping	Interpretation
Dimension score	$-3$ to $+3$ for most active dimensions, with conflict/funding using a narrower range.	Negative scores are penalties for fundamental flaws or missing evidence, with zero as neutral and positive scores indicating support.
Evidence-grade score	Weighted dimension total normalized to 0–1.	Produces the continuous normalized score reported for papers and cohorts.
A	normalized score $\geq 0.8333$	Strong evidence quality across the active dimensions.
B	normalized score $\geq 0.6667$ and $<0.8333$	Adequate-to-strong evidence quality with limited concerns.
C	normalized score $\geq 0.5000$ and $<0.6667$	Moderate evidence quality with notable limitations.
D	normalized score $\geq 0.3333$ and $<0.5000$	Limited evidence quality with substantial weaknesses.

Appendix

Table 10: Summary of Kurate’s scoring and grade mapping.

Hard dealbreaker	Short description
Control group	The study needs a usable comparison group for the claim being made. A causal claim cannot rest on a paper with no suitable control or comparison.
Comparator validity	The comparator must be clear and appropriate. A very weak, poorly described, or inappropriate comparator can make the claimed effect uninterpretable.
Randomization	Assignment should be random, or at least not presented as randomized when it was absent, quasi-random, unsupported, or compromised in a way that could bias treatment assignment.
Blinding and outcome susceptibility	Lack of blinding is most serious when the primary outcome is subjective or otherwise easy to bias. Lack of blinding is not automatically fatal when blinding is infeasible and the claim is limited accordingly.
Analysis aligned with research question	The model, comparator, outcome, timepoint, and analysis population should answer the stated research question. If they do not, the primary claim may not be supported.
ITT or justified alternative	The analysis should preserve the randomized comparison, usually by intention-to-treat (ITT), or give a credible reason for an alternative population. Unjustified completer-only, subgroup, or as-treated analyses can undermine the comparison.

Appendix

Table 11: Hard dealbreaker criteria that can flag or exclude a paper from evidence use.

Explore similar work

Jun 15, 2026cs.CL

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio

Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442 expert-curated meta-analyses from Nature Portfolio journals. Each entry pairs a research question with PI/ECO criteria, a retrieval corpus of 140k PubMed articles, verified positive studies, hard negatives that are topically similar but PI/ECO-ineligible, and complete search strategies and date bounds. Benchmarking twelve pipeline configurations (nine RAG variants and a protocol-driven agent) reveals a critical screening bottleneck: despite a retrieval ceiling of 90.9% recall at K=200, no system recovers more than 52.7% of ground-truth included literature. Current LLMs fail to reliably separate eligible studies from PI/ECO-failing distractors in pools of comparable topical relevance. Stage-attributed metrics capture where systems succeed and fail; a single end-to-end score does not.

Anzhe Xie, Weihang Su, Yujia Zhou +2

Tsinghua University Beijing, China

Meta Relevance

Ask PDF

Jul 13, 2026cs.CL

ResearchQA: Benchmarking Citation-Grounded Question-Answering on Scientific Papers

Large language models are increasingly used to assist scientific reading, but existing evaluation methods often fail to detect whether answers are supported by verifiable citations. We introduce ResearchQA, a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers spanning eight domains and four question types: lookup, comprehension, multi-hop, and adversarial. ResearchQA is designed for citation-grounded evaluation: it permits multiple valid supporting passages for a claim and rewards grounded refusal when the source paper does not support an answer. We evaluate eight leading closed- and open-weight models in a citation-grounded chat-with-paper setting using a deterministic citation matcher and an LLM-based rubric evaluator. Citation-based metrics separate systems more clearly than LLM-evaluator scores: section coverage and citation accuracy vary substantially across models, while evaluator scores remain tightly compressed. We further find that open-weight models approach the best closed-model citation accuracy while achieving 3 to 6 times lower per-example latency. We release the benchmark, evaluation harness, and evaluator prompt.

Saba Imran, Debanjum Singh Solanky

Khoj Inc.

Citations Scholar Information Database Question-Answer Pairs

Ask PDF

Jul 6, 2026cs.AI

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect. We introduce FirstResearch, a first-principles research-question formation framework for scientific LLM agents whose core artifact is a structured Research Question Certificate. The certificate records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule, making the proposed question inspectable before downstream execution. On ten LLM-agent research topics, FirstResearch outperforms controlled prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2 under a primary DeepSeek-blind-judge protocol. A Gemini-2.5-Flash independent-judge rescore of the same 40 baseline packages preserves the system-level ranking, with FirstResearch scoring 4.86/5 versus 4.38/5 for the strongest baseline and Pearson agreement of 0.865 on average score. A one-repeat ablation checkpoint further suggests that the certificate-centered core is the strongest component: certificate-only scoring reaches 4.90/5 under DeepSeek and 4.88/5 under Gemini, while removing certificates drops below 1/5 under both judges. These results are preliminary and use LLM judges rather than human domain experts, but they support a narrow scientific-discovery claim: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable. Code, prompts, saved outputs, and reproduction scripts are available at https://github.com/louiswang524/FirstResearch.

Yufeng Wang

Frontier Large Language Model Agents First-Principles Calculation

Ask PDF