SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews

Miguel Zabaleta; Baihan Lin

cs.AIAug 29, 2026

SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews

Authors: Miguel Zabaleta, Baihan Lin

Organizations: Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA

Abstract

Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We introduce SciLitBench, a multi-stage benchmark spanning title and abstract screening, full-text screening, and schema-guided data extraction, with 42,981 retrieved records, 1,012 full texts, and annotations for 888 included papers. Across 22 open-weight LLMs from six model families, explicit inclusion and exclusion criteria improve title and abstract screening $F_2$ by 28.8%, while researcher-authored rationales improve full-text screening by 15%. Data extraction reveals a different reliability regime: performance declines from 0.97 accuracy for publication year to 0.37 Jaccard overlap for computational approach, while the strongest models recover only 30% of annotated evaluation evidence and 25% of limitations. SciLitBench identifies a practical boundary between high-recall screening and evidence-complete extraction and provides a reproducible resource for evaluating LLM-assisted evidence synthesis.

Figures & tables

Benchmark	Domain	Stages	Long	Rationale	Size
SciLitBench (ours)	SR	TA+FT+DE	$\surd$	$\surd$	43k TA; 1k FT; 17k DE
SYNERGY [ 26 ]	SR	TA	$\times$	$\times$	169k recs; 26 SRs
Evidence Inference [ 27 ]	RCT	FT	$\surd$	$\surd$	7k q–doc inst.
EBM-NLP (PICO) [ 28 ]	RCT abs.	DE	$\times$	span	5k abs.
MS 2 [ 29 ]	Med SR	Summ.	$\surd$	$\times$	470k docs; 20k sums
BioASQ [ 30 ]	Biomed	IR+QA	snip	$\surd$	Yearly batches

Table 1: SciLitBench in context. SciLitBench uniquely spans three SR stages (TA/FT/DE) on long PDFs with released inclusion rationales, enabling LLM-centric evaluation. Abbreviations: SR (systematic review), TA (title and abstract screening), FT (full-text screening), DE (structured data extraction), RCT (randomized controlled trial), IR (basic information retrieval), QA (question answering), Summ. (summarization).

Figure 1: Construction of SciLitBench across title/abstract screening, full-text screening, and data extraction. Stage 1: Title and abstract (TA) screening. From 42,981 retrieved records, 2,000 were manually annotated and divided into development, few-shot, and evaluation subsets. Model and ensemble experiments were used to identify the highest-precision configuration achieving perfect recall on the evaluation split. This configuration was applied to the remaining 40,981 records, yielding 4,165 predicted positives for manual re-annotation and 36,816 predicted negatives retained as silver exclusions. Manual review of the predicted positives identified 1,767 additional inclusions and 2,398 exclusions; combined with the 52 inclusions from the initial annotated set, this produced 1,819 title/abstract inclusions. A random sample of 500 silver exclusions was additionally re-annotated to estimate false-negative contamination. Stage 2: Full-text (FT) screening. Full text was retrieved for 1,012 records carried forward from Stage 1. A manually annotated set of 200 papers was used for full-text model evaluation and model selection. The highest-precision configuration achieving perfect recall was then applied to the remaining 812 papers, producing 785 predicted positives for manual re-annotation and 27 silver exclusions. Manual review identified 728 additional inclusions and 57 exclusions. Combined with the 172 inclusions from the initial annotated set and after removing 12 duplicates, this yielded 888 papers for data extraction. Stage 3: Data extraction (DE). Schema development began from researcher-defined extraction goals and a 50-paper sample used to construct six initial schemas. Llama 3.3-70B generated raw extraction reports across the corpus, from which a collaborator pass proposed category merges, splits, edge cases, and exceptions. Researchers reviewed these suggestions, resolved conflicts, and finalized six extraction schemas: publication year, domain, review stage, approach, evaluation results, and limitations. Researchers then applied the finalized schemas to all 888 papers, yielding 16,777 human annotations. The same schemas were used to obtain machine annotations from 22 LLMs. Closed fields were evaluated programmatically, whereas the open-ended evaluation-results and limitations fields were assessed using a calibrated LLM-as-a-judge protocol based on a 100-paper calibration set and 29,724 labeled judge decisions. Solid arrows indicate the principal data flow through the review pipeline; dashed boxes denote evaluation or audit components rather than additional review stages.

Figure 2: Model capacity and full-text reliability across screening stages. a) Title and abstract screening performance scales with model size across 22 open-weight LLMs. Smaller models (<20B) generally perform poorly, while larger models achieve higher reliability; reasoning-oriented variants such as Qwen3 and gpt-oss reach strong performance despite moderate size. b) Full-text screening reliability depends jointly on parameter count and context window. Models with sufficient scale and long contexts achieve near-perfect reliability, while models with large context windows but insufficient parameters, or high capacity but limited context, often fail to produce parseable yes/no decisions.

Figure 3: Protocol choices that improve screening reliability. a) In title and abstract screening, prompts that make the inclusion criteria explicit improve average $F_{2}$ by 17.0% over a generic classification prompt; adding exclusion reasoning raises the total gain to 28.8%. b) In full-text screening, adding researcher rationales to few-shot examples raises average $F_{2}$ from 0.551 to 0.633, a 15% relative improvement. c) and d) Majority voting improves title and abstract screening only at suitable thresholds. In the strong pool, threshold 3 maximizes $F_{2}$ at 0.808; in the lenient pool, threshold 9 maximizes $F_{2}$ at 0.803, exceeding both the average baselines and the best single models.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

		zero-shot			few-shot 2			few-shot 5
model	model size	basic	incl	incl+ excl	basic	incl	incl+ excl	basic	incl	incl+ excl
Gemma 3	4B	0.148	0.186	0.157	0.129	0.136	0.146	0.154	0.151	0.164
Gemma 3	12B	0.179	0.212	0.257	0.188	0.219	0.317	0.329	0.307	0.339
Gemma 3	27B	0.253	0.293	0.421	0.227	0.249	0.324	0.359	0.330	0.428
gpt-oss	20B	0.620	0.655	0.628	0.644	0.658	0.679	0.677	0.694	0.686
gpt-oss	120B	0.686	0.681	0.697	0.687	0.684	0.675	0.674	0.659	0.686

Appendix

Table 2: Systematic evaluation in title and abstract screening. We report

F_{2}

(recall-weighted

F

-score) for each model across prompting and few-shot configurations. From single-model results, we select the top five configurations to form a strong pool (marked ∗ ) and build a majority-vote ensemble (predict “include” if at least

t_{\text{s}}

of 5 vote yes). To prioritize high recall, we also form a lenient pool (marked + ) of ten configurations with the highest recall among those with precision above 0.2, excluding configurations already in the strong pool; pool membership is frozen to resolve ties reproducibly. We further evaluate a combined ensemble that predicts “include” if either pool crosses its threshold. For these three ensemble variants we report

F_{2}

, precision, and recall, and select the best system subject to recall

=1.0

for curation.

		zero-shot				few-shot
model	model size	one-step		two-step		no thought traces		thought traces
		basic	strict	unstruct.	CoT	few-shot 2	few-shot 4	few-shot 2	few-shot 4
Gemma 3	4B	0.661	0.877	0.927	0.921	0.702	0.761	0.854	0.809
Gemma 3	12B	0.434	0.605	0.893	0.911	0.793	0.624	0.855	0.829
Gemma 3	27B	0.908	0.808	0.920	0.948	0.817	0.691	0.865	0.761
gpt-oss	20B	0.908	0.882	0.897	0.901	0.865	0.784	0.910	0.941

Appendix

Table 3: Systematic evaluation in full-text screening. We report

F_{2}

(the recall-weighted F-score) for each model across prompting and few-shot configurations. We also evaluate ensembling variants—including majority-vote ensembles over a strong pool of the top five single-model configurations and a lenient pool of configurations achieving recall (=1.0), a combined ensemble that triggers inclusion if either pool meets its threshold, and a token-length–aware routing ensemble (logistic regression with context-length bins). In our setting, none of these ensembles improves over the best single model, Llama 3.3-70B in zero-shot with the unstructured system prompt; we therefore use this configuration to score the remaining unlabeled full texts and manually re-annotate predicted positives for downstream extraction corpus curation.

Model	Model Size	Year Acc	Domain Jac	Review Stage Jac	Approach Jac
Gemma 3	4B	0.87	0.13	0.24	0.10
Gemma 3	12B	0.91	0.29	0.37	0.19
Gemma 3	27B	0.88	0.52	0.36	0.27
gpt-oss	20B	0.97	0.54	0.61	0.37
gpt-oss	120B	0.80	0.55	0.55	0.30
Llama 3	8B	0.07	0.02	0.06	0.04

Appendix

Table 4: Closed-field extraction performance of open-weight models on the full 888-paper set. Year is scored by accuracy; domain, review stage, and approach are scored by Jaccard overlap. The “Best open-weight” row reports the highest value observed separately for each field and therefore does not represent a single deployable model.

Model	Model Size	Eval Recall	Eval Precision	Limit. Recall	Limit. Precision
Gemma 3	4B	0.161	0.454	0.056	0.074
Gemma 3	12B	0.201	0.378	0.070	0.232
Gemma 3	27B	0.196	0.584	0.009	0.046
gpt-oss	20B	0.251	0.785	0.120	0.229
gpt-oss	120B	0.230	0.808	0.131	0.200
Llama 3	8B	0.003	0.092	0.005	0.048

Appendix

Table 5: Open-field recall and precision by model under calibrated LaaJ evaluation. Dashes (–) indicate unsupported estimates per the criteria of Lee et al. [36] ; full criteria in Methods.

Figure 5: Effect of few-shot learning on title and abstract screening. Average $F_{2}$ by model family when increasing the number of in-context examples (0, 2, 5). Gains are modest and heterogeneous across families: some improve slightly with a small number of examples, while others show no benefit or slight regressions, indicating few-shot prompting is not a universally reliable lever for TA screening.

Figure 6: Effect of chain-of-thought (CoT) prompting in full-text screening. Average $F_{2}$ by model family under two-step, zero-shot prompting, comparing an unstructured system prompt to a CoT variant that requires the model to reason through the criteria before emitting a binary decision. CoT yields a modest overall gain in $F_{2}$ (0.681 → 0.708), but effects are heterogeneous across families: it improves Gemma, Qwen, and Phi (notably lifting weaker families), while slightly degrading GPT, Llama, and Mistral on average.

Systematic Review Automation

See all 18 papers

Fine-Tuning A Large Language Model for Systematic Review Screening
Mar 25, 2026Kweku Yamoah, Noah Schroeder, Emmanuel Dorley +2Systematic Review Automation LLM Fine-Tuning
University of Florida, Gainesville, FL, 32611
Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs
Apr 29, 2026Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Erika YahataSoftware Engineering LLM Evaluation
AIBL · CESAR School · CMCC +1
A Benchmark Framework for Screening Automation in Systematic Reviews
Sep 16, 2026Gauransh Kumar, Luciano Marchezan, Guillaume Genois +2LLM Evaluation Evidence Selection
Université de Montréal Montreal, Canada

Mar 25, 2026J/K move · Enter open · S save

Fine-Tuning A Large Language Model for Systematic Review Screening

Kweku Yamoah, Noah Schroeder, Emmanuel Dorley, Neha Rani+1

University of Florida, Gainesville, FL, 32611

Systematic Review Automation LLM Fine-Tuning Evidence Selection LLM Evaluation

Open paper AskPDF arXiv

Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews.

Abstract

Systematic Review Automation

Explore similar work

Fine-Tuning A Large Language Model for Systematic Review Screening

Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs

A Benchmark Framework for Screening Automation in Systematic Reviews