Fine-Tuning A Large Language Model for Systematic Review Screening

Kweku Yamoah; Noah Schroeder; Emmanuel Dorley; Neha Rani; Caleb Schutz

cs.CLMar 25, 2026

Fine-Tuning A Large Language Model for Systematic Review Screening

Authors: Kweku Yamoah, Noah Schroeder, Emmanuel Dorley, Neha Rani, Caleb Schutz

Organizations: University of Florida, Gainesville, FL, 32611

Abstract

Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews.

Figures & tables

Parameter	Value	Rationale
Fine-tuning regime
Adaptation method	Full fine-tuning	Update all weights (no adapters) to maximize task-specific capacity [ 10 , 11 ] .
Optimization
Optimizer	AdamW (8-bit)	Reduces optimizer-state memory while preserving AdamW behavior [ 17 ] .
Learning rate	$2\times 10^{-5}$	Conservative step size to reduce overfitting under limited supervision.
Warmup steps	5	Short warmup relative to the total step budget.

Table 1: Fine-tuning configuration for LFM2.5-1.2B-Instruct . We report the training regime, hyperparameters, and implementation choices used in our experiments.

Dataset	Total	Exclude (0)	Include (1)	Inclusion Rate
Training set	315	194	121	38.4%
Test set	56	39	17	30.4%
Total	371	233	138	37.2%

Table 2: Distribution of labeled abstracts across training and test partitions.

Setting	$N$	Acc.	Bal. Acc.	Macro- $F_{1}$	Macro- $F_{2}$	W- $F_{1}$	W- $F_{2}$
Base (no fine-tune) Full dataset	8,277	6.52	53.07	6.22	4.86	11.52	7.54
Fine-tuned Held-out test split	56	94.64	94.49	93.77	94.19	94.68	94.65
Fine-tuned Full dataset	8,277	86.40	88.78	48.95	50.41	92.31	88.48

Table 3: Binary classification performance across evaluation settings. We report results for three settings: (i) the base model (no fine-tuning) evaluated on the full dataset, (ii) the fine-tuned model evaluated on the held-out test split used during fine-tuning, and (iii) the fine-tuned model evaluated on the full dataset. Metrics are percentages. Accuracy is overall accuracy; Balanced Accuracy is mean per-class recall (useful under strong class imbalance). We report Macro/Weighted

F_{1}

and additionally Macro/Weighted

F_{2}

, where

F_{2}

weights recall four times more than precision. Macro metrics weight classes equally; weighted metrics weight by class support.

Setting	Class	Support	Prec.	Rec.	F 1
Base (no fine-tune) Full dataset	0 (Exclude)	8,243	100.00	6.14	11.57
	1 (Include)	34	0.44	100.00	0.87
Fine-tuned Held-out test split	0 (Exclude)	39	97.37	94.87	96.10
	1 (Include)	17	88.89	94.12	91.43
Fine-tuned Full dataset	0 (Exclude)	8,243	99.96	86.38	92.67
	1 (Include)	34	2.69	91.18	5.22

Table 4: Per-class performance across evaluation settings. We report precision, recall, and F 1 for each class (0= Exclude , 1= Include ). Recall equals per-class accuracy (i.e., accuracy restricted to the examples of that true class): for class 0,

\mathrm{TN}/(\mathrm{TN}+\mathrm{FP})

; for class 1,

\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})

. Support is the number of true instances per class. Values are percentages.

Setting	True class	Pred 0	Pred 1
Base (no fine-tune) Full dataset	True 0 (Exclude)	6.14	93.86
Base (no fine-tune) Full dataset	True 1 (Include)	0.00	100.00
Fine-tuned Held-out test split	True 0 (Exclude)	94.87	5.13
Fine-tuned Held-out test split	True 1 (Include)	5.88	94.12
Fine-tuned Full dataset	True 0 (Exclude)	86.38	13.62
Fine-tuned Full dataset	True 1 (Include)	8.82	91.18

Table 5: Confusion matrices as percentages (row-normalized). For each setting, we report the confusion matrix normalized by the number of true examples in each class (rows sum to 100%). Columns correspond to predicted labels (0= Exclude , 1= Include ); rows correspond to true labels. This presentation highlights false positive and false negative rates independently of class prevalence.

Setting	$N$	$P_{o}$ (%)	Cohen’s $\kappa$	PABAK	Gwet AC1
Base (no fine-tune) Full dataset	8,277	6.52	0.001	-0.870	-0.863
Fine-tuned Held-out test split	56	94.64	0.875	0.893	0.906
Fine-tuned Full dataset	8,277	86.40	0.045	0.728	0.843

Table 6: Model–human agreement across evaluation settings. We report agreement statistics computed from the binary confusion matrix for three settings: (i) the base model (no fine-tuning) evaluated on the full dataset, (ii) the fine-tuned model evaluated on the held-out test split used during fine-tuning, and (iii) the fine-tuned model evaluated on the full dataset.

P_{o}

denotes observed agreement (i.e., accuracy) reported as a percentage. We additionally report Cohen’s

\kappa

(chance-corrected agreement), PABAK (prevalence-adjusted, bias-adjusted

\kappa

), and Gwet’s AC1 (a chance-corrected agreement measure that is often more stable under strong prevalence imbalance).

Statistic	Estimate	CI 95 (lo)	CI 95 (hi)
Human + LLM runs (4 raters)
Fleiss’ $\kappa$	0.603	0.599	0.607
Gwet AC1	0.842	0.834	0.850
LLM-only consistency (pairwise Cohen’s $\kappa$ across temperatures)
$T{=}0.1$ vs. $T{=}0.4$	1.000	–	–
$T{=}0.1$ vs. $T{=}0.8$	1.000	–	–

Table 7: Agreement and consistency over Phase-1 screening labels (

N=8{,}277

). Multi-rater agreement is computed across four raters: the human decision and three LLM inference runs at decoding temperatures

T\in\{0.1,0.4,0.8\}

. We report Fleiss’

\kappa

and multi-rater Gwet’s AC1 with 95% bootstrap confidence intervals (item-wise resampling). We additionally report LLM-only consistency as pairwise Cohen’s

\kappa

between the three temperature runs (computed over all items; no CI). Perfect pairwise agreement (

\kappa{=}1.0

) indicates identical predicted labels across temperatures in this setting.

Systematic Review Automation

See all 18 papers

SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews
Aug 29, 2026Miguel Zabaleta, Baihan LinSystematic Review Automation LLM Evaluation
Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA
Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations
Jun 16, 2026Mika Mäntylä, Patricia Matsubara, Katia Romero Felizardo +5LLM Evaluation Evidence Selection
University of Helsinki, Finland · UFMS, Brazil · UTFPR – Federal University of Technology - Paraná, Brazil +3
Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs
Apr 29, 2026Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Erika YahataSoftware Engineering LLM Evaluation
AIBL · CESAR School · CMCC +1

Aug 29, 2026J/K move · Enter open · S save

SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews

Miguel Zabaleta, Baihan Lin

Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA

Systematic Review Automation LLM Evaluation Evidence Selection LLM Information Extraction Scientific Workflow Automation

Open paper AskPDF arXiv

Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We introduce SciLitBench, a multi-stage benchmark spanning title and abstract screening, full-text screening, and schema-guided data extraction, with 42,981 retrieved records, 1,012 full texts, and annotations for 888 included papers. Across 22 open-weight LLMs from six model families, explicit inclusion and exclusion criteria improve title and abstract screening

F_2

by 28.8%, while researcher-authored rationales improve full-text screening by 15%. Data extraction reveals a different reliability regime: performance declines from 0.97 accuracy for publication year to 0.37 Jaccard overlap for computational approach, while the strongest models recover only 30% of annotated evaluation evidence and 25% of limitations. SciLitBench identifies a practical boundary between high-recall screening and evidence-complete extraction and provides a reproducible resource for evaluating LLM-assisted evidence synthesis.

Abstract

Systematic Review Automation

Explore similar work

SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews

Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations

Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs