Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency

Nikol Figalová; Lynn Huestegge; Anne Böckler-Raettig

cs.AIAug 27, 2026

Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency

Authors: Nikol Figalová, Lynn Huestegge, Anne Böckler-Raettig

Organizations: Institute of Psychology, Julius-Maximilians-Universität Würzburg, Würzburg, Germany

Abstract

Background. Large language models (LLMs) are increasingly used for screening in evidence synthesis, where false negatives can remove relevant studies before full-text assessment. We compared human and LLM title-and-abstract screening workflows in a preregistered study embedded in a conceptually complex scoping review. Methods. After a conservative title-only screen, 1,131 records were screened by one review lead, four trained assistants screening non-overlapping subsets, and seven complete LLM runs using different models and processing configurations, including a nominally identical repeat run. We compared retained workload, operational recall against 316 verified eligible records, agreement, run-to-run consistency, and procedural burden. Because eligibility was verified only for records advanced and assessed in the parent review, recall estimates were operational. Results. No workflow recovered all verified eligible records. The human workflows and two GPT-5.4 file-batch runs retained 42.2-45.0% of records while achieving 82.3-82.9% recall. Gemini 3.1 file batches achieved the highest recall (83.9%) but retained 56.7% of records. All-at-once configurations recovered fewer eligible records than corresponding file-batch configurations. Two nominally identical GPT-5.4 file-batch runs agreed on 91.7% of records but differed on 94 records, including 29 verified eligible records retained by only one run. Discussion. LLM screening performance depended on the implemented workflow, not model identity alone. Processing configuration, workload, record-level variation, and human-LLM decision integration are therefore substantive properties of deployed systems. For high-recall tasks, LLMs are better suited to validated, auditable, human-supervised workflows than autonomous exclusion.

Figures & tables

Figure 1: Operational recall as a function of retained screening workload across human and LLM-based screening workflows.

Model	Date	Processing configuration	Role in study
ChatGPT 5.4 Thinking	March 17, 2026	Files of approx. 100 records	Advancement workflow
ChatGPT 5.4 Thinking	March 17, 2026	All 1,131 records at once	Comparative run
ChatGPT 5.4 Thinking	March 18, 2026	Files of approx. 100 records	Nominally identical repeat of the March 17 file-batch workflow
ChatGPT 5.4 Thinking	March 19–24, 2026	Sequential groups of 10 records	Exploratory comparative run
Gemini 3 Thinking	March 17, 2026	Files of approx. 100 records	Advancement workflow
Gemini 3 Thinking	March 17, 2026	All 1,131 records at once	Comparative run

Table 1: LLM-based title-and-abstract screening workflows evaluated in the study. Each run comprised one complete pass through all 1,131 benchmark records.

RQ	Evaluation dimension	Outcome measures	Analytical set
RQ1	Recovery and retained workload	Operational recall and missed eligible records quantified recovery of verified eligible records. Retained workload quantified the number of benchmark records retained for possible full-text assessment.	316 verified eligible records; all 1,131 benchmark records
RQ2	Agreement between screening outputs	Overall agreement, Cohen’s $\kappa$ , Gwet’s AC1, and positive and negative agreement compared binary retained/not-retained decisions.	All 1,131 benchmark records
RQ3	Conditional classification performance	Precision, specificity, and $F_{1}$ described classification performance among records with verified full-text outcomes.	859 full-text-assessed records
RQ4	Run-to-run consistency	The two nominally identical GPT-5.4 file-batch runs were compared using record-level agreement, retained workload, operational recall, missed eligible records, and discordant decisions.	All 1,131 benchmark records, including 316 verified eligible records
RQ5	Missed-record patterns and retrospective checks	Record-level analyses examined overlap in missed eligible records, unique recovery, and records that alternative procedures would have advanced. Thirteen non-advanced records retained by the later sequential run were reassessed post hoc.	All 1,131 benchmark records, including 316 verified eligible records; 13 reassessed records
Explor.	Pairwise workflow complementarity	A liberal union rule quantified recovery and retained workload when a record was retained if either member of a workflow pair retained it.	All 1,131 benchmark records, including 316 verified eligible records

Table 2: Evaluation dimensions, outcome measures, and analytical sets used to address the research questions.

Workflow	Retained workload, $n$ (%)	Eligible retained, $n$ (%)	Eligible missed, $n$ (%)	Operational recall (95% CI)
Single reviewer	477 (42.2%)	262 (82.9%)	54 (17.1%)	0.829 (0.785–0.870)
Distributed team	509 (45.0%)	260 (82.3%)	56 (17.7%)	0.823 (0.778–0.864)
Gemini 3: file batches	714 (63.1%)	244 (77.2%)	72 (22.8%)	0.772 (0.725–0.816)
Gemini 3: all at once	667 (59.0%)	209 (66.1%)	107 (33.9%)	0.661 (0.608–0.715)
GPT-5.4: file batches	493 (43.6%)	261 (82.6%)	55 (17.4%)	0.826 (0.785–0.867)
GPT-5.4: all at once	391 (34.6%)	211 (66.8%)	105 (33.2%)	0.668 (0.614–0.718)

Table 3: Retained workload and recovery of verified eligible records for each screening workflow.

Workflow pair	Agreement, % (95% CI)	Positive agreement (95% CI)	Negative agreement (95% CI)	Cohen’s $\kappa$ (95% CI)	Gwet’s AC1 (95% CI)
Human–human
Single reviewer vs distributed team	72.8 (70.2–75.3)	0.688 (0.653–0.720)	0.759 (0.732–0.784)	0.447 (0.393–0.499)	0.464 (0.413–0.516)
Single reviewer vs LLM
Single reviewer vs Gemini 3 file batches	55.0 (52.1–57.8)	0.573 (0.539–0.605)	0.525 (0.488–0.560)	0.135 (0.083–0.187)	0.102 (0.045–0.160)
Single reviewer vs GPT-5.4 file batches	71.2 (68.4–73.8)	0.664 (0.629–0.698)	0.748 (0.720–0.773)	0.412 (0.356–0.465)	0.435 (0.380–0.488)
Distributed team vs LLM

Table 4: Selected pairwise agreement comparisons for binary retained/not-retained decisions across the 1,131-record benchmark.

Workflow	FP	TN	Precision (95% CI)	Specificity (95% CI)	$F_{1}$ (95% CI)
Single reviewer	178	365	0.595 (0.565–0.627)	0.672 (0.634–0.711)	0.693 (0.662–0.723)
Distributed team	208	335	0.556 (0.527–0.585)	0.617 (0.575–0.657)	0.663 (0.633–0.692)
Gemini 3: file batches	427	116	0.364 (0.346–0.381)	0.214 (0.179–0.250)	0.494 (0.470–0.518)
Gemini 3: all at once	335	208	0.384 (0.359–0.409)	0.383 (0.343–0.424)	0.486 (0.453–0.518)
GPT-5.4: file batches	202	341	0.564 (0.535–0.594)	0.628 (0.587–0.669)	0.670 (0.640–0.699)
GPT-5.4: all at once	153	390	0.580 (0.543–0.617)	0.718 (0.681–0.755)	0.621 (0.583–0.658)

Table 5: Conditional classification performance among the 859 records with verified full-text outcomes.

Workflow or reviewer	Records screened	Time, min	Min per record	Records per hour
Single reviewer	1,131	1,176	1.04	57.7
Distributed team, total	1,131	1,628	1.44	41.7
Assistant 1	300	418	1.39	43.1
Assistant 2	331	505	1.53	39.3
Assistant 3	200	300	1.50	40.0
Assistant 4	300	405	1.35	44.4

Table 6: Logged human screening time and throughput during formal title-and-abstract screening.

Evidence Selection

Evidence selection chooses the documents, passages, claims, or observations that best support an answer or decision.

See all 61 papers

Jun 16, 2026

Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations

Several studies have examined the use of large language models (LLMs) for title-abstract screening in systematic reviews (SRs), reporting mixed accuracy. However, questions of reliability remain largely unaddressed. In this study, we go beyond quantitative LLM-human agreement metrics and qualitatively investigate how and why LLMs fail. We also propose actionable recommendations. We analyzed disagreements between LLMs and researchers across six software engineering SRs and over 1,000 primary study papers. For each SR, papers were screened independently by human experts and LLMs in zero-shot mode, resulting in Kappa values ranging from 0.52 to 0.77. Qualitative analysis suggests that human-LLM disagreement results from recurring, identifiable causes, such as boundary ambiguity in key terms, keyword overemphasization, and incorrect topic inference. Based on these findings, we propose recommendations such as validating semantic understanding before deployment, running multiple LLMs, and focusing validation efforts on borderline cases. Future studies are needed to validate the impact of our recommendations, and community efforts are needed to develop normative guidelines on LLM usage in SRs.

Mika Mäntylä, Patricia Matsubara, Katia Romero Felizardo +5

University of Helsinki, Finland · UFMS, Brazil · UTFPR – Federal University of Technology - Paraná, Brazil +3

LLM Evaluation Evidence Selection LLM Reliability

Ask PDF

Apr 29, 2026

Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs

Context: Study screening in systematic literature reviews is costly, inconsistency-prone, and risk-asymmetric, since false negatives can compromise validity. Despite rapid uptake of Large Language Models (LLMs), there is limited evidence on how such models behave during the study screening phase, particularly regarding the choice of specific LLMs and their comparison with classical models. Objective: To assess LLM performance and variability in screening, quantify the impact of input metadata (abstract, title, keywords), and compare LLMs with classical classifiers under a shared protocol. Methods: We analyzed 12 LLMs from 4 providers (OpenAI, Google Gemini, Anthropic, Llama) and 4 classical models (Logistic Regression, Support Vector Classification, Random Forest, and Naive Bayes) on 2 real Systematic Literature Reviews (SLRs), totaling 518 papers. The experimental design investigated 3 critical dimensions: (i) LLMs performance variability, (ii) the impact of input feature composition (abstract, title, and keywords) on LLM performance, and (iii) the real gain of using LLMs instead of more traditional classification models. Results: LLMs exhibited substantial heterogeneity and residual non-determinism even at temperature zero. Abstract availability was decisive: removing it consistently degraded performance, while adding title and/or keywords to the abstract yielded no robust gains. Compared to classical models, performance differences were not consistent enough to support generalizable LLM superiority. Discussion: LLM adoption should be justified by operational and governance constraints (reproducibility, cost, metadata availability), supported by pilot validation and explicit reporting of variability and input configuration.

Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Erika Yahata

AIBL · CESAR School · CMCC +1

Software Engineering LLM Evaluation Evidence Selection Text Classification

Ask PDF

Aug 29, 2026

SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews

Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We introduce SciLitBench, a multi-stage benchmark spanning title and abstract screening, full-text screening, and schema-guided data extraction, with 42,981 retrieved records, 1,012 full texts, and annotations for 888 included papers. Across 22 open-weight LLMs from six model families, explicit inclusion and exclusion criteria improve title and abstract screening

F_2

by 28.8%, while researcher-authored rationales improve full-text screening by 15%. Data extraction reveals a different reliability regime: performance declines from 0.97 accuracy for publication year to 0.37 Jaccard overlap for computational approach, while the strongest models recover only 30% of annotated evaluation evidence and 25% of limitations. SciLitBench identifies a practical boundary between high-recall screening and evidence-complete extraction and provides a reproducible resource for evaluating LLM-assisted evidence synthesis.

Miguel Zabaleta, Baihan Lin

Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA

Systematic Review Automation LLM Evaluation Evidence Selection LLM Information Extraction Scientific Workflow Automation

Ask PDF

Abstract

Evidence Selection

Explore similar work

Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations

Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs

SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews