LLM4SCREENLIT: Recommendations on assessing the performance of large language models for screening literature in systematic reviews
Abstract
Context: Large language models (LLMs) are increasingly used to screen literature for systematic reviews (SRs), but the standard confusion-matrix metrics used to evaluate them can mislead under the imbalanced, cost-asymmetric conditions of screening. Objective: We develop and justify LLM4SCREENLIT-practical recommendations for researchers conducting LLM-screening evaluations and for editors and reviewers assessing such studies-differentiated by study type (retrospective benchmarking vs deployment for a specific SR). Method: Using Delgado-Chaves et al. (2025), an 18-LLM benchmark across three biomedical SRs, as a motivating example, we reviewed 28 additional papers and extracted their reported metrics. We propose a Weighted Matthews Correlation Coefficient (WMCC) that integrates MCC's chance-correction with asymmetric misclassification costs, and validated it on three software-engineering (SE) reanalyses, the largest covering 9 LLMs x 24 SE secondary studies (34,528 articles). Results: Across the 29 papers, only 10% reported MCC, only 24% reported full confusion matrices, and none of the five papers claiming workload savings priced false-negative cost. In the largest SE reanalysis, MCC and WMCC disagree on the best LLM in 55% of evaluable studies; in the most striking 9,695-article SE study, the Accuracy-best LLM loses 63.3% of relevant evidence (Lost Evidence), the MCC-best 43.9%, but the WMCC-best only 5.8%. Sensitivity analysis (median crossover at w~=2.7, all <7) supports w=10 as a conservative default. Conclusions: SR-screening evaluations should prioritize Lost Evidence and use cost-sensitive WMCC alongside MCC for ranking. Reporting must include the full confusion matrix and treat unclassifiable outputs as positives requiring human review. Designs should be leakage-aware, with non-LLM baselines when the study aims to inform SR practice and labels are available.
Figures & tables
| Models: | ||||
| Metrics: | gemma:7b | llama3-Athene:70b | llama3.1:8b | mistral-nemo:12b |
| True Negatives (TNs) | 4324 | 4242 | 4048 | 4326 |
| False Negatives (FNs) | 172 | 125 | 90 | 172 |
| True Positives (TPs) | 0 | 47 | 82 | 0 |
| False Positives (FPs) | 0 | 82 | 281 | 3 |
| Total Articles (N*) | 4496 | 4496 | 4501 | 4501 |
| ID | Performance Metrics | CM | Origin | Type | Other |
|---|---|---|---|---|---|
| Baselines | |||||
| Akinseloyin-2024 [ 25 ] | L-Rel; MAP; RecAT%; WS | No | Sandner | J | No |
| Attri-2024 [ 26 ] | Acc; %Pos; Rec; Spec | No | Kim | A | No |
| Cai-2023 [ 27 ] | F1; Prec; Rec; WS | No | Both | J | No |
| Cao-2024 [ 28 ] | Acc; Rec; Spec | No | Sandner; A | GL | No |
| Castillo-2023 [ 29 ] | Acc; F1; NegPred; Nulls; Prec; Rec; Spec | Yes | A | C | No |
| Confusion Matrix | |||||||||||
| SLR | Threshold | TP | FP | FN | TN | N | Acc | Recall | Lost Ev. | MCC | WMCC |
| SLR1 | Likert 4 | 50 | 35 | 14 | 35 | 134 | 63.4% | 0.781 | 21.9% | 0.292 | 0.195 |
| SLR1 | Likert 5 | 48 | 17 | 16 | 53 | 134 | 75.3% | 0.750 | 25.0% | 0.507 | 0.330 |
| SLR2 | Likert 4 | 128 | 68 | 20 | 232 | 448 | 80.3% | 0.865 | 13.5% | 0.605 | 0.557 |
| SLR2 | Likert 5 | 113 | 27 | 35 | 273 | 448 | 86.1% | 0.764 | 23.6% | 0.683 | 0.529 |
| Confusion Matrix | ||||||||
| Dataset | Classifier | TP | FP | FN | TN | Lost Ev. | MCC | WMCC |
| RL4SE | ChatGPT | 77 | 310 | 17 | 685 | 18.1% | 0.298 | 0.511 |
| (94+/995-) | LR | 56 | 130 | 38 | 865 | 40.4% | 0.347 | 0.485 |
| DSMLCompo | ChatGPT | 114 | 572 | 36 | 1961 | 24.0% | 0.281 | 0.522 |
| (150+/2533-) | CNB | 112 | 790 | 38 | 1743 | 25.3% | 0.211 | 0.421 |
| UpdateCollab. | ChatGPT | 51 | 286 | 6 | 532 | 10.5% | 0.276 | 0.542 |
| Confusion Matrix | ||||||||
| LLM (selection criterion) | TP | FP | FN | TN | Acc | Lost Ev. | MCC | WMCC |
| Claude 3.7 Sonnet ( Acc -best) | 189 | 213 | 326 | 8967 | 94.4% | 63.3% | 0.387 | 0.466 |
| GPT-4.1 mini ( MCC -best) | 289 | 331 | 226 | 8849 | 94.3% | 43.9% | 0.481 | 0.604 |
| GPT-4o ( WMCC -best) | 485 | 1743 | 30 | 7437 | 81.7% | 5.8% | 0.401 | 0.724 |
| gpt-4.1-nano ( Recall -best, excl. include-all) | 505 | 7598 | 10 | 1582 | 21.5% | 1.9% | 0.093 | 0.228 |
| Ministral 8B (include-all) | 515 | 9180 | 0 | 0 | 5.3% | 0.0% | NaN | NaN |
| R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 | |
| Title/Abstract (origin) | T | T | T | T | T | T | T | T | C | T |
| Full-text screening | T | T | T | T | T | T | T | T | C | T |
| Data extraction | P a | P a | P b | P c | T | T | T | T | C | P b |
| Small | Medium | Large | ||||
| (lower tercile) | (middle tercile) | (upper tercile) | ||||
| Range | Median | Range | Median | Range | Median | |
| MCC | – | |||||
| WMCC ( ) | – | |||||
| Model | Prompt | SR | TP | FP | FN | TN | Recall | Lost Ev. | MCC | WMCC | Null% | CI |
| GPT-3.5-turbo-0613 | PositiveX | MobileMDE | 47 | 50 | 8 | 187 | 0.855 | 14.5% | 0.534 | 0.624 | 1.7% | — |
| … | … | … | … | … | … | … | … | … | … | … | … | … |
| Gold Standard | Gold Standard | Total | |
| True | False | ||
| Predicted True | TP | FP | TP+FP |
| Predicted False | FN | TN | FN+TN |
| Total | TP+FN | FP+TN | N |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,509 papers