Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews

Gilberto Sussumu Hida; Danilo Monteiro Ribeiro; Clayton Suguio Hida

cs.CLAug 13, 2026

Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews

Authors: Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Clayton Suguio Hida

Organizations: Cesar School – AIBL Recife - PE - Brazil · UEAP Macap´a - AP - Brazil

Abstract

This study analyses LLMs in imbalanced binary classification, using study screening in systematic reviews as the application domain. An experiment was conducted in five reviews, comparing individual and batch processing, with and without prevalence metadata. The results indicate a limited influence of the prevalence metadata, with no evidence that it improves performance. In contrast, batch processing produced larger behavioral changes that varied according to the prevalence of the class. The aggregate and item-level analyses did not always coincide. Therefore, batch processing should be evaluated not only in terms of cost, but also in relation to its effects on decision-making behavior.

Figures & tables

Figure 1: Workflow of the analysis conducted in the study.

ID	Ref.	N	Inc.	Exc.	Prev.
rsl_0	[ 12 ]	1,812	53	1,759	2.9%
rsl_1	[ 20 ]	2,048	71	1,977	3.5%
rsl_2	[ 14 ]	1,527	160	1,367	10.5%
rsl_3	[ 23 ]	955	166	789	17.4%
rsl_4	[ 21 ]	300	159	141	53.0%

Table 1: Characterization of the selected SESR-Eval reviews.

Figure 2: Gwet’s AC1 by systematic review, experimental condition, and model.

Figure 3: Decision Flip Rate by condition pair, SLR, and model.

Figure 4: $\Delta$ F2 by condition pair, SLR, and model (95% CI, bootstrap ). Points to the right of the dashed line indicate F2 gain; points to the left indicate loss.

RSL	Prev.	Llama	GPT-5-mini
rsl_0	2.9%	+0.078	-0.007
rsl_1	3.5%	+0.043	+0.006
rsl_2	10.5%	+0.035	-0.003
rsl_3	17.4%	+0.105	+0.060
rsl_4	53.0%	-0.253	-0.110

Table 2: Net shift of the batch effect without metadata (A

\leftrightarrow

C), by prevalence.

Class-Imbalanced Learning

Class-imbalanced learning develops classifiers that recognize rare classes without sacrificing performance on common ones.

See all 257 papers

Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs
Apr 29, 2026Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Erika YahataSoftware Engineering LLM Evaluation
AIBL · CESAR School · CMCC +1
A Benchmark Framework for Screening Automation in Systematic Reviews
Sep 16, 2026Gauransh Kumar, Luciano Marchezan, Guillaume Genois +2LLM Evaluation Evidence Selection
Université de Montréal Montreal, Canada
Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?
Oct 7, 2026Aleksi Huotala, Miikka Kuutila, Mika MäntyläEvidence Selection LLM Evaluation
University of Helsinki, Helsinki, Finland. · LUT University, Lahti, Finland.

Apr 29, 2026J/K move · Enter open · S save

Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs

Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Erika Yahata

AIBL · CESAR School · CMCC+1

Software Engineering LLM Evaluation Evidence Selection Text Classification

Open paper AskPDF arXiv

Context: Study screening in systematic literature reviews is costly, inconsistency-prone, and risk-asymmetric, since false negatives can compromise validity. Despite rapid uptake of Large Language Models (LLMs), there is limited evidence on how such models behave during the study screening phase, particularly regarding the choice of specific LLMs and their comparison with classical models. Objective: To assess LLM performance and variability in screening, quantify the impact of input metadata (abstract, title, keywords), and compare LLMs with classical classifiers under a shared protocol. Methods: We analyzed 12 LLMs from 4 providers (OpenAI, Google Gemini, Anthropic, Llama) and 4 classical models (Logistic Regression, Support Vector Classification, Random Forest, and Naive Bayes) on 2 real Systematic Literature Reviews (SLRs), totaling 518 papers. The experimental design investigated 3 critical dimensions: (i) LLMs performance variability, (ii) the impact of input feature composition (abstract, title, and keywords) on LLM performance, and (iii) the real gain of using LLMs instead of more traditional classification models. Results: LLMs exhibited substantial heterogeneity and residual non-determinism even at temperature zero. Abstract availability was decisive: removing it consistently degraded performance, while adding title and/or keywords to the abstract yielded no robust gains. Compared to classical models, performance differences were not consistent enough to support generalizable LLM superiority. Discussion: LLM adoption should be justified by operational and governance constraints (reproducibility, cost, metadata availability), supported by pilot validation and explicit reporting of variability and input configuration.

Abstract

Class-Imbalanced Learning

Explore similar work

Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs

A Benchmark Framework for Screening Automation in Systematic Reviews

Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?