cs.SEAug 25, 2026

ARISMA: Guidelines for AI- and LLM-Assisted Systematic Reviews, Scoping Reviews, and Mapping Studies

Authors: Mahyar Tourchi Moghaddam, Mina Alipour

Organizations: University of Southern Denmark, Odense 5230, DK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Systematic reviews, scoping reviews, mapping studies, and related evidence syntheses are increasingly difficult to conduct with fully manual workflows as search volumes, update cycles, and synthesis requirements continue to expand. At the same time, artificial intelligence, machine learning, and large language models are rapidly entering review practice across query formulation, screening, extraction, categorization, appraisal support, and reporting. Yet the empirical evidence remains uneven, task-dependent, and insufficient to justify unconstrained automation. Existing standards such as PRISMA 2020, PRISMA-S, PRISMA-ScR, PRISMA-P, PRESS, and SWiM remain essential, but none provides an end-to-end operational standard for when AI use is methodologically appropriate, how it should be validated, which review decisions must remain human-led, and how AI involvement should be reported so that readers can audit it. This paper proposes ARISMA, an AI Reporting and Integration standard for Systematic Methods and Analysis. ARISMA treats AI as an inspected, benchmarked, logged, and reversible assistant rather than an autonomous reviewer. It is built around one governing principle: every consequential scientific decision must remain human-interpretable, human-auditable, and human-accountable. The paper contributes a lifecycle taxonomy, process guidance, stepwise recommendations across the review pipeline, a governance and provenance model, a tool-support framework, an AI-integrated reporting checklist, and a validation matrix. It also addresses legal, privacy, infrastructure, and sustainability considerations. The framework was iteratively refined through structured expert consultation. The result is a practical and auditable guideline for responsible AI-assisted evidence synthesis.

Figures & tables

Figure 1: General ARISMA process. AI support is attached to auditable artefacts and constrained by human validation checkpoints.
Figure 2: ARISMA Retrieval and enrichment pipeline: search, deduplication, and snowballing.
Figure 3: Screening workflow. AI is calibrated against human judgments and cannot silently determine the evidence base.
Figure 4: AI governance and validation cycle in ARISMA: protocol first, validation deployment, and continuous oversight. Evidence synthesists remain responsible for the review. AI is an inspected, benchmarked, logged, and reversible assistant, not an autonomous reviewer.
Figure 5: ARISMA evidence flow with iterative snowballing, provenance tracking, validation, and human checkpoints. n is the number of records/reports at that step. All AI outputs are inspected, validated, and overridden by humans as needed.
Figure 6: Verified provenance chain from extraction to synthesis. AI can prepopulate and draft, but verified evidence tables remain the source of truth.
ItemWhat should be reportedAdditional AI and LLM reporting required
TitleIdentify the review type clearlyState if the review was AI-assisted
AbstractStructured summary of rationale, methods, results, and conclusionsName the AI-supported steps and the level of human oversight
RationaleWhy the review was neededWhy AI was considered necessary or useful for this workflow
ObjectivesExplicit review question and scopeWhich steps AI was allowed to influence and which were human-only
Review designSystematic, scoping, mapping, EGM, rapid, or otherWhy the chosen AI role is appropriate for that review family
Eligibility criteriaFull inclusion and exclusion criteriaHow criteria were translated into prompts, rules, or labels
Table 1: Reporting items and additional AI/LLM reporting requirements
StepWhat must be validatedMinimum ARISMA evidence
Search-term generationRecall of known or sentinel studiesPilot benchmark set, documented failures, revised search after peer review
DeduplicationFalse positives and false negativesTool log plus human audit of uncertain duplicate clusters
Title and abstract screeningSensitivity for relevant recordsHuman-coded pilot set, error review, explicit rule for handling AI exclusions
Full-text screeningJustification quality and false exclusionsHuman consensus sign-off on all final exclusions
Extraction of descriptive fieldsAgreement with source papersRandom or full audit, depending on field criticality
Extraction of numeric fieldsExactness of synthesis-critical valuesFull human verification of all values used in aggregation or meta-analysis
Table 2: Validation requirements and minimum ARISMA evidence
More in this topic

Systematic Review Automation

See all 18 papers

Explore similar work

CardsList
Sep 10, 2026

PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews

Large language models (LLMs) and AI-enabled software increasingly participate in systematic-review decisions, yet the information needed to audit these workflows is reported inconsistently. We analyze SciLitBench, a corpus of 888 review-automation papers with 14,726 annotations, to characterize changes in methods, review-stage use, evaluation and reported limitations. Automation has shifted toward LLM- and software-facing workflows, including stages that can alter the evidence base. Since 2023, 38.0% of software/product papers reported no evaluation, compared with 9.3% of LLM papers. Reporting coverage increased with LLM workflow complexity, yet 52% of positive-only LLM evaluations still reported an unmet reliability or performance requirement. From these patterns, we introduce PRISMA-LLM, an empirically grounded framework separating implementation disclosure from consequence-sensitive evaluation and limitation reporting.

Miguel Zabaleta, Baihan Lin

Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA

LLM AuditingSystematic Review AutomationAI-Assisted Scientific ResearchLLM ReliabilityAutomated Evaluation
AskPDF
Jul 27, 2026

Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews

Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, while capable of summarizing text, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require. Objectives: To support researchers intending to conduct SLRs using GenAI or those conducting empirical studies evaluating how well GenAI supports SLR tasks. Methods: First, we conducted a rapid review to identify studies that propose guidelines for evaluating and using GenAI and LLMs to support SLRs. Second, we drew on thought experiments, relevant guidance from the literature, and our own experience conducting SLRs and evaluating tools to develop recommendations for how to use and assess GenAI in the context of SLRs. Results: We discuss the problems researchers face when evaluating GenAI for SLRs. We identify and explain process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools. Finally, we summarize our results as a set of process recommendations, which we name GUEST (GenAI Use and Evaluation in SLR Tasks). Conclusion: We argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies. However, it offers the prospect of cost-effective assistance for some repetitive tasks and for additional validation of some complex tasks. Our GUEST recommendations should help software engineering researchers both to conduct and report trustworthy SLRs using GenAI and to provide rigorous independent evaluation studies.

Barbara Kitchenham, Sebastián Pizard, Lech Madeyski +3

Keele University, United Kingdom · Universidad de la República, Uruguay · Wrocław University of Science and Technology, Poland +3

LLM EvaluationSystematic Review AutomationHuman-in-the-Loop EvaluationGenerative AI Evaluation
AskPDF
Oct 15, 2026

ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews

Systematic literature reviews follow the PRISMA process of screening scholarly works and extracting evidence from included studies. Deep research tools based on large language models (LLMs) can search and summarize the literature but do not record which studies are excluded or where extracted values come from. We develop a conversational multi-agent system called ReviewPilot that automates the PRISMA workflow from paper screening to information extraction. Researchers can tailor criteria and schemas through dialogue, and a project memory keeps their configurations and revisions. Each screening decision comes with its reason and each extracted value with its source. On epidemiological expert reviews, ReviewPilot outperforms a single-prompt LLM and AgentSLR across three LLMs, with screening macro F1 up to 0.776 and extraction F1 up to 0.788. Case studies in biomedicine and human-computer interaction show close agreement with human annotators, suggesting that ReviewPilot can transfer beyond epidemiology. Code is available at https://github.com/ZhaoqianXue/ReviewPilot.

Zhaoqian Xue, Shaoze Zhou, Xuexin Li +10

University of Arizona · Florida International University · Vanderbilt University +7

Systematic Review AutomationAI-Assisted Scientific ResearchMulti-Agent LLM SystemsLLM Information Extraction
AskPDF