Can Large Language Models Review the Scientific Literature? A Test Against a 102-Article Benchmark
Authors: Abdullah Al Kalbani, David M. Schultz, Alexandra K. Anderson-Frey, Zhonghua Zheng, Catherine McGuire
Organizations: Department of Earth and Environmental Sciences, The University of Manchester, Manchester, United Kingdom · Department of Computer Science, The University of Manchester, Manchester, United Kingdom · Centre for Crisis Studies and Mitigation, The University of Manchester, Manchester, United Kingdom · Department of Atmospheric and Climate Science, University of Washington, Seattle, Washington, United States · Research IT, The University of Manchester, Manchester, United Kingdom
Large language models (LLMs) are increasingly used to support scientific literature reviews, yet evaluations often examine individual tasks rather than the review process as a whole. We assess six AI tools across four stages: finding relevant articles, interpreting them, synthesizing across them, and proposing future research. The benchmark is a published review of the Spanish plume weather pattern and its 102-article corpus. In the initial searches, five tools collectively identified 57 articles, with the best individual tool finding 42. Institutional library access increased discovery from 42 to 82 articles for Codex and from 15 to 79 for Claude Code. Given the complete corpus of 102 PDFs, three tools correctly identified all 18 articles discussing the Spanish plume substantially, but incorrectly classified 40% to 79% of the 84 passing mentions as substantial. The same three tools produced syntheses evaluated blindly by two domain experts. Although largely accurate, the syntheses provided limited insight and sometimes contained unsupported claims, quotation errors, and exaggerated statements of consensus. Finally, all three tools proposed research directions broader than the Spanish plume literature, although two approached questions identified in the benchmark review. No single tool performed best across all tasks. We provide eight recommendations for researchers using AI tools in literature reviews. Our findings suggest that LLMs can assist with discovering and interpreting scientific literature, particularly when institutional access is available, but cannot replace expert judgment in evaluating evidence or synthesizing research.
Figures & tables
Figure 1: Workflow of the study, showing the inputs, four tasks, and their evaluation. The closed corpus contains the same 102 source-article PDFs for interpretation, synthesis, and research-gap proposals. The benchmark review and its classifications were withheld from the tools and used for evaluation.
Tool
Underlying model
Type
Plan (paid)
Tasks
ChatGPT Deep Research
GPT-5.6 Sol
online assistant
ChatGPT Plus
find
Gemini Deep Research
Gemini 3.1 Pro
online assistant
Google AI Pro
find
Claude (web interface)
Claude Opus 5
online assistant
Claude Pro
find
Codex
GPT-5.6 Sol
coding agent
ChatGPT Plus
find, interpret, synthesize, propose
Claude Code
Claude Opus 5
coding agent
Claude Pro
find, interpret, synthesize, propose
NotebookLM
Gemini 3.5
retrieval-grounded assistant
Google AI Pro
interpret, synthesize, propose
Table 1: The tools used, their underlying models at the time of the runs, and the tasks each performed. Model versions are those the products reported; NotebookLM does not report its model version; Gemini 3.5 is from Google’s announcement ( Google, 2026 ) . Plan is the subscription each tool was used on. All are the provider’s entry-level paid plan, which sets usage limits and, for the deep-research products, how much search each run may do. The tasks find, interpret, synthesize, and propose are represented by sections 3 , 4 , 5 , and 6 , respectively.
Tool
Type
Reported by the tool
Found
Codex (GPT-5.6 Sol)
coding agent
42
42
ChatGPT Deep Research (GPT-5.6 Sol)
online assistant
40
40
Gemini Deep Research (Gemini 3.1 Pro)
online assistant
35
16
Claude Code (Claude Opus 5)
coding agent
15
15
Claude web interface (Claude Opus 5)
online assistant
7
7
Found by at least one tool
—
57
Table 2: Articles found by each tool in searches run in late July and early August 2026. “Reported by the tool” is the tool’s own count after screening; “Found” is the number of those that the tool named and that are among the 102 articles. Gemini did not name all 35 articles it reported. Browser and login state were not controlled uniformly across tools.
Figure 2: Articles found (black) and not found (light gray) by at least one of the five tools, by publication year. The stacked bars show the 102-article corpus of Schultz et al. (2025) . The dashed line and filled triangle mark the median year of the articles found, and the dotted line and open triangle that of the corpus as a whole. Some of the articles shown as not found may be among those that Gemini reported but did not name.
Journal
Publisher
Articles in corpus
Articles found
Percentage
Weather and Forecasting
AMS
5
5
100%
International Journal of Climatology
Wiley (RMetS)
6
6
100%
Natural Hazards and Earth System Sciences
Copernicus
3
3
100%
Monthly Weather Review
AMS
4
3
75%
Quarterly Journal of the Royal Meteorological Society
Wiley (RMetS)
13
9
69%
Meteorological Applications
Wiley (RMetS)
3
2
67%
Table 3: Share of articles found, by journal and publisher, for journals holding at least three benchmark articles. “Articles found” counts articles that at least one tool found; articles that Gemini reported but did not name could increase these numbers. American Meteorological Society is abbreviated AMS, Royal Meteorological Society is abbreviated RMetS, and American Geophysical Union is abbreviated AGU. RMetS and AGU journals are published by Wiley.
Figure 3: Share of each publication period’s benchmark articles found by Codex (solid) and Claude Code (dashed), before (gray) and after (black) library login. The number of articles in each period is given below its label.
Figure 4: Articles found by (a) Codex and (b) Claude Code before and after login, by publisher. Bars and labels show the percentage of each publisher’s corpus articles found; n is the number of articles in that publisher group. Wiley includes the Royal Meteorological Society journals (57 articles) and five articles in other journals on its platform. The publisher groups do not indicate whether an article is free to read.
Codex
NotebookLM
Claude
Substantial articles correct (of 18)
18
18
18
Passing mentions misclassified (of 84)
66
65
34
Passing mentions correct (of 84)
18
19
50
Accuracy on passing/substantial
35%
36%
67%
Cohen’s κ on that split
0.09
0.09
0.34
Precision (substantial class)
0.21
0.22
0.35
Table 4: The databases built by the three tools, scored against the benchmark database as the gold standard, and the quotation audit. Accuracy on the two usage flags is the share of the 102 articles on which each flag (the term used for the airstream, for the synoptic pattern, or both) matches the benchmark database. Precision is the share of the articles a tool classed as substantial that the benchmark database also classes as substantial, recall is the share of the 18 substantial articles that the tool classed as substantial, and F1 is the harmonic mean of the two. Configuration (classic or modified plume) is scored only on the eight articles for which the benchmark database gives one; for one of them it gives both, so either answer counts as correct. The benchmark database marks a prefrontal trough in seven articles. The quotation rows are the first author’s own check against the supplied PDFs, not the tools’ self-reports. How the quotations were matched is described in the public repository.
Figure 5: Classification of the 102 articles as substantial or passing mention by (left) Codex, (middle) NotebookLM and (right) Claude. Rows give the benchmark database’s classification and columns the tool’s. Each cell gives the number of articles and, in parentheses, the share of its row; darker cells hold a larger share.
Reviewer
Tool
Quality
Depth
Veracity
Insight
Utility
Mean
Reviewer A
NotebookLM
4.2
3.3
4.2
3.4
3.3
3.68
Codex
3.6
3.0
3.7
3.2
3.2
3.34
Claude
4.4
4.3
4.3
4.2
4.4
4.32
Reviewer B
NotebookLM
2.9
2.6
3.7
2.3
2.4
2.78
Codex
3.1
2.9
3.8
2.4
2.4
2.92
Claude
2.8
2.9
3.6
2.5
2.3
2.82
Table 5: Mean blinded rating (1–5) per criterion, averaged over the ten themes, for each reviewer and tool. Reviewer A is the lead author of the benchmark review; Reviewer B is an independent severe-convection researcher.
Tool
Essential (4)
Informative (14)
Passing (84)
Total cited
NotebookLM
4
12
25
41
Codex
4
9
14
27
Claude
4
14
61
79
Table 6: Corpus articles cited in each synthesis, by the benchmark’s classification of how the article cites the Spanish plume.
Figure 6: Corpus articles cited in each synthesis. Each column is one of the 102 articles, ordered by publication year and then identifier (SP001–SP102). Colors show the benchmark classification ( Schultz et al., 2025 ) ; in the tool rows, white cells are articles not cited. The totals on the right count the articles in each row. Year groups are as wide as the number of articles in them, not the time they span. Identifiers are shown only for the 4 essential and 14 informative articles.
Figure 7: Corpus articles cited (black) and not cited (light gray) by each synthesis, by year of publication. The dashed line and triangle mark the median year of the articles each tool cited.
We qualitatively compared literature reviews produced with varying degrees of AI assistance. The same LLM, given the same corpus of 280 papers but different selections, produced dramatically different reviews, from mainstream and politically neutral to critical and post-colonial, though neither orientation was intended. LLM outputs always appear at first glance to be well written, well informed and thought out, but closer reading reveals gaps, biases and lack of depth. Our comparison of six versions shows a series of pitfalls and suggests precautions necessary when using AI assistance to make a literature review. Main issues are: (1) The bias of ignorance (you do not know what you do not get) in the selection of relevant papers. (2) Alignment and digital sycophancy: commercial AI models slavishly take you further in the direction they understand you give them, reinforcing biases. (3) Mainstreaming: because of their statistical nature, LLM productions tend to favor mainstream perspectives and content; in our case there was only 20% overlap between paper selections by humans and the LLM. (4) Limited capacity for creative restructuring, with vague and ambiguous statements. (5) Lack of critical perspective, coming from distant reading and political correctness. Most pitfalls can be addressed by prompting, but only if the user knows the domain well enough to detect them. There is a paradox: producing a good AI-assisted review requires expertise that comes from reading the literature, which is precisely what AI was meant to reduce. Overall, AI can improve the span and quality of the review, but the gain of time is not as massive as one would expect, and a press-button strategy leaving AI to do the work is a recipe for disaster. We conclude with recommendations for those who write, or assess, such LLM-augmented reviews.
Saadi Lahlou, Annabelle Gouttebroze, Atrina Oraee +1
London School of Economics and Political Science · Paris Institute for Advanced Study
Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We introduce SciLitBench, a multi-stage benchmark spanning title and abstract screening, full-text screening, and schema-guided data extraction, with 42,981 retrieved records, 1,012 full texts, and annotations for 888 included papers. Across 22 open-weight LLMs from six model families, explicit inclusion and exclusion criteria improve title and abstract screening F2 by 28.8%, while researcher-authored rationales improve full-text screening by 15%. Data extraction reveals a different reliability regime: performance declines from 0.97 accuracy for publication year to 0.37 Jaccard overlap for computational approach, while the strongest models recover only 30% of annotated evaluation evidence and 25% of limitations. SciLitBench identifies a practical boundary between high-recall screening and evidence-complete extraction and provides a reproducible resource for evaluating LLM-assisted evidence synthesis.
Miguel Zabaleta, Baihan Lin
Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA
The rapid growth of scholarly literature has made identifying relevant publications increasingly difficult, and conventional search systems still depend heavily on manually formulated queries and effortful manual inspection. Generative large language models (LLMs) offer a more flexible alternative, supporting literature retrieval and the screening of candidate studies against eligibility criteria. This chapter surveys 34 peer-reviewed papers applying generative LLMs to these two tasks, identified via a Boolean search over the OpenAIRE Graph (1,589 records screened to 34 inclusions). Reviewed studies are characterised by LLMs employed, model access and adaptation, prompting and architectural techniques, ground-truth sources, and evaluation metrics.