Large Language Models (LLMs) are transforming scholarly tasks like search and summarization, but their reliability remains uncertain. Current evaluation metrics for testing LLM reliability are primarily automated approaches that prioritize efficiency and scalability, but lack contextual nuance and fail to reflect how scientific domain experts assess LLM outputs in practice. We developed and validated a schema for evaluating LLM errors in scholarly question-answering systems that reflects the assessment strategies of practicing scientists. In collaboration with domain experts, we identified 20 error patterns across seven categories through thematic analysis of 68 question-answer pairs. We validated this schema through contextual inquiries with 10 additional scientists, which showed not only which errors experts naturally identify but also how structured evaluation schemas can help them detect previously overlooked issues. Domain experts use systematic assessment strategies, including technical precision testing, value-based evaluation, and meta-evaluation of their own practices. We discuss implications for supporting expert evaluation of LLM outputs, including opportunities for personalized, schema-driven tools that adapt to individual evaluation patterns and expertise levels.
Figures & tables
Figure 1. Errors identified by domain experts and model developers with entity tags for anonymity. The expert recognized a chronological error about test sequences that the developer missed, showing how domain expertise can yield more precise error analysis. A worked example of the open coding process, presented as four labeled boxes stacked vertically, each outlined with a thin green border and separated by white space. The figure is primarily textual. \parBox 1 is labeled Question:'' in bold and reads: How was shake-down testing of the microphone array conducted prior to deployment in a production environment?'' \parBox 2 is labeled LLM Output:'' in bold and reads: The shake-down testing of the microphone array was conducted at [LOCATION] in [YEAR] prior to its deployment in a production environment. During this testing, the array's operation was nominal with only 8 microphones out of 185 exhibiting problems that required replacement with backup units. The sensors were subjected to extremes of temperature and humidity during the deployment.'' Identifying details such as location and year have been replaced with bracketed placeholder tags for anonymity. \parBox 3 is labeled Domain Expert Feedback:'' in bold and reads: The LLM answer is accurate but incomplete. The [TEST] was the primary shakedown test initially conducted for the array, but the LLM doesn't mention this. That's a problem since most of the lessons learned on improving the overall array derived from the [TESTS].'' Again, specific test names are replaced with bracketed placeholders. \parBox 4 is split into two side-by-side cells. The left cell is labeled Domain Expert Codes:'' in bold and reads: Incomplete answer; chronological misunderstanding.'' The phrase chronological misunderstanding'' is visually highlighted with red text, distinguishing it from the plain-text code that precedes it. The right cell is labeled Model Developer Codes:'' in bold and reads: ``lacks detail; crucial information missing.''
ID
Title & Description
Part 1
Part 2
Incorrect Answer
1
Completely incorrect: Answer is entirely wrong.
6
0
2
Partially incorrect: Mix of correct and incorrect information.
-
-
2.a
Basic accuracy issues: High-level concepts correct but details wrong.
8
2
2.b
Misinterpretation: Incorrect understanding of paper content.
10
0
2.c
Self-contradiction: Contains internal logic errors or contradictions.
5
0
Table 1. Typology of incorrect answers and hallucinations found in LLM responses to scholarly questions. The columns Part 1 and Part 2 refer to the number of experts who identified each error during unprimed evaluation and structured evaluation of the validation study, respectively. Asterisks ( ∗ ) indicate additional error types discovered during the schema validation phase. Dashes in column Part 2 indicate error types that were not included in our inventory, as these errors were not identified during the schema development process. Examples are in Appendix C .
ID
Title & Description
Part 1
Part 2
Incomplete Answer
5
Major omissions: Important content or sections are left out of the answer.
-
-
5.a
Ignored whole section: Entire relevant sections of content are overlooked.
4
1
5.b
Missed main point: Core argument or central thesis is not captured.
8
1
5.c
Ignored whole paper: Failed to consider one or more papers in multi-document scenarios.
Table 2. Additional error categories and system failures identified in LLM responses to scholarly questions. Column Part 2 does not have values for system failures because we did not include questions in the inventory that would address these errors.
Figure 2. Distribution of error types across question categories. Each row is normalized to sum to 1, showing the proportion of errors within each question type. Column labels indicate the total occurrences of each error type across all questions ( n ). Question types are sorted by total error frequency (descending); error types are sorted by total occurrences (descending). Darker cells indicate that a given error type accounts for a larger share of errors for that question category. A heatmap titled Errors by Question Type'' displaying the proportional distribution of seven error types across eleven question categories. The y-axis is labeled Question Type'' and the x-axis ``Error Type.'' Cell shading uses a sequential blue color scale shown in a vertical legend on the right, ranging from white at 0.00 to dark navy at values above 0.40. Darker cells indicate that a given error type accounts for a larger share of errors for that question category. Each cell contains its exact numeric proportion. \parThe seven columns, from left to right, are: Incorrect Answer (n=95), Incomplete Answer (n=91), Formatting Issues (n=35), Contains Hallucinations (n=31), Synthesis Issues (n=30), Question Interpretation (n=23), and System Failures (n=23). The eleven rows, from top to bottom, are: Methodological Inquiry and Improvement, Meta-Analysis and Contextualization, Critical Evaluation and Validation, Application and Practical Implications, Technical Details and Specifications, Definitions and Concepts, Future Directions and Extrapolation, Binary / Factual Verification, External Context, Numerical Analysis and Derivation, and Procedural Information. \parThe dominant visual pattern is a dark band spanning the two leftmost columns, Incorrect Answer and Incomplete Answer, which together account for roughly 50 to 75 percent of errors across nearly all question types. Shading fades progressively lighter toward the right-hand columns. The single darkest cell in the heatmap is Procedural Information crossed with Incomplete Answer at 0.44, rendered in deep navy. Definitions and Concepts crossed with Incorrect Answer at 0.41 is comparably dark. Several cells break the otherwise pale right side of the heatmap: Procedural Information shows 0.33 for Synthesis Issues, Meta-Analysis and Contextualization shows 0.23 for Synthesis Issues, Numerical Analysis and Derivation shows 0.30 for System Failures, and Future Directions and Extrapolation shows 0.25 for System Failures—all rendered in darker blue than their neighbors in those columns. Multiple cells are 0.00 and appear white, including four of seven error types for Procedural Information (Formatting Issues, Contains Hallucinations, Question Interpretation, and System Failures), Synthesis Issues for both Definitions and Concepts and External Context, Question Interpretation for Numerical Analysis and Derivation, and System Failures for Binary / Factual Verification.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Common Codes: Incomplete, Lacks detail, Repetitive, Unknown word or improper wording/spelling, Cut off/truncated, Misinterpreted the question, LLM refused to answer stating information not provided, Crucial/key information missing, irrelevant/superfluous, Lack of synthesis, Hallucinated citation, Source confusion; mixes up references and their content, Different interpretation of question, Hallucinated page numbers, Ignored paper in multidocument context, Mixes up papers and their content in multidocument context, Misleading, chronological misunderstanding/out of order, Inaccurate, Incorrect
Unique to Model Developer: Ignored whole section, Referenced wrong equation or math symbols, Missed main point, Misunderstood main point, No issues, Incomplete references, Minor omission, Missed information from table, Incomplete definition, Verbose, Hallucinated table, Mixes up information within document, Missed figure, Sentence IDs are irrelevant
Unique to Domain Experts: Improper referencing, Incorrect notation, Misleading due to ignoring large sections of paper, Misleading due to minor omission, Misleading in important way, Not comprehensive, Partially incorrect, Inconsistent referencing style, Vague/lacks specificity, Inconsistent with itself, Complete, Accurate, Thorough, All papers included in answer
Appendix
Table 3. Open codes from the initial analysis, showing common codes found by both domain experts and model developers, and codes unique to each group. These reflect different priorities in evaluating LLM like ’Referenced wrong equation’ and ’Missed information from table’.
ID
Question
Schema #
Can you evaluate the answer’s completeness using the following questions?
Q1
Are there any major sections from the paper(s) that the system seemed to ignore that would have been necessary for a complete answer?
5.a
Q2
Are there any major topics or points from the paper(s) that the system seemed to ignore that would have been necessary for a complete answer?
5.b
Q3
In a multi-document setting, does the answer include information from all relevant papers mentioned in your question?
5.c
Q4
If the answer defines technical terms, are the definitions complete?
5.d
Q5
If the answer references figures, tables, or cited works, are the references complete?
5.e
Appendix
Table 4. Question inventory for assessing LLM responses to scholarly questions. Column 3 shows which schema item the question maps to.
The identification and localization of errors is a core task in peer review, yet the exponential growth of scientific output has made it increasingly difficult for human reviewers to reliably detect errors given the limited pool of experts. Recent advances in Large Language Models (LLMs) have sparked interest in their potential to support such evaluation tasks, from academic peer review to automated scientific assessment. However, despite the growing use of LLMs in review systems, their capabilities to pinpoint errors remain underexplored. In this work, we introduce Fault Localization Across Writing in Science (FLAWS), an automated benchmark consisting of 713 paper-error pairs designed to evaluate how effectively LLMs detect errors that undermine key claims in research papers. We construct the benchmark by systematically inserting claim-invalidating errors into peer-reviewed papers using LLMs, paired with an automated evaluation metric that measures whether models can identify and localize these errors. Developing such a benchmark presents unique challenges that we overcome: ensuring that the inserted errors are well-defined, challenging, and relevant to the content of the paper, avoiding artifacts that would make identification trivial, and designing a scalable, automated evaluation metric. On the resulting benchmark, we evaluate five frontier LLMs: Claude Sonnet 4.5, DeepSeek Reasoner v3.1, Gemini 2.5 Pro, GPT 5, and Grok 4. Among these, GPT 5 is the top-performing model, achieving 39.1% identification accuracy when k=10, where k is the number of top-ranked error text candidates generated by the LLM.
Large language models (LLMs) are increasingly used in scholarly question-answering (QA) systems to help researchers synthesize vast amounts of literature. However, these systems often produce subtle errors (e.g., unsupported claims, errors of omission), and current provenance mechanisms like source citations are not granular enough for the rigorous verification that scholarly domain requires. To address this, we introduce PaperTrail, a novel interface that decomposes both LLM answers and source documents into discrete claims and evidence, mapping them to reveal supported assertions, unsupported claims, and information omitted from the source texts. We evaluated PaperTrail in a within-subjects study with 26 researchers who performed two scholarly editing tasks using PaperTrail and a baseline interface. Our results show that PaperTrail significantly lowered participants' trust compared to the baseline. However, this increased caution did not translate to behavioral changes, as people continued to rely on LLM-generated scholarly edits to avoid a cognitively burdensome task. We discuss the value of claim-evidence matching for understanding LLM trustworthiness in scholarly settings, and present design implications for cognition-friendly communication of provenance information.
Anna Martin-Boyle, Cara A. C. Leckey, Martha C. Brown +1
University of Minnesota Minneapolis, Minnesota, USA · NASA Langley Research Center Hampton, Virginia, USA
Large Language Models (LLMs) are increasingly described as performing at the level of human experts on knowledge economy tasks. These claims are primarily based on how LLMs perform on benchmarking tasks that measure average performance across standardized datasets. Primary limitations of many benchmarking tasks are that they often measure performance based on content directly included in LLM training data, and they frequently do not assess the reliability of LLM performance or the magnitude of LLM errors. However, in high stakes contexts, these qualities are critically important. Through a novel LLM benchmarking task that requires writing computer code to complete a data analysis task, we compare the performance of a frontier LLM against submissions from human experts and explicitly measure the variance of responses and the magnitude of errors. Our study reveals that the human experts perform better on average on a range of metrics and demonstrate less variability in performance. Our results provide evidence that LLMs do not consistently perform at the level of human experts and demonstrate the importance of measuring variance and assessing error magnitude in LLM benchmark evaluations.