cs.HCFeb 24, 2026
SaveAn Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems
Organizations: University of Minnesota Minneapolis, United States · NASA Langley Research Center Hampton, Virginia, United States
Abstract
Large Language Models (LLMs) are transforming scholarly tasks like search and summarization, but their reliability remains uncertain. Current evaluation metrics for testing LLM reliability are primarily automated approaches that prioritize efficiency and scalability, but lack contextual nuance and fail to reflect how scientific domain experts assess LLM outputs in practice. We developed and validated a schema for evaluating LLM errors in scholarly question-answering systems that reflects the assessment strategies of practicing scientists. In collaboration with domain experts, we identified 20 error patterns across seven categories through thematic analysis of 68 question-answer pairs. We validated this schema through contextual inquiries with 10 additional scientists, which showed not only which errors experts naturally identify but also how structured evaluation schemas can help them detect previously overlooked issues. Domain experts use systematic assessment strategies, including technical precision testing, value-based evaluation, and meta-evaluation of their own practices. We discuss implications for supporting expert evaluation of LLM outputs, including opportunities for personalized, schema-driven tools that adapt to individual evaluation patterns and expertise levels.
Figures & tables
Figure 1. Errors identified by domain experts and model developers with entity tags for anonymity. The expert recognized a chronological error about test sequences that the developer missed, showing how domain expertise can yield more precise error analysis. A worked example of the open coding process, presented as four labeled boxes stacked vertically, each outlined with a thin green border and separated by white space. The figure is primarily textual. \parBox 1 is labeled
Question:'' in bold and reads: How was shake-down testing of the microphone array conducted prior to deployment in a production environment?'' \parBox 2 is labeled LLM Output:'' in bold and reads: The shake-down testing of the microphone array was conducted at [LOCATION] in [YEAR] prior to its deployment in a production environment. During this testing, the array's operation was nominal with only 8 microphones out of 185 exhibiting problems that required replacement with backup units. The sensors were subjected to extremes of temperature and humidity during the deployment.'' Identifying details such as location and year have been replaced with bracketed placeholder tags for anonymity. \parBox 3 is labeled Domain Expert Feedback:'' in bold and reads: The LLM answer is accurate but incomplete. The [TEST] was the primary shakedown test initially conducted for the array, but the LLM doesn't mention this. That's a problem since most of the lessons learned on improving the overall array derived from the [TESTS].'' Again, specific test names are replaced with bracketed placeholders. \parBox 4 is split into two side-by-side cells. The left cell is labeled Domain Expert Codes:'' in bold and reads: Incomplete answer; chronological misunderstanding.'' The phrase chronological misunderstanding'' is visually highlighted with red text, distinguishing it from the plain-text code that precedes it. The right cell is labeled Model Developer Codes:'' in bold and reads: ``lacks detail; crucial information missing.''| ID | Title & Description | Part 1 | Part 2 |
| Incorrect Answer | |||
| 1 | Completely incorrect: Answer is entirely wrong. | 6 | 0 |
| 2 | Partially incorrect: Mix of correct and incorrect information. | - | - |
| 2.a | Basic accuracy issues: High-level concepts correct but details wrong. | 8 | 2 |
| 2.b | Misinterpretation: Incorrect understanding of paper content. | 10 | 0 |
| 2.c | Self-contradiction: Contains internal logic errors or contradictions. | 5 | 0 |
Table 1. Typology of incorrect answers and hallucinations found in LLM responses to scholarly questions. The columns Part 1 and Part 2 refer to the number of experts who identified each error during unprimed evaluation and structured evaluation of the validation study, respectively. Asterisks ( ) indicate additional error types discovered during the schema validation phase. Dashes in column Part 2 indicate error types that were not included in our inventory, as these errors were not identified during the schema development process. Examples are in Appendix C .
| ID | Title & Description | Part 1 | Part 2 |
| Incomplete Answer | |||
| 5 | Major omissions: Important content or sections are left out of the answer. | - | - |
| 5.a | Ignored whole section: Entire relevant sections of content are overlooked. | 4 | 1 |
| 5.b | Missed main point: Core argument or central thesis is not captured. | 8 | 1 |
| 5.c | Ignored whole paper: Failed to consider one or more papers in multi-document scenarios. | 1 | 2 |
| 5.d | Incomplete definition: Definition lacks crucial information. | 6 | 1 |
Table 2. Additional error categories and system failures identified in LLM responses to scholarly questions. Column Part 2 does not have values for system failures because we did not include questions in the inventory that would address these errors.
Figure 2. Distribution of error types across question categories. Each row is normalized to sum to 1, showing the proportion of errors within each question type. Column labels indicate the total occurrences of each error type across all questions ( ). Question types are sorted by total error frequency (descending); error types are sorted by total occurrences (descending). Darker cells indicate that a given error type accounts for a larger share of errors for that question category. A heatmap titled
Errors by Question Type'' displaying the proportional distribution of seven error types across eleven question categories. The y-axis is labeled Question Type'' and the x-axis ``Error Type.'' Cell shading uses a sequential blue color scale shown in a vertical legend on the right, ranging from white at 0.00 to dark navy at values above 0.40. Darker cells indicate that a given error type accounts for a larger share of errors for that question category. Each cell contains its exact numeric proportion. \parThe seven columns, from left to right, are: Incorrect Answer (n=95), Incomplete Answer (n=91), Formatting Issues (n=35), Contains Hallucinations (n=31), Synthesis Issues (n=30), Question Interpretation (n=23), and System Failures (n=23). The eleven rows, from top to bottom, are: Methodological Inquiry and Improvement, Meta-Analysis and Contextualization, Critical Evaluation and Validation, Application and Practical Implications, Technical Details and Specifications, Definitions and Concepts, Future Directions and Extrapolation, Binary / Factual Verification, External Context, Numerical Analysis and Derivation, and Procedural Information. \parThe dominant visual pattern is a dark band spanning the two leftmost columns, Incorrect Answer and Incomplete Answer, which together account for roughly 50 to 75 percent of errors across nearly all question types. Shading fades progressively lighter toward the right-hand columns. The single darkest cell in the heatmap is Procedural Information crossed with Incomplete Answer at 0.44, rendered in deep navy. Definitions and Concepts crossed with Incorrect Answer at 0.41 is comparably dark. Several cells break the otherwise pale right side of the heatmap: Procedural Information shows 0.33 for Synthesis Issues, Meta-Analysis and Contextualization shows 0.23 for Synthesis Issues, Numerical Analysis and Derivation shows 0.30 for System Failures, and Future Directions and Extrapolation shows 0.25 for System Failures—all rendered in darker blue than their neighbors in those columns. Multiple cells are 0.00 and appear white, including four of seven error types for Procedural Information (Formatting Issues, Contains Hallucinations, Question Interpretation, and System Failures), Synthesis Issues for both Definitions and Concepts and External Context, Question Interpretation for Numerical Analysis and Derivation, and System Failures for Binary / Factual Verification.Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Common Codes: Incomplete, Lacks detail, Repetitive, Unknown word or improper wording/spelling, Cut off/truncated, Misinterpreted the question, LLM refused to answer stating information not provided, Crucial/key information missing, irrelevant/superfluous, Lack of synthesis, Hallucinated citation, Source confusion; mixes up references and their content, Different interpretation of question, Hallucinated page numbers, Ignored paper in multidocument context, Mixes up papers and their content in multidocument context, Misleading, chronological misunderstanding/out of order, Inaccurate, Incorrect |
|---|
| Unique to Model Developer: Ignored whole section, Referenced wrong equation or math symbols, Missed main point, Misunderstood main point, No issues, Incomplete references, Minor omission, Missed information from table, Incomplete definition, Verbose, Hallucinated table, Mixes up information within document, Missed figure, Sentence IDs are irrelevant |
| Unique to Domain Experts: Improper referencing, Incorrect notation, Misleading due to ignoring large sections of paper, Misleading due to minor omission, Misleading in important way, Not comprehensive, Partially incorrect, Inconsistent referencing style, Vague/lacks specificity, Inconsistent with itself, Complete, Accurate, Thorough, All papers included in answer |
Table 3. Open codes from the initial analysis, showing common codes found by both domain experts and model developers, and codes unique to each group. These reflect different priorities in evaluating LLM like ’Referenced wrong equation’ and ’Missed information from table’.
| ID | Question | Schema # |
| Can you evaluate the answer’s completeness using the following questions? | ||
| Q1 | Are there any major sections from the paper(s) that the system seemed to ignore that would have been necessary for a complete answer? | 5.a |
| Q2 | Are there any major topics or points from the paper(s) that the system seemed to ignore that would have been necessary for a complete answer? | 5.b |
| Q3 | In a multi-document setting, does the answer include information from all relevant papers mentioned in your question? | 5.c |
| Q4 | If the answer defines technical terms, are the definitions complete? | 5.d |
| Q5 | If the answer references figures, tables, or cited works, are the references complete? | 5.e |
Table 4. Question inventory for assessing LLM responses to scholarly questions. Column 3 shows which schema item the question maps to.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers