PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&A

Anna Martin-Boyle; Cara A. C. Leckey; Martha C. Brown; Harmanpreet Kaur

cs.HCFeb 24, 2026

PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&A

Authors: Anna Martin-Boyle, Cara A. C. Leckey, Martha C. Brown, Harmanpreet Kaur

Organizations: University of Minnesota Minneapolis, Minnesota, USA · NASA Langley Research Center Hampton, Virginia, USA

Abstract

Large language models (LLMs) are increasingly used in scholarly question-answering (QA) systems to help researchers synthesize vast amounts of literature. However, these systems often produce subtle errors (e.g., unsupported claims, errors of omission), and current provenance mechanisms like source citations are not granular enough for the rigorous verification that scholarly domain requires. To address this, we introduce PaperTrail, a novel interface that decomposes both LLM answers and source documents into discrete claims and evidence, mapping them to reveal supported assertions, unsupported claims, and information omitted from the source texts. We evaluated PaperTrail in a within-subjects study with 26 researchers who performed two scholarly editing tasks using PaperTrail and a baseline interface. Our results show that PaperTrail significantly lowered participants' trust compared to the baseline. However, this increased caution did not translate to behavioral changes, as people continued to rely on LLM-generated scholarly edits to avoid a cognitively burdensome task. We discuss the value of claim-evidence matching for understanding LLM trustworthiness in scholarly settings, and present design implications for cognition-friendly communication of provenance information.

Figures & tables

Figure 1. PaperTrail system architecture showing the three-stage pipeline for argument-grounded provenance. Paper PDFs and user-LLM dialog feed into the Argument Extraction Engine, which combines three different extraction methods deployed strategically across pipeline stages based on design-time tradeoffs between computational cost and semantic capability. The system produces claim-evidence annotations that power the PaperTrail interface for scholarly question-answering. A system architecture diagram showing PaperTrail's three-stage pipeline for argument-grounded provenance. On the left, two inputs feed into the system: source document PDFs (represented as stacked document icons) and user-LLM dialog (shown as a chat interface). These flow into a central box labeled ``Argument Extraction Engine,'' which contains three extraction method icons arranged left to right: similarity-based extraction, RAG-based extraction, and LLM-based extraction. The engine outputs "Claim-Evidence Annotations," depicted as annotated document cards, which feed into the PaperTrail Interface on the right. The interface is shown as a multi-panel application with a ``Claim Coverage'' indicator bar in the upper right corner and a question input field at the bottom. — Figure 1. PaperTrail system architecture showing the three-stage pipeline for argument-grounded provenance. Paper PDFs and user-LLM dialog feed into the Argument Extraction Engine, which combines three different extraction methods deployed strategically across pipeline stages based on design-time tradeoffs between computational cost and semantic capability. The system produces claim-evidence annotations that power the PaperTrail interface for scholarly question-answering. A system architecture diagram showing PaperTrail's three-stage pipeline for argument-grounded provenance. On the left, two inputs feed into the system: source document PDFs (represented as stacked document icons) and user-LLM dialog (shown as a chat interface). These flow into a central box labeled `Argument Extraction Engine,'' which contains three extraction method icons arranged left to right: similarity-based extraction, RAG-based extraction, and LLM-based extraction. The engine outputs "Claim-Evidence Annotations," depicted as annotated document cards, which feed into the PaperTrail Interface on the right. The interface is shown as a multi-panel application with a` Claim Coverage'' indicator bar in the upper right corner and a question input field at the bottom.

Figure 2. The Argument Extraction Engine (top) provides three extraction methods with different computational cost and accuracy tradeoffs. Colors follow a computational cost gradient: red indicates LLM-based extraction (highest computational cost), gold represents RAG-based extraction (medium cost), and teal denotes similarity-based extraction (lowest cost). We deploy these methods strategically across pipeline stages based on design-time considerations: (1) offline paper-level information extraction that preprocesses research papers into structured claims and evidence; (2) real-time answer-level extraction that decomposes LLM-generated answers into claims and supporting evidence; and (3) real-time claim-evidence matching that uses retrieval-augmented generation (RAG) to filter and align relevant paper claims with answer claims, producing source provenance indicators. A detailed diagram of the Argument Extraction Engine's three-stage pipeline, organized as a matrix. The top row shows the three extraction methods arranged along a horizontal axis labeled ``Computational Cost \& Semantic Understanding'' from low to high: similarity-based extraction (teal), RAG-based extraction (gold), and LLM-based extraction (red). Below, three rows represent the pipeline stages. Stage 1 (Offline Paper Processing) shows paper PDFs undergoing text and metadata extraction, then LLM-based claim extraction (red), followed by similarity-based evidence extraction (teal), producing a paper claims and evidence corpus. Stage 2 (Real-Time Answer Processing) shows an LLM answer undergoing LLM-based claim and evidence extraction (red), producing answer claims and evidence. Stage 3 (Real-Time Claim Matching) takes the corpus, user query, and answer claims as inputs; applies RAG-based extraction (gold) to extract relevant claims and evidence; then matches claims and evidence to produce output for the PaperTrail interface. Color coding throughout follows the computational cost gradient: teal for lowest cost, gold for medium, and red for highest. — Figure 2. The Argument Extraction Engine (top) provides three extraction methods with different computational cost and accuracy tradeoffs. Colors follow a computational cost gradient: red indicates LLM-based extraction (highest computational cost), gold represents RAG-based extraction (medium cost), and teal denotes similarity-based extraction (lowest cost). We deploy these methods strategically across pipeline stages based on design-time considerations: (1) offline paper-level information extraction that preprocesses research papers into structured claims and evidence; (2) real-time answer-level extraction that decomposes LLM-generated answers into claims and supporting evidence; and (3) real-time claim-evidence matching that uses retrieval-augmented generation (RAG) to filter and align relevant paper claims with answer claims, producing source provenance indicators. A detailed diagram of the Argument Extraction Engine's three-stage pipeline, organized as a matrix. The top row shows the three extraction methods arranged along a horizontal axis labeled ``Computational Cost & Semantic Understanding'' from low to high: similarity-based extraction (teal), RAG-based extraction (gold), and LLM-based extraction (red). Below, three rows represent the pipeline stages. Stage 1 (Offline Paper Processing) shows paper PDFs undergoing text and metadata extraction, then LLM-based claim extraction (red), followed by similarity-based evidence extraction (teal), producing a paper claims and evidence corpus. Stage 2 (Real-Time Answer Processing) shows an LLM answer undergoing LLM-based claim and evidence extraction (red), producing answer claims and evidence. Stage 3 (Real-Time Claim Matching) takes the corpus, user query, and answer claims as inputs; applies RAG-based extraction (gold) to extract relevant claims and evidence; then matches claims and evidence to produce output for the PaperTrail interface. Color coding throughout follows the computational cost gradient: teal for lowest cost, gold for medium, and red for highest.

Figure 3. The user interface comprises three main panels. The general layout, shown at the top, consists of the Left Panel (A) , the Middle Panel (B) , and the Right Panel (C) . The Left Panel (A) contains the user’s main workspace, including the Task Context (A1) , a References List (A2) , and the Text Editor (A3) . The Middle Panel (B) serves as the Chat Interface (B1) , which includes a Question Bank (B2) and Chat Controls (B3) . The Right Panel (C) displays information provenance, with its content changing based on the condition. The lower half shows the differences between the conditions. In the PaperTrail interface (left), the Middle Panel (B) shows interactive Answer Claims (B4) . When a user clicks a claim, the corresponding Paper Claim (C2) is highlighted in the Right Panel (C) , which also shows the overall Claim Coverage (C1) for the LLM’s answer. In the baseline interface (right), the Middle Panel (B) contains a sentence-level Source Highlight (B5) . Clicking this highlight surfaces the verbatim Paper Source (C3) text in the Right Panel (C) . A composite screenshot of the PaperTrail user interface in three parts. The top portion shows the shared three-panel layout: the Left Panel (A) contains the task workspace with task context instructions (A1), a references list of four Mars exploration papers (A2), and a text editor with an LLM-generated draft (A3); the Middle Panel (B) shows the chat interface (B1) with a conversation history, a question bank dropdown (B2), and a send button (B3); the Right Panel (C) is a placeholder for provenance information. The bottom portion shows side-by-side comparisons of the two study conditions. On the left, the PaperTrail condition displays answer text with highlighted interactive answer claims (B4) in the middle panel, while the right panel shows a ``Claim Coverage'' bar indicating 3 of 5 relevant claims matched (C1), followed by paper claim cards (C2) organized into ``Claims included in answer'' (teal cards) and ``Claims omitted from answer'' (red cards), each with expandable evidence sections. A selected claim is highlighted with a yellow vertical bar. On the right, the baseline condition shows sentence-level source highlights (B5) in the middle panel with a tooltip prompting users to click to see sources; the right panel displays verbatim paper source text (C3) organized by citation under expandable dropdown sections. — Figure 3. The user interface comprises three main panels. The general layout, shown at the top, consists of the Left Panel (A) , the Middle Panel (B) , and the Right Panel (C) . The Left Panel (A) contains the user’s main workspace, including the Task Context (A1) , a References List (A2) , and the Text Editor (A3) . The Middle Panel (B) serves as the Chat Interface (B1) , which includes a Question Bank (B2) and Chat Controls (B3) . The Right Panel (C) displays information provenance, with its content changing based on the condition. The lower half shows the differences between the conditions. In the PaperTrail interface (left), the Middle Panel (B) shows interactive Answer Claims (B4) . When a user clicks a claim, the corresponding Paper Claim (C2) is highlighted in the Right Panel (C) , which also shows the overall Claim Coverage (C1) for the LLM’s answer. In the baseline interface (right), the Middle Panel (B) contains a sentence-level Source Highlight (B5) . Clicking this highlight surfaces the verbatim Paper Source (C3) text in the Right Panel (C) . A composite screenshot of the PaperTrail user interface in three parts. The top portion shows the shared three-panel layout: the Left Panel (A) contains the task workspace with task context instructions (A1), a references list of four Mars exploration papers (A2), and a text editor with an LLM-generated draft (A3); the Middle Panel (B) shows the chat interface (B1) with a conversation history, a question bank dropdown (B2), and a send button (B3); the Right Panel (C) is a placeholder for provenance information. The bottom portion shows side-by-side comparisons of the two study conditions. On the left, the PaperTrail condition displays answer text with highlighted interactive answer claims (B4) in the middle panel, while the right panel shows a `Claim Coverage'' bar indicating 3 of 5 relevant claims matched (C1), followed by paper claim cards (C2) organized into` Claims included in answer'' (teal cards) and ``Claims omitted from answer'' (red cards), each with expandable evidence sections. A selected claim is highlighted with a yellow vertical bar. On the right, the baseline condition shows sentence-level source highlights (B5) in the middle panel with a tooltip prompting users to click to see sources; the right panel displays verbatim paper source text (C3) organized by citation under expandable dropdown sections.

Measure	Baseline Mean	PaperTrail Mean	Statistic	Effect Size
Primary Factors
Trust	4.22 $\pm 1.22$	3.68 $\pm 1.24$	$t=2.61,\ p=.015\ast$	0.44
Reliance	0.73 $\pm 0.21$	0.75 $\pm 0.29$	$W=146.00,\ p=.313$	-0.23
Confidence	4.27 $\pm 1.51$	4.05 $\pm 1.58$	$t=0.64,\ p=.525$	0.14
Experiental Factors
Cognitive Load	4.10 $\pm 0.83$	4.12 $\pm 0.78$	$t=-0.21,\ p=0.838$	0.03

Table 1. Comparison of measures between baseline and PaperTrail conditions.

	Ed.	Exp.	Freq.	Gender	Conf.	Rel.	Trust	Cog. Load	Usability	Clicks	Messages
Demographics
Age	0.47*	0.79***	0.19	-0.04	-0.03	0.12	-0.08	-0.29	-0.08	-0.09	-0.02
Education		0.53**	0.10	-0.16	0.07	0.18	0.02	-0.11	-0.05	-0.07	0.16
Experience			0.10	-0.20	-0.01	0.02	-0.01	-0.21	-0.10	0.04	0.14
Frequency				0.06	0.05	0.13	0.00	0.27	0.06	-0.06	0.27
Gender					-0.07	0.34	-0.10	0.16	0.03	0.13	0.05

Table 2. Spearman correlations in the baseline condition. Education refers to level of educational attainment, Experience refers to years of experience, Frequency refers to frequency of AI use, and Clicks and Messages refer to the total counts of each.

	Ed.	Exp.	Freq.	Gender	Conf.	Rel.	Trust	Cog. Load	Usability	Clicks	Messages
Demographics
Age	0.47*	0.79***	0.19	-0.04	-0.20	0.00	0.03	-0.18	0.23	-0.38	-0.28
Education		0.53**	0.10	-0.16	-0.41*	0.25	0.01	-0.15	-0.08	0.04	-0.24
Experience			0.10	-0.20	-0.29	0.22	-0.01	-0.12	0.07	-0.20	-0.23
Frequency				0.06	-0.19	0.11	-0.12	0.17	-0.09	-0.11	0.08
Gender					0.03	0.14	-0.29	0.13	0.05	-0.28	-0.11

Table 3. Spearman correlations in the PaperTrail condition. Education refers to level of educational attainment, Experience refers to years of experience, Frequency refers to frequency of AI use, and Clicks and Messages refer to the total counts of each.

LLM Answer Verification

Large language model answer verification checks whether generated answers are correct, supported by evidence, and safe to accept.

See all 148 papers

Quantifying Claim Smoothing by LLM Research Agents
Oct 26, 2026Tiasa Singha Roy, Manya Wadhwa, Greg DurrettLLM Reliability LLM Agent Reliability
New York University
An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems
Feb 24, 2026Anna Martin-Boyle, William Humphreys, Martha Brown +2LLM Evaluation Language Model Error Detection
University of Minnesota Minneapolis, United States · NASA Langley Research Center Hampton, Virginia, United States
CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
May 13, 2026Dongsheng Ma, Jiayu Li, Zhengren Wang +9VLM Evaluation Citation Verification
Peking University · Shanghai Artificial Intelligence Laboratory

Oct 26, 2026J/K move · Enter open · S save

Quantifying Claim Smoothing by LLM Research Agents

Tiasa Singha Roy, Manya Wadhwa, Greg Durrett

New York University

LLM Reliability LLM Agent Reliability Claim Attribution in Language Models

Open paper AskPDF arXiv

Deep research, or any use of large language models (LLMs) to read a large document collection and produce insights, requires synthesizing information from different sources. In this work, we find that LLMs produce long-form research reports that read as confident syntheses but which fail to accurately reflect the underlying documents. We call this failure smoothing. In collapsing diverse and potentially conflicting sources into a single narrative, the model discards the evidential "jaggedness" of its corpus and asserts conclusions more decisive than the citations can support. We introduce a claim level scoring pipeline, SmoothScore, to quantify this effect. Our pipeline extracts key conclusive claims from model generated reports, then finds evidence relevant to each claim and clusters that evidence along high level aspects. Finally, each claim is scored with respect to the evidence to quantify the degree of smoothing done by the agent. Across 7 systems and 4 datasets, we find that agents consistently produce stronger claims than warranted.

Abstract

LLM Answer Verification

Explore similar work

Quantifying Claim Smoothing by LLM Research Agents

An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence