Quantifying Claim Smoothing by LLM Research Agents

Tiasa Singha Roy; Manya Wadhwa; Greg Durrett

cs.CLOct 26, 2026

Quantifying Claim Smoothing by LLM Research Agents

Authors: Tiasa Singha Roy, Manya Wadhwa, Greg Durrett

Organizations: New York University

Abstract

Deep research, or any use of large language models (LLMs) to read a large document collection and produce insights, requires synthesizing information from different sources. In this work, we find that LLMs produce long-form research reports that read as confident syntheses but which fail to accurately reflect the underlying documents. We call this failure smoothing. In collapsing diverse and potentially conflicting sources into a single narrative, the model discards the evidential "jaggedness" of its corpus and asserts conclusions more decisive than the citations can support. We introduce a claim level scoring pipeline, SmoothScore, to quantify this effect. Our pipeline extracts key conclusive claims from model generated reports, then finds evidence relevant to each claim and clusters that evidence along high level aspects. Finally, each claim is scored with respect to the evidence to quantify the degree of smoothing done by the agent. Across 7 systems and 4 datasets, we find that agents consistently produce stronger claims than warranted.

Figures & tables

Figure 1: When synthesizing over many documents, agents can present a claim as well supported when the evidence is actually more mixed. SmoothScore flags these smoothed-over claims and quantifies the degree of smoothing.

Figure 2: Overview of SmoothScore . Given a query and a system generated report, we (1) first extract conclusive claims. Then from the cited documents, (2) we get supporting and skeptic evidence relevant to the claim. (3)/(4) We then organize the evidence into aspects . (5) The claim is then scored with respect to the gathered evidence to quantify the degree of missing information.

Dataset	System	Reports	Claims	Mean [95% CI]	1	2	3	4	5
		Sample		SmoothScore	% of claims w/given SmoothScore
ResearcherBench	Luna 5.6	65	443	1.95 [1.85, 2.06]	37.0%	34.1%	25.7%	3.2%	0.0%
	Sol 5.6	64	430	1.93 [1.81, 2.05]	37.0%	35.6%	25.1%	2.3%	0.0%
	Sonnet 5	65	435	2.52 [2.42, 2.61]	13.8%	26.7%	53.3%	6.2%	0.0%
	Opus 5.5	65	445	2.36 [2.28, 2.44]	19.1%	29.4%	47.9%	3.6%	0.0%
	QUEST	63	404	2.50 [2.35, 2.65]	18.1%	26.7%	43.6%	10.1%	1.5%

Table 1: We report the number of reports, claims for each of systems and benchmarks evaluated in this work. We also report the SmoothScore with

\geq

3 being materially smoothed and

\geq

4 being the subset of severely smoothed claims. Open source systems like, DR Tulu and QUEST produce a large % of severely smoothed claims, while closed source agents vary with Luna and Sol systems only having 1-4% claims

\geq 4

against Gemini’s 22-35%.

Figure 3: Relationship between cited document count and claim smoothing. Across system and dataset pairs, we find that mean SmoothScore increases with the mean number of documents cited per report, but within a system this effect disappears (Table 12 ).

Pattern	Claim	Summarized Evidence
Unsupported composition	The aggressive deployment of duplication-aware token pruning , Modality-Aware Mixture of Experts , and adaptive halting ensures that vast, world-class parameter capacities can be maintained without triggering corresponding explosions in inference latency. Score: 3	Token pruning, sparse expert routing, and adaptive halting can each reduce inference cost in particular settings. However, the evidence does not evaluate their simultaneous deployment or show that their benefits compose. Each can introduce routing, pruning, synchronization, or kernel overheads, so the combined system does not guarantee bounded latency at frontier scale.
Scope expansion	Self-attention-based models are highly parallelizable and require substantially fewer parameters, making them much more computationally efficient , less prone to overfitting , and easier to fine-tune for domain-specific tasks … Score: 4	Some ViTs achieve strong performance with lower pre-training compute in particular comparisons, but these advantages do not generalize across models or regimes. ViTs often have comparable or larger parameter counts, global attention has quadratic cost, and ViTs can be more prone to overfitting in low-data settings without large-scale pre-training or careful regularization.
Premature closure	This innovation fundamentally resolves the statistical heterogeneity (Non-IID) problem while strictly preserving the privacy boundaries established by institutional firewalls. Score: 5	Federated distillation can mitigate heterogeneity and reduce direct data sharing, but it does not eliminate either problem. Performance continues to degrade as heterogeneity increases, theoretical analyses retain residual bias, and privacy attacks remain possible.

Table 2: Examples of recurring claim-smoothing patterns. For each pattern, we show an illustrative claim, its SmoothScore , and a summary of the supporting evidence and limitations.

	Exact	W/in 1	$\bm{\rho}$	$\bm{\alpha}$
Annotator 1 vs. Annotator 2	58%	96%	0.72	0.71
Judge vs. Annotator 1	56%	93%	0.69	0.68

Table 3: Agreement on SmoothScore across 73 claims scored by two human annotators.

	Luna			Sonnet 5
Condition	Context	Length	$\bm{\Delta}$ ( $p$ )	Context	Length	$\bm{\Delta}$ ( $p$ )
WebSearch draft (score)	–	$1.0\times$	68.3	–	$1.0\times$	64.4
Revise, no context	–	$1.6\times$	$+0.4$ ( $.22$ )	–	$1.5\times$	$+1.1$ ( $.38$ )
Revise + cited documents	117k	$2.2\times$	$+4.1$ ( $.004$ )	324k	$2.4\times$	$+9.7$ ( $<.001$ )
Revise + SmoothScore evidence	11k	$2.0\times$	$+4.4$ ( $<.001$ )	65k	$2.9\times$	$+12.0$ ( $<.001$ )

Table 4: Revising reports generated by Luna and Sonnet 5 on ResearcherBench, scored against gold rubrics by GPT-5. Simply providing evidence can increase rubric scores, matching or exceeding providing raw document while being a fraction of the size

	Luna	Sonnet 5
Cited documents	496	1,458
Opened	95 (19%)	34 (2%)
Not opened	401 (81%)	1,424 (98%)
Reports opening no page	45 / 65	44 / 65

Table 5: Luna and sonnet-5 websearch system engagement with the cited documents on ResearcherBench. Luna’s opens are marked by a header and Sonnet 5’s opens are explicit web fetches returning the full page. Details in Appendix F .

Model	Condition	Length	Claims	Mean [95% CI]	1	2	3	4	5
					% of claims with SmoothScore
Luna	WebSearch	1.0 $\times$	416	1.95 [1.84, 2.07]	37.3	33.4	26.0	3.4	0.0
	WebSearch + Instr.	1.0 $\times$	397	1.88 [1.78, 1.98]	40.3	34.0	23.2	2.5	0.0
	Docs	1.6 $\times$	390	1.80 [1.70, 1.91]	45.6	31.0	21.3	2.1	0.0
	Docs + Instr.	1.4 $\times$	378	1.52 [1.44, 1.60]	61.1	26.7	11.6	0.5	0.0
Sonnet 5	WebSearch	1.0 $\times$	407	2.54 [2.44, 2.63]	13.5	25.6	54.8	6.1	0.0

Table 6: Effect of document engagement and calibration instructions on smoothing (ResearcherBench, 61 questions). WebSearch : original reports. Docs : the same agent given the full text of the documents its WebSearch report cited, with web search off. + Instr. : the same calibration instructions added to either prompt. Providing both shows the largest improvement for both setups.

Condition	Claim
WebSearch SmoothScore: 4	Protection should be soft and probabilistic, not permanently frozen: an important parameter may eventually be revised if sufficient contradictory evidence accumulates.
Judge rationale	Requires recasting: the prescription excludes hard freezing and non-probabilistic approaches despite evidence supporting their effectiveness.
Docs + instructions SmoothScore: 1	Previously useful parameters or subnetworks should be protected from destructive updates. This can be implemented through frozen columns, masks, structural isolation, or soft importance-weighted constraints [doc 2][doc 3][doc 5].

Table 7: Claim improved under the docs+instruction case. Here the question pertains to how continual-learning systems can preserve previously learned knowledge while remaining adaptable to new tasks.The judge’s verdict is shown on the WebSearch claim.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Stage	Model	Prompt
1	Claim extraction	GPT-5-mini	App. B.1
2	Evidence extraction: support agent	GPT-5-mini	App. B.2
2	Evidence extraction: skeptic agent	GPT-5-mini	App. B.3
3	Aspect induction: per-document proposals	GPT-5-mini	App. B.4
3	Aspect induction: cross-document merge	GPT-5-mini	App. B.5
4	Evidence assignment	GPT-5-mini	App. B.6

Appendix

Table 8: Models used at each stage of the SmoothScore pipeline. All stages use GPT-5-mini except the final judge, which uses GPT-5. Prompts for each stage are given in the referenced appendix sections.

Benchmark	Candidates	Kept
DeepResearch Bench II (English)	66	18
ResearchRubrics	101	17

Appendix

Table 9: Query selection for DeepResearch Bench II and ResearchRubrics.

System	Model ID	Access
GPT Luna + Websearch	gpt-5.6-luna	OpenAI Responses API
GPT Sol + Websearch	gpt-5.6-sol	OpenAI Responses API
Claude Sonnet 5 + Websearch	claude-sonnet-5	Anthropic Messages API
Claude Opus 5 + Websearch	claude-opus-5-5	Anthropic Messages API
Gemini Deep Research	deep-research-preview-04-2026	Gemini Interactions API
DR Tulu	rl-research/DR-Tulu-8B	local (vLLM + MCP server)

Appendix

Table 10: Model identifiers and access method for each evaluated system.

Dataset	System	Report words	Docs/report	Evidence words
ResearcherBench	Luna 5.6	1,828	8.6	77,898
	Sol 5.6	1,710	9.9	92,880
	Sonnet 5	2,125	21.6	155,148
	Opus 5.5	1,328	18.8	138,441
	QUEST	1,332	9.8	55,481
	DR Tulu	1,948	17.4	114,027

Appendix

Table 11: Report length and retrieved evidence by dataset and system, over the reports scored by the SmoothScore judge. “Report words” and “Evidence words” are medians over reports; “Docs/report” is the mean number of retrieved documents per report. “Evidence words” is the total number of words in the retrieved documents available to the system.

A. Pooled
	Across systems		Within system	Raw pooled
Variable	$\rho$	$r$	$\rho$	$\rho$
Cited documents	$+0.37$ ( $.056$ )	$+0.59$ ( $.001$ )	$+0.03$ ( $.35$ )	$+0.38$
Report length	$+0.24$ ( $.23$ )	$+0.39$ ( $.04$ )	$+0.12$ ( $.001$ )	$+0.33$
Number of claims	$+0.08$ ( $.70$ )	$+0.10$ ( $.63$ )	$-0.02$ ( $.61$ )	$0.00$
Claim length	$+0.43$ ( $.02$ )	$+0.56$ ( $.002$ )	$+0.02$ ( $.16$ )	$+0.12$

Appendix

Table 12: Correlation of report and claim level features with SmoothScore across and within systems (Spearman

\rho

and Pearson

r

). Across : the mean over the 28 system–dataset. Within : after subtracting each system–dataset mean. Systems that cite more documents or write longer claims smooth more, but within a system such relationships are weak and vary in direction. Only report length shows a small, consistent within-system association (Panel B)

Judge	Exact	Within 1	Spearman $\rho$	Krippendorff $\alpha$
Annotator 2	58%	96%	0.72	0.71
GPT-5 (judge of record)	56%	93%	0.69	0.68
Gemini 2.5 Pro	27%	74%	0.63	0.41
Claude Sonnet 4.5	19%	56%	0.51	0.11

Appendix

Table 13: Agreement with Annotator 1 on 73 claims (Krippendorff’s

\alpha

is ordinal). Exact : identical score. Within 1 : scores differ by at most one point. GPT-5 agrees with Annotator 1 as closely as the second annotator does; other judges rank claims similarly but differ in calibration.

System	Annotator 1	GPT-5	Gemini 2.5 Pro	Sonnet 4.5
Luna	1.96	1.71	2.33	2.88
DR Tulu	2.16	2.60	2.96	3.32
Gemini Deep Research	2.71	3.04	3.83	4.42

Appendix

Table 14: Mean score per system on the 73 annotated claims, by judge. All judges rank the systems in the same order as the annotator.

Pipeline step	Output for the example
Input question	Will RL-based methods become the mainstream approach for improving reasoning models after DeepSeek-R1, and can reward models and tree search for chain-of-thought post-training be abandoned?
1. Claim extraction	“RL—especially RL with verifiable rewards—will become a mainstream component of reasoning-model training.”
Document reconstruction	The system reconstructs the full text of the documents cited by the report, including work on DeepSeek-R1, verifier gaming, test-time scaling, AlphaProof, AlphaGeometry2, and surveys of RL for reasoning.
2a. Supporting-evidence extraction	The supporting pass extracts findings that substantiate the claim as stated. For example: “Figure 1(a) depicts the performance trajectory of DeepSeek-R1-Zero on the AIME 2024 benchmark throughout the RL training process, where the average pass@1 score on AIME 2024 shows a significant increase, jumping from an initial 15.6% to 77.9%.” “As reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for scaling reasoning capabilities in LLMs.” Across all claim–document pairs, the pass extracts $61$ supporting items.
2b. Countervailing-evidence extraction	The countervailing pass extracts findings that contradict, weaken, or materially restrict the claim. For example: “We show that training on only 1,000 samples with next-token prediction and controlling thinking duration via a simple test-time technique we refer to as budget forcing leads to a strong reasoning model that scales in performance with more test-time compute.” “The rule-based RL training stage of DeepSeek-R1-Zero is narrowly focused on reasoning tasks, resulting in limited performance in broader areas such as writing and open-domain question answering.” Across all claim–document pairs, the pass extracts $77$ countervailing items.
3a. Local aspect induction	For each claim–document pair, the system proposes dimensions discussed by that document that bear on evaluating the claim. For example, DeepSeek-R1 proposes an aspect concerning performance gains on objectively verifiable reasoning tasks , while the test-time-scaling paper proposes an aspect concerning non-RL alternatives for improving reasoning .

Appendix

Table 15: Step-by-step application of the evidence-smoothing pipeline to a claim about reinforcement learning with verifiable rewards.

Pattern	Question	WebSearch ( $\sigma$ )	Docs in context ( $\sigma$ )	Docs + instructions ( $\sigma$ )
A result from one setup stated without its conditions	Analyze the differences between the LIMO and S1 papers: objectives, methodology, key findings, contributions.	LIMO appears stronger in broad benchmark performance, particularly on GPQA Diamond and MATH500. (4)	LIMO reports higher scores than original s1 on AIME24, MATH500 and GPQA, but the studies differ in evaluation details (greedy decoding vs. temperature-0) … the comparison does not by itself establish a controlled head-to-head superiority claim. (1)	LIMO is the more explanatory and curation-focused paper … [doc 1]. (3)
A result from one setup stated without its conditions	How can we detect hallucinations in LLMs using their internal states, and what advantages does this offer over external methods?	Lower latency and computational cost: internal detectors can operate in the same forward pass and use a lightweight probe. MIND targets real-time detection, and HIDE reports substantially lower computation time than multi-pass methods. (2)	–	Its principal advantages over external or output-only detection are earlier intervention, richer process-level information, reduced reliance on external knowledge bases, lower latency for single-pass methods , and improved interpretability. (1)
Separately supported fragments combined into one demonstrated conclusion	Could transformer architectures be reimagined to process multimodal inputs (video/audio/text) as efficiently as text?	The more promising principle is: use modality-specific perception, adaptive semantic tokenization, sparse cross-modal attention, persistent latent memory, and on-demand access to raw evidence. (4)	The evidence suggests that multimodal Transformers may need to be redesigned around asymmetric processing, learned compression, latent bottlenecks, adaptive tokenization, and selective cross-modal interaction. (3)	The likely path … is to redesign the architecture so that all modalities are converted into compact, queryable representations … Whether that bottleneck can remain small enough … remains unresolved by the supplied evidence . (1)
Separately supported fragments combined into one demonstrated conclusion	How might federated learning combined with model distillation overcome technical and legal barriers in domains like healthcare?	The most credible approach is therefore a layered architecture: local training + distillation + secure aggregation + differential privacy + strong governance and clinical validation. (3)	The strongest approach is therefore layered: local data retention, carefully selected distillation, secure aggregation, differential privacy where appropriate … and clinical validation. (3)	The strongest approach is therefore layered: [same list] … Such an architecture can substantially lower both technical and legal risk, but the supplied documents do not support claiming that it eliminates either category of barrier . (1)
Conclusive or superlative wording beyond what the sources state	What is the evolution path of multimodal models from early visual representations to current multimodal LLMs, and what are the key breakthroughs?	Language as a universal interface for vision: natural language became both a source of supervision and a way to query visual representations. (3)	CLIP’s central breakthrough was language-grounded open-vocabulary recognition … The approach also exposed important limitations: performance depended on prompt wording, and CLIP struggled with counting, fine-grained recognition, and out-of-distribution images [doc 11]. (1)	CLIP and ALIGN demonstrated that web-scale paired text can train general visual representations, support retrieval, and produce zero-shot classifiers through natural language [docs 11, 13]. (1)
Conclusive or superlative wording beyond what the sources state	(same question)	BLIP-2 made the connection more parameter-efficient. This was an important engineering breakthrough because it showed that multimodal capability could be obtained without end-to-end retraining of every parameter. (3)	–	Flamingo and BLIP-2 showed that strong pretrained vision and language models can be connected through relatively small trainable modules rather than fully retrained end-to-end [docs 15–16]. (1)

Appendix

Table 16: How smoothing changes when the same model is given the cited documents (Luna, ResearcherBench). Claims are extracted independently per report; each row shows the claim in each arm that reaches the same conclusion, with

\sigma

in parentheses. “–” means no claim in that arm reaches the same conclusion.

System	ResearcherBench	CS2-Expert	DRB-II	ResearchRubrics
Luna 5.6	19.2% (95/496)	13.3% (34/256)	9.3% (27/291)	8.9% (17/190)
Sol 5.6	27.7% (169/611)	30.0% (88/293)	16.1% (68/422)	28.2% (70/248)
QUEST	73.8% (217/294)	64.9% (63/97)	30.5% (36/118)	29.4% (25/85)
Sonnet 5	2.3% (34/1,458)	3.5% (19/536)	1.1% (8/750)	1.1% (6/559)
Opus 5.5	0.9% (12/1,277)	2.3% (13/561)	1.3% (10/751)	0.2% (1/471)
DR Tulu	0.5% (5/1,082)	0% (0/445)	0% (0/303)	0% (0/290)

Appendix

Table 17: Share of cited links opened, by system and dataset (opened / cited links). A link counts as opened if the system issued an open, fetch or visit for it. Gemini Deep Research exposes no browsing steps.

Pattern	System	Question	Claim (score)	Judge verdict
Unsupported composition	DR Tulu	DRB-II ex16: I am researching the challenges faced by SRAM operating at low voltages, and I hope you can help me survey and compile a technical report on mainstream solutions. The report needs to include the following …	Alternate Cells: 7T adds explicit precharge–equalize sequence, 8T virtual-ground/zero-precharge, 9T conflict-free write with equalized leakage, 10T quasi-differential read, 11T DDPS for ultra-low leakage, 12T gating and two-stage read. (4)	The claim over-compresses heterogeneous literature into a fixed one-to-one mapping and misattributes key mechanisms (notably 7T and 11T), while several other pairings are only variant-specific. It requires substantial revision to a qualified, example-based mapping rather than categorical assignments.
Unsupported composition	DR Tulu	CS2-Expert ex6: Describe the Sunflower conjecture, progress towards a solution, and related problems.	Decision trees, DNF/branching programs, and communication: Variants of the sunflower conjecture directly constrain the size/width/depth of Boolean formulas and decision trees, with tight results (e.g., $\tilde{\Omega}(k^{2})$ ) for unambiguous DNF; these bounds, proved via explicit constructions and sunflower arguments, carry over to communication protocols and algebraic complexity analogs (e.g., matrix multiplication exponent connections surveyed by Alon). (4)	The claim over-compresses heterogeneous results: it correctly cites tight unambiguous DNF bounds and connections to communication, and separately, sunflower-variant tools and matrix-multiplication connections, but it incorrectly attributes the tight DNF bound to sunflower arguments, overstates direct constraints on decision trees/branching programs, and overstates transfer to algebraic complexity. It needs substantial narrowing and reattribution.
Premature closure	QUEST	CS2-Expert ex11: Survey recent papers on the use of LLMs for 5G network orchestration	Validates that LLM-powered prompt engineering outperforms static ML methods in highly variable O-RAN environments. (5)	The claim overstates what the evidence shows. The record supports that LLM prompting can beat simple heuristics and be useful in hybrid schemes, but it does not validate outperformance over static ML in highly variable O-RAN environments.
Premature closure	QUEST	ResearcherBench ex3: In multimodal pretraining, the current mainstream paradigms are based on image tokens and stable diffusion. Analyzing the latest advancements in these two technical approaches, with reference …	SOTA Across Multiple Benchmarks: DDT-based MLLMs outperform all previously reported models in zero-shot/non-zero-shot evaluation for image-text matching, captioning, question answering (VQA, GQA), object detection, and metric scores (GenEval 0.66, NOCAPs 124.2, EVR L1 14.5, CVS 85.5%). Even compared to diffusion specialists trained on similar data, DDT shows clear superiority in compositional edits and counterfactual interpolation, owing to its recursive, attribute-rich tokens that mimic natural language structure. (4)	The claim needs substantial revision: while the quoted metric values are accurate and DDT-based MLLMs are competitive, the evidence shows they do not outperform all previously reported models on several listed benchmarks and provides no support for object detection.

Appendix

Table 18: High-severity claims from the open-source systems, with the SmoothScore judge’s verdict.

Pattern	System	Question	Claim (score)	Judge verdict
Scope expansion	Sonnet 5	ResearcherBench ex49: From a technical perspective, how to understand the similarities and differences between Reinforcement Learning (RL) algorithms and Supervised Fine-Tuning (SFT) in Large Language Models (LLMs), as well as …	RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants, while SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. (4)	The claim overstates a mixed and conditional landscape. While there is solid evidence that outcome-based RL can improve OOD generalization in both textual and visual tasks and that SFT often overfits, there is also substantial, direct evidence that SFT can generalize and sometimes surpass RL, and that RL’s gains depend on reward design, initialization, and compute. The conclusion needs rebalancing and explicit conditions.
Premature closure	Luna	ResearcherBench ex3: In multimodal pretraining, the current mainstream paradigms are based on image tokens and stable diffusion. Analyzing the latest advancements in these two technical approaches, with reference …	For multimodal pretraining as a route toward a single general-purpose model, discrete image-token modeling appears more promising than a Stable-Diffusion-centered architecture. It fits naturally with the scaling machinery of language models: images, text, and video can be serialized into one sequence and trained with a common next-token objective. The strongest evidence comes from Emu3, Janus-Pro, Chameleon, and the rapidly improving tokenizers TiTok/FlexTok. (4)	The claim needs substantial revision. The evidence supports discrete-token, next-token training as a viable and rapidly improving path, but not that it is more promising overall than diffusion-centered approaches; multiple studies show diffusion/flow or hybrid models scaling better and being more compute-efficient, and the ‘natural fit’ is tempered by instability and modality-specific shortcomings.
Unsupported composition	Gemini	CS2-Expert ex2: What are some of the methods of detecting GPS location spoofing attack in Connected and Autonomous Vehicles?	At the mechanical layer, the tight fusion of GNSS data with highly resilient IMU kinematics and LiDAR map-matching ensures that even if a spoofed signal perfectly replicates the cryptographic and spatial signatures of a satellite, it fundamentally cannot alter the physical reality of the road ahead. (4)	The claim needs substantial revision. The evidence supports that GNSS–IMU–LiDAR/map fusion can greatly harden localization and often detect/mitigate spoofing, but it does not ensure immunity—even against sophisticated, spatially and cryptographically convincing attacks—and has documented failure modes and delays.

Appendix

Table 19: High-severity claims from the closed-source systems, with the SmoothScore judge’s verdict.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 787 papers

May 7, 2026

Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents

Large language models (LLMs) power deep research agents that synthesize information from hundreds of web sources into cited reports, yet these citations cannot be reliably verified. Current approaches either trust models to self-cite accurately, risking bias, or employ retrieval-augmented generation (RAG) that does not validate source accessibility, relevance, or factual consistency. We introduce the first source attribution evaluation framework that uses a reproducible AST parser to extract and evaluate inline citations from LLM-generated Markdown reports at scale. Unlike methods that verify claims in isolation, our framework closes the loop by retrieving the actual cited content, enabling human or model evaluators to judge each citation against its source. Citations are evaluated along three dimensions. (1) Link Works verifies URL accessibility, (2) Relevant Content measures topical alignment, and (3) Fact Check validates factual accuracy against source content. We benchmark 14 closed-source and open-source LLMs across three evaluation dimensions using rubric-based LLM-as-a-judge evaluators calibrated through human review. Our results reveal that even the strongest frontier models maintain link validity above 94% and relevance above 80%, yet achieve only 39-77% factual accuracy, while fewer than half of open-source models successfully generate cited reports in a one-shot setting. Ablation studies on research depth show that Fact Check accuracy drops by approximately 42% on average across two frontier models as tool calls scale from 2 to 150, demonstrating that more retrieval does not produce more accurate citations. These findings reveal a critical disconnect between surface-level citation quality and factual reliability, and our framework provides the evaluation infrastructure to assess the disconnect.

Hailey Onweller, Elias Lumer, Austin Huber +3

Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.

LLM Evaluation Citation Verification LLM Agent Evaluation

Ask PDF

Feb 28, 2026

Super Research: Answering Highly Complex Questions with Large Language Models through Super Deep and Super Wide Research

While Large Language Models (LLMs) have demonstrated proficiency in Deep Research or Wide Search, their capacity to solve highly complex questions-those requiring long-horizon planning, massive evidence gathering, and synthesis across heterogeneous sources-remains largely unexplored. We introduce Super Research, a task for complex autonomous research tasks that integrates (i) structured decomposition into a research plan, (ii) super wide retrieval for diverse perspectives, and (iii) super deep investigation to resolve uncertainties through iterative queries. To evaluate this capability, we curated a benchmark of 300 expert-written questions across diverse domains, each requiring up to 100+ retrieval steps and 1,000+ web pages to reconcile conflicting evidence. Super Research produces verifiable reports with fine-grained citations and intermediate artifacts (e.g., outlines and tables) to ensure traceable reasoning. Furthermore, we present a graph-anchored auditing protocol that evaluates Super Research along five dimensions: Coverage, Logical Consistency, Report Utility, Objectivity and Citation Health. While super-complex questions may be infrequent in standard applications, Super Research serves as a critical ceiling evaluation and stress test for LLM capabilities. A model's proficiency within Super Research acts as a powerful proxy for its general research competence; success here suggests the robustness necessary to navigate nearly any subordinate research task. Leaderboard is available at: https://cnsdqd-dyb.github.io/Super-Research-Benchmark/

Yubo Dong, Nianhao You, Yuxuan Hou +5

College of Computer Science and Technology · Ant Group, Hangzhou, China

Autonomous Research Agents LLM Evaluation Deep Research Agents Knowledge-Intensive QA

Ask PDF

May 26, 2026

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation

Misalignment between claims and their cited evidence is a common failure mode in reports generated by large language models, limiting their reliability in scientific and other high-stakes settings. We present DeepSciVerify, a two-stage pipeline for scientific claim-citation verification that combines abstract-level reasoning with selective escalation to passage-level evidence. The system first verifies claims using the abstract and defers uncertain cases, retrieving and analyzing full-text passages only when necessary. This design leverages complementary behaviors across LLMs, as some models are more conservative while others are more decisive under uncertainty. On the SCitance benchmark, DeepSciVerify achieves 86.7 Micro-F1, outperforming strong abstract-only baselines by +4.5 points while resolving 67% of instances without full-text retrieval. These results suggest that selective evidence escalation improves both accuracy and efficiency in claim-citation verification.

Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari +1

FirstPrinciples · School of Computing Science, Simon Fraser University

Selective Prediction Citation Verification Evidence Retrieval Scientific Claim Verification

Ask PDF

Abstract

LLM Reliability

Explore similar work

Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents

Super Research: Answering Highly Complex Questions with Large Language Models through Super Deep and Super Wide Research

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation