Redesigning and Auditing Deep Research Writing for Faithful Reports

Hiroaki Hayashi; Pranav Narayanan Venkit; Prafulla Kumar Choubey; Chien-Sheng Wu

cs.CLAug 12, 2026

Redesigning and Auditing Deep Research Writing for Faithful Reports

Authors: Hiroaki Hayashi, Pranav Narayanan Venkit, Prafulla Kumar Choubey, Chien-Sheng Wu

Organizations: Salesforce AI Research

Abstract

Rubric-based evaluations of deep-research (DR) systems often obscure fine-grained factual failures in generated reports. We introduce CLAIMPROBE, a claim-level audit that decomposes DR reports into claims and measures hallucination, misattribution, citation hygiene, and necessary-fact recall against retrieved evidence. Using CLAIMPROBE, we find that strong DR pipelines can omit key evidence and misattribute claims even when their rubric scores remain stable. We then propose CLAIMWRITER, a hierarchical claim-based writer that extracts source facts, maps them to a query-derived outline, and drafts each section from a source-linked claim representation. Across three prior DR frameworks, replacing only the report writer with CLAIMWRITER reduces hallucination by 2.6 to 4.5 times and improves necessary-fact recall by 1.2 to 1.7 times, while largely preserving overall report quality. CLAIMWRITER also enables localized revision: when sources change, it propagates changed source facts into revised reports at the highest rate among update methods, while also being more cost-effective.

Figures & tables

Panel of Figure 1 — Figure 1: System overview. A selected host deep-research pipeline supplies sources to ClaimWriter , which extracts source facts, materializes a report fact graph with {{CITE:fN}} markers, and assembles the user-facing report.md . ClaimProbe audits this fact-linked representation (Section 2 ); when sources change, ClaimWriter applies fact-level updates and emits an updated report (Section 6.2 ).

Host $+$ writer	comp.	insi.	inst.	read.	ov.
EDR	.556	.548	.543	.510	.544
EDR $+$ ClaimWriter	.553	.545	.537	.470	.536
EDR $+$ ClaimWriter-BU	.554	.550	.539	.466	.538
AI-Q	.558	.557	.554	.528	.553
AI-Q $+$ ClaimWriter	.550	.544	.536	.473	.535
AI-Q $+$ ClaimWriter-BU	.556	.552	.541	.469	.540

Table 1: RACE scores (

n{=}100

, judge GPT-5.5). Columns are comprehensiveness, insight, instruction-following, readability, and overall (higher is better).

		GPT-5.5 ( $n{=}10$ )						GPT-5.4-mini ( $n{=}100$ )
Host	Writer	Hall $\downarrow$	Hall ${}_{\text{str}}$ $\downarrow$	Mis $\downarrow$	Rec $\uparrow$	Rec ${}_{\text{str}}$ $\uparrow$	Unc	Hall $\downarrow$	Hall ${}_{\text{str}}$ $\downarrow$	Mis $\downarrow$	Rec $\uparrow$	Rec ${}_{\text{str}}$ $\uparrow$	Unc
AI-Q	Baseline	14.84	78.35	24.68	36.08	0 7.85	83.23	23.27	73.51	22.31	51.35	20.59	73.83
	$+$ ClaimWriter-BU	0 3.41	36.80	0 5.29	56.31	28.46	88.92	0 5.08	28.09	0 4.13	68.71	46.59	79.61
	$+$ ClaimWriter	0 3.37	39.02	0 5.10	56.99	29.97	88.46	0 5.20	28.49	0 3.62	68.29	47.75	80.61
ODR	Baseline	23.80	75.67	0 7.67	33.61	0 9.21	66.68	27.12	67.08	0 5.92	59.34	32.82	63.60
	$+$ ClaimWriter-BU	0 5.16	42.34	0 2.23	63.79	43.89	78.06	0 6.78	31.16	0 2.28	80.48	67.79	73.38

Table 2: ClaimProbe audit on the host

\times

writer

\times

judge-model factorial. All values are percentages, micro-averaged over pooled (numerator, denominator) pairs. Green cells are the best writer in each (host, judge-model) block; Unc is contextual and left uncolored.

		5 sources perturbed		25 sources perturbed
Host	Edit strategy	Out Tok	Yield [%]	Out Tok	Yield [%]
AI-Q	native (replay)	159K	0 0.0	354K	0 0.5
AI-Q	$+$ Base (diff)	0 20K	16.7	0 21K	0 6.9
EDR	$+$ Base (diff)	0 68K	40.5	0 70K	30.9
EDR	$\mathbf{+}$ ClaimWriter	0 34K	47.6	0 72K	39.7
EDR	$\mathbf{+}$ ClaimWriter-BU	0 78K	45.2	100K	41.2

Table 3: Dynamic-update cost and update yield. Output tokens are per-report averages over the tasks. Update Yield is the clean rate: the percentage of relevant perturbed facts whose new value reaches the regenerated report while the old value is removed (higher is better).

\mathbf{+}

ClaimWriter is the fact-diff (top-down) updater and

\mathbf{+}

ClaimWriter-BU the bottom-up fact-diff updater. Cell colour: green favourable, yellow mid, red unfavourable, dark-red catastrophic (the cost ceiling).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Task	Hum. $\kappa$	AI/maj	AI/unan.
A	Hallucination	$+0.173$	$+0.299$	$+0.484$
B	Misattribution	$+0.305$	$+0.570$	$+0.797$
C	Relevance	$+0.500$	$+0.600$	$+0.743$

Appendix

Table 4: Human and LLM judge agreement on the three judging tasks (judge GPT-5.4-mini).

		Gemini-3.5-flash ( $n{=}10$ )
Host	Writer	Hall $\downarrow$	Hall ${}_{\text{str}}$ $\downarrow$	Mis $\downarrow$	Rec $\uparrow$	Rec ${}_{\text{str}}$ $\uparrow$	Unc
AI-Q	Baseline	57.14	83.69	15.89	15.97	0 5.43	28.72
	$+$ ClaimWriter-BU	19.81	49.43	0 0.86	41.22	27.14	19.05
	$+$ ClaimWriter	23.03	54.92	0 0.99	39.71	25.89	22.86
ODR	Baseline	58.72	81.94	0 2.51	22.49	0 9.25	23.25
	$+$ ClaimWriter-BU	23.50	54.96	0 0.27	59.64	46.72	35.71

Appendix

Table 5: ClaimProbe audit with the Gemini-3.5-flash judge (

n{=}10

; ids 1–5, 51–55). All values are percentages, micro-averaged over pooled (numerator, denominator) pairs. Green cells are the best writer in each host block; Unc is contextual and left uncolored.

#Ptb.	Method	Hall	Hall ${}_{\text{str}}$	Mis	Rec	Rec ${}_{\text{str}}$	Unc
Baseline	EDR	$6.1$	$43.7$	$12.8$	$54.6$	$20.4$	$99.0$
Baseline	AI-Q	$10.4$	$68.0$	$44.5$	$30.8$	$8.4$	$90.1$
Subset 5	EDR-base	$6.4$	$44.0$	$13.3$	$54.6$	$20.3$	$98.2$
	EDR- ClaimWriter	$6.2$	$44.1$	$12.6$	$54.1$	$20.0$	$99.0$
	AI-Q (native)	$9.4$	$67.1$	$43.5$	$37.9$	$8.0$	$89.5$
Subset 25	EDR-base	$6.8$	$43.9$	$12.1$	$54.6$	$20.1$	$98.9$

Appendix

Table 6: ClaimProbe on perturbed regenerations (

\%

, judge GPT-5.5). EDR update strategies cluster within

\pm 1

of the unperturbed EDR baseline; updates do not drift.

System	in tok / report	out tok / report
EDR	$1{,}019{,}276$	$\phantom{00}92{,}252$
EDR $+$ ClaimWriter	$9{,}697{,}825$	$\phantom{0}700{,}626$
AI-Q	$\phantom{0}882{,}649$	$\phantom{0}344{,}646$
AI-Q $+$ ClaimWriter	$5{,}754{,}095$	$\phantom{0}528{,}723$
ODR	$2{,}331{,}138$	$\phantom{0}293{,}372$
ODR $+$ ClaimWriter	$2{,}779{,}813$	$\phantom{0}248{,}320$

Appendix

Table 7: Steady-state writing-stage token cost (averages per report).

+

ClaimWriter and

+

ClaimWriter-BU re-extract facts from the entire retrieved corpus, so their input cost is higher than the host baselines (which only process what they synthesize). The fact-graph cost is paid once per first-run and amortizes against the incremental-update budget in Table 3 .

Evidence-Grounded Generation

Evidence-grounded generation produces answers and documents whose claims are supported by traceable sources.

See all 149 papers

Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research
Aug 25, 2026Eran Hirsch, David Wan, Han Wang +3Deep Research Agents Multi-Agent LLM Systems
Bar-Ilan University · UNC Chapel Hill · University of Texas at Austin
DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
Dec 19, 2025Janghoon Han, Heegyu Kim, Changho Lee +6Deep Research Agents LLM Agent Evaluation
LG AI Research · University of Illinois Chicago · University of Michigan, Ann Arbor
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Jan 13, 2026Ruizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research Agents LLM Agent Evaluation
University of Science and Technology of China · Metastone Technology, Beijing, China

Aug 25, 2026J/K move · Enter open · S save

Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research

Eran Hirsch, David Wan, Han Wang, Elias Stengel-Eskin+2

Bar-Ilan University · UNC Chapel Hill · University of Texas at Austin

Deep Research Agents Multi-Agent LLM Systems LLM Agent Evaluation Citation Verification

Open paper AskPDF arXiv

Deep research (DR) systems produce long-form cited reports by orchestrating multiple agents that search and synthesize information from the web. Citations are the primary mechanism for evaluating the faithfulness of these reports, yet current DR systems exhibit poor citation recall. Moreover, improving citation recall is challenging because DR systems are complex multi-agent architectures where information passes through agents like a telephone game, and both content and citations can get corrupted along the way. We propose an evaluation method that pinpoints which agent introduced each error by locally testing agent invocations for faithfulness and verifiability relative to their own inputs. Furthermore, we propose a four-type taxonomy to categorize the discovered errors: hallucination, uncited input reliance, uncited output, or insufficient citations. Applying our method to three top-ranked open-source DR systems, we obtain actionable diagnostics. Almost every agent makes a lot of mistakes with the exception being those that summarize a single document. We find that the dominant error type varies systematically across agents, where the orchestrator mistakes are mostly citation-related. We find that 84.7% of final-report errors in AI-Q originate at the orchestrator, roughly 31% of them hallucinations and the rest citation mistakes. Guided by these insights, we demonstrate that two simple interventions raise citation recall by 5% without degrading output quality.

Abstract

Evidence-Grounded Generation

Explore similar work

Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research

DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports