Large language models and deep research agents supply citation URLs to support their claims, yet the reliability of these citations has not been systematically measured. We address six research questions about citation URL validity using 10 models and agents on DRBench (53,090 URLs) and 3 models on ExpertQA (168,021 URLs across 32 academic fields). We find that 3--13% of citation URLs are hallucinated -- they have no record in the Wayback Machine and likely never existed -- while 5--18% are non-resolving overall. Deep research agents generate substantially more citations per query than search-augmented LLMs but hallucinate URLs at higher rates. Domain effects are pronounced: non-resolving rates range from 5.4% (Business) to 11.4% (Theology), with per-model effects even larger. Decomposing failures reveals that some models fabricate every non-resolving URL, while others show substantial link-rot fractions indicating genuine retrieval. As a solution, we release urlhealth, an open-source tool for URL liveness checking and stale-vs-hallucinated classification using the Wayback Machine. In agentic self-correction experiments, models equipped with urlhealth reduce non-resolving citation URLs by 6–79× to under 1%, though effectiveness depends on the model's tool-use competence. The tool and all data are publicly available. Our characterization findings, failure taxonomy, and open-source tooling establish that citation URL validity is both measurable at scale and correctable in practice.
Figures & tables
Model
Provider
Type
Study
URLs
gemini-2.5-pro-deepresearch
Google
Deep research
DRBench
11,309
openai-deepresearch
OpenAI
Deep research
DRBench
4,121
gemini-2.5-pro-with-search
Google
Search-aug.
DRBench
1,609
gemini-2.5-flash-with-search
Google
Search-aug.
DRBench
2,433
gpt-4o-search-preview
OpenAI
Search-aug.
DRBench
387
gpt-4o-mini-search-preview
OpenAI
Search-aug.
DRBench
402
Table 1: Model inventory. “Search-aug.” denotes search-augmented LLMs. URL counts reflect total unique URLs extracted from model outputs.
Figure 1: Non-resolving URL rates for DRBench models, grouped by provider. Each bar decomposes into hallucinated URLs (red; no Wayback Machine archive, indicating the URL likely never existed) and stale URLs (orange; archived but currently dead). Numbers above bars indicate total URLs per model.
Model
Provider
URLs
% Non-resolving
% Hallucinated
% Stale
claude-3-5-sonnet-search
Anthropic
641
7.8 [5.8, 10.0]
3.0 [1.7, 4.4]
4.8
claude-3-7-sonnet-search
Anthropic
1,735
8.5 [7.1, 9.7]
3.2 [2.4, 4.1]
5.2
openai-deepresearch
OpenAI
4,121
10.1 [9.1, 11.0]
3.5 [3.0, 4.1]
6.6
gemini-2.5-flash-search
Google
2,433
5.4 [4.5, 6.3]
4.6 [3.7, 5.4]
0.8
gemini-2.5-pro-search
Google
1,609
5.9 [4.8, 7.1]
4.8 [3.8, 5.9]
1.1
gpt-4.1
OpenAI
336
5.4 [3.0, 7.7]
5.4 [3.0, 7.7]
0.0
Table 2: DRBench per-model URL validity results sorted by hallucinated URL rate. Brackets show bootstrap 95% CIs. “% Stale” = % Non-resolving − % Hallucinated. Model names abbreviated for space.
Figure 2: Non-resolving URL rates by academic field and model for ExpertQA. Fields are sorted by overall non-resolving URL rate.
Claude
Gemini
GPT-5.1
URL health
Total URLs
7,985
4,203
4,829
URLs / question
18.4
9.7
11.1
LIVE (%)
79.3 [78.4, 80.2]
88.9 [88.0, 89.9]
78.0 [76.8, 79.1]
DEAD (%)
0.1 [0.1, 0.2]
0.2 [0.0, 0.3]
0.6 [0.4, 0.8]
LIKELY_HALL. (%)
0.4 [0.3, 0.6]
0.5 [0.3, 0.8]
1.8 [1.4, 2.1]
Table 3: URL health outcomes and self-correction behavior for three models on 435 ExpertQA questions with urlhealth as an agentic tool. Percentages are relative to total URLs proposed by each model. Brackets show bootstrap 95% CIs.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Queries
Language
Use in this study
DRBench
100
ZH, EN
Cross-model URL validity
ExpertQA
2,177
EN
Domain-stratified analysis
Appendix
Table 4: Datasets used in this study. DRBench provides pre-collected outputs from 23 deep research and search-augmented models, enabling cross-model comparison of citation URL validity. ExpertQA spans 32 academic and professional fields, enabling domain-stratified analysis of fabrication rates.
Scenario
Claude
Gemini
GPT-5.1
Baseline
9.38%
4.20%
8.47%
Reddit-excluded
9.38%
4.24%
10.36%
Reddit-as-non-resolving
9.38%
4.93%
26.71%
Reddit URLs
0
166
15,273
Reddit % of total
0.0%
0.7%
18.2%
Appendix
Table 5: Sensitivity of ExpertQA non-resolving rates to Reddit URL classification. Baseline: Reddit URLs classified as alive (Section 4.1 ). Reddit-excluded: Reddit URLs removed from both numerator and denominator. Worst case: all Reddit URLs treated as non-resolving.
True live
Blocked
Dead
Not dead
By model
claude-sonnet-4-5 ( n =200)
19.5%
72.0%
8.5% [5.0, 12.5]
91.5% [87.5, 95.0]
gemini-2.5-pro ( n =200)
22.0%
64.5%
13.5% [9.0, 18.5]
86.5% [81.5, 91.0]
gpt-5.1 ( n =200)
54.5%
34.5%
11.0% [7.0, 15.5]
89.0% [84.5, 93.0]
By original HTTP status code
403 ( n =383)
15.9%
83.8%
0.3%
99.7%
Appendix
Table 6: Headless-browser audit of 600 stratified-sampled UNKNOWN URLs from the ExpertQA baseline. Verdicts: true_live = page renders real content; blocked = paywall, bot wall, or rate-limit page from an operational site; dead = error page, empty content, or unreachable. Brackets show bootstrap 95% CIs.
Field
Overall
Claude
Gemini
GPT
Top 10 by non-resolving URL rate (%)
Theology
11.4
11.3
10.2
11.7
Classical Studies
11.1
7.1
7.6
13.8
Healthcare / Medicine
10.8
17.4
5.3
7.1
Literature
10.7
9.6
7.3
11.6
Psychology
10.2
14.7
4.6
8.6
Appendix
Table 7: Non-resolving URL rates (%) by academic field in ExpertQA. “Claude” = claude-sonnet-4-5 , “Gemini” = gemini-2.5-pro , “GPT” = gpt-5.1 . Top 10 and bottom 5 fields are shown; full results in the supplementary materials.
Figure 3: Non-resolving URL rates by subfield within Healthcare/Medicine. The top 15 subfields (sorted by rate) range from 14.8% (Virologist) to 21.4% (General practitioner). Subfield labels A–O are keyed in the accompanying table.
Figure 4: Distribution of urlhealth correction rounds per question (435 questions each, 3 models). The three models exhibit distinct self-correction profiles. Gemini 2.5 Pro (green) completes in 1–2 rounds every time: its two-phase architecture (Google Search grounding followed by a single verification turn) caps it at two rounds, and 44% of questions need only one. GPT-5.1 (orange) clusters at 2 rounds (61%), with 87% finishing in 2–3 rounds; the long tail to 6 rounds is rare ( < 2% of questions). Claude Sonnet 4.5 (blue) concentrates at 3–5 rounds (83% of questions, median 4), reflecting the cost of verifying its large citation sets (18.4 URLs/question vs. 9.7–11.1 for the other models). All three models achieve near-zero hallucinated citations in final output despite these different convergence patterns.