cs.AIFeb 18, 2026
SaveSourceBench: Can AI Answers Reference Quality Web Sources?
Organizations: University of California, San Diego · GenseeAI Inc.
Abstract
Large language models (LLMs) increasingly answer queries by citing web sources, but existing evaluations emphasize answer correctness rather than evidence quality. We introduce SourceBench, a benchmark for measuring the quality of cited web sources across 100 real-world queries spanning informational, factual, argumentative, social, and shopping intents. SourceBench uses an eight-metric framework covering content quality (content relevance, factual accuracy, objectivity) and page-level signals (e.g., freshness, authority/accountability, clarity), and includes a human-labeled dataset with a calibrated LLM-based evaluator that matches expert judgments closely. We evaluate eight LLMs, Google Search, and three AI search tools over 3996 cited sources using SourceBench and conduct further experiments to understand the evaluation results. Overall, our work reveals four key new insights that can guide future research in the direction of GenAI and web search.
Figures & tables
| Type | Source | Count | Primary Intent |
|---|---|---|---|
| Shopping | VA-COS NLQ ( Papenmeier et al., 2021 ) | 20 | Product search and purchase guidance |
| Reasoning | HotpotQA ( Yang et al., 2018 ) | 20 | Multi-hop factual research and lookups |
| Factuality | Pinocchios ( Hu et al., 2024 ) | 20 | Fact-checking and claim verification |
| Argumentative | DebateQA ( Xu et al., 2024 ) | 20 | Multi-perspective subjective reasoning |
| Social Q&A | Quora ( Quora and Kaggle, 2017 ) | 20 | Community-based forum inquiries |
| Total | 100 |
Table 1: Requests Composition in SourceBench.
| Metric | Focus | Best Score (5) | Worst Score (1) |
| CR | Content relevance & problem resolution | Comprehensive and directly resolves the user need; no further lookup required. | Irrelevant or mismatched to the query. |
| FA | Factual accuracy & verifiability | Claims are verifiable and supported by citations, prioritizing primary sources. | Contains false or harmful claims; unverifiable assertions that contradict known facts. |
| NE | Neutrality & objectivity | Neutral/clinical delivery; avoids emotional manipulation and does not push a single agenda. | Propagandistic or manipulative tone that distorts judgment. |
| FR | Freshness & maintenance | Timely and clearly maintained, dated/updated recently or follow query’s time context. | Legacy/obsolete: time-sensitive info is outdated such that it becomes incorrect or misleading. |
| AA | Author accountability & expertise | Named author with verifiable bio/credentials and clear accountability links. | Deceptive or missing identity signals (fabricated persona; no credible author info). |
| OA | Ownership accountability | Ownership and funding are transparent with clear organizational accountability. | Ownership is hidden. No contact/organizational info; signals of shell or obfuscation. |
Table 2: SourceBench Rubrics . Each metric is scored on a 1–5 scale with 5 (best) and 1 (worst) meaning described below.
| Dimension | CR | FA | NE | FR | OA | AA | DA | LC | Avg |
|---|---|---|---|---|---|---|---|---|---|
| MAE |
Table 3: Qwen-Plus Few-shot Alignment with Human Graders ( ). Values represent the Mean Absolute Error (MAE) between Model scores and the average of Human Graders.
| System | CR | FA | NE | FR | OA | AA | DA | LC | Avg | Weighted | Rank |
| GPT-5 | 3.923 | 4.772 | 4.547 | 4.490 | 4.793 | 4.433 | 4.736 | 4.016 | 4.462 | 89.081 | 1 |
| Grok-4.1-Fast | 4.328 | 4.508 | 3.961 | 4.180 | 4.315 | 3.993 | 4.334 | 3.633 | 4.153 | 83.381 | 2 |
| GPT-4o | 4.241 | 4.207 | 3.925 | 4.524 | 3.963 | 4.150 | 4.180 | 3.344 | 4.067 | 81.518 | 4 |
| Claude-Sonnet-4.5 | 4.209 | 4.217 | 3.818 | 4.447 | 4.178 | 4.202 | 4.079 | 3.336 | 4.061 | 81.282 | 5 |
| Gemini-3-Pro | 3.633 | 4.133 | 3.851 | 4.743 | 4.141 | 3.891 | 3.993 | 3.502 | 3.988 | 79.379 | 8 |
| Gemini-3-Flash | 3.633 | 4.100 | 3.925 | 4.459 | 4.097 | 4.057 | 3.962 | 3.416 | 3.960 | 78.981 | 9 |
Table 4: Overall Evaluation Results . Weighted gives the first three columns a 1.5 weight and scales the max to 100. Avg shows the average score of the eight metrics. Evaluating systems are first grouped by type and then ordered by their weighted overall score.
| System | Argument | Reasoning | Factual | Social | Shopping | Overall |
|---|---|---|---|---|---|---|
| GPT-5 | 90.192 | 88.337 | 88.995 | 87.840 | 85.553 | 89.116 |
| Grok-4.1-Fast | 90.351 | 78.012 | 86.002 | 78.820 | 84.486 | 83.381 |
| GPT-4o | 82.605 | 78.283 | 86.605 | 79.211 | 80.884 | 81.517 |
| Claude-Sonnet-4.5 | 79.398 | 79.158 | 85.560 | 74.398 | 80.104 | 81.281 |
| Gemini-3-Pro | 81.421 | 72.632 | 81.873 | 72.372 | 81.355 | 79.359 |
| Gemini-3-Flash | 81.053 | 72.285 | 80.402 | 74.257 | 78.730 | 78.905 |
Table 5: Evaluation Results by Request Category. Results shown are weighted score across metrics for each system and query type.
Figure 1: Correlation between Metrics . Heatmap of correlation between different metrics; results from all evaluating systems are aggregated.
| System | Answer Score | CR | FA | NE | FR | OA | AA | DA | LC | Weighted Source |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5 | 18.33% | 3.923 | 4.772 | 4.547 | 4.490 | 4.793 | 4.433 | 4.736 | 4.016 | 89.081 |
| Gensee | 14.04% | 4.435 | 4.432 | 3.895 | 4.343 | 3.970 | 3.970 | 4.102 | 3.343 | 81.795 |
| Grok-4.1-Fast | 10.00% | 4.328 | 4.508 | 3.961 | 4.180 | 4.315 | 3.993 | 4.334 | 3.633 | 83.381 |
| Perplexity-Sonar-Pro | 10.00% | 3.691 | 4.075 | 3.619 | 4.358 | 3.806 | 4.220 | 4.183 | 3.643 | 78.547 |
| Claude-Sonnet-4.5 | 9.26% | 4.209 | 4.217 | 3.818 | 4.447 | 4.178 | 4.202 | 4.079 | 3.336 | 81.282 |
| Gemini-3-Pro | 8.33% | 3.633 | 4.133 | 3.851 | 4.743 | 4.141 | 3.891 | 3.993 | 3.502 | 79.379 |
Table 6: HotpotQA (Reasoning) Results. The answer score shown is exact match percentage to ground truth. The last column shows the weighted SourceBench overall scores. Rows ordered by answer score.
Figure 2: Correlation in HotpotQA .
| System | PD | DA | CR | FA | NE | FR | OA | AA | DA | LC | Weighted Source |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5 | 3.621 | 0.383 | 3.923 | 4.772 | 4.547 | 4.490 | 4.793 | 4.433 | 4.736 | 4.016 | 89.081 |
| Claude-Sonnet-4.5 | 3.800 | 0.633 | 4.209 | 4.217 | 3.818 | 4.447 | 4.178 | 4.202 | 4.079 | 3.336 | 81.282 |
| Gemini-3-Flash | 3.885 | 0.717 | 3.633 | 4.100 | 3.925 | 4.459 | 4.097 | 4.057 | 3.962 | 3.416 | 78.981 |
| Gensee | 3.893 | 0.233 | 4.435 | 4.432 | 3.895 | 4.343 | 3.970 | 3.970 | 4.102 | 3.343 | 81.795 |
| Perplexity-Sonar-Pro | 3.901 | 0.483 | 3.691 | 4.075 | 3.619 | 4.358 | 3.806 | 4.220 | 4.183 | 3.643 | 78.547 |
| GPT-4o | 3.907 | 0.526 | 4.241 | 4.207 | 3.925 | 4.524 | 3.963 | 4.150 | 4.180 | 3.344 | 81.518 |
Table 7: DebateQA Results. Rows are ordered by the weighted score ranking from the provided dataset. PD (Perspective Diversity) and DA (Dispute Awareness) are prepended from the previous analysis.
| Metric | GPT-5 | GPT-4o | Grok-4.1 | Gem-3-pr | Gem-3-Fl | Gem-2.5-Fl | Plxty | Exa | Tavily | Gensee | Claude |
|---|---|---|---|---|---|---|---|---|---|---|---|
| % in Google | 15.99 | 27.53 | 29.67 | 20.95 | 23.5 | 31.96 | 40 | 44.66 | 55.45 | 28.4 | 37.1 |
| W. Source | 89.081 | 81.518 | 83.381 | 79.379 | 78.981 | 78.339 | 78.547 | 80.108 | 78.268 | 81.795 | 81.282 |
Table 8: GenAI and Search Result Overlapping
| System | CR | FA | NE | FR | OA | AA | DA | LC | Weighted Avg |
|---|---|---|---|---|---|---|---|---|---|
| DS-Chat-Tavily | 3.783 | 4.087 | 3.783 | 4.217 | 4.232 | 3.855 | 4.246 | 3.507 | 70.085 |
| DS-Chat-Gensee | 4.244 | 4.564 | 4.154 | 4.423 | 4.321 | 3.987 | 4.462 | 3.922 | 75.989 |
| DS-Reasoning-Tavily | 4.190 | 4.603 | 4.034 | 4.397 | 4.552 | 4.103 | 4.655 | 3.724 | 75.823 |
| DS-Reasoning-Gensee | 4.040 | 4.467 | 3.987 | 4.351 | 4.230 | 4.081 | 4.392 | 3.797 | 74.103 |
Table 9: DeepSeek Results.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Criterion | Rationale, Citations & Guidance | Scoring Guidelines (1–5 Scale) |
|---|---|---|
| Group 1: Content Assessment | ||
| 1. Semantic Relevance | Core Utility. Dewan et al. (2025) validated that “Intent Alignment” is distinct from factual accuracy. Wang et al. (2025) showed fine-grained scales outperform binary metrics. Implementation Guidance: Use Text_content_length as a signal. If length is 300 chars, penalize relevance unless it is a perfectly concise factual answer. | 5 (Fully Meets): Comprehensive, direct answer. No further searching needed. 4 (Highly Meets): Highly relevant but lacks minor details or slightly outdated. 3 (Moderately Meets): Broadly on topic but misses specific intent; requires more clicks. 2 (Slightly Meets): Tangential keyword match only. Misses core question. 1 (Fails to Meet): Irrelevant, wrong language, or technical failure. |
| 2. Factual Accuracy | Evidence Verification. Li et al. (2024) identify lack of citations as a primary cause of hallucination. Loru et al. (2025) found accuracy is the strongest predictor of credibility. Note: Primary sources (Gov, Edu, Science) are weighted higher than secondary sources (News). | 5 (Verified): Claims cited with primary sources (Gov, Edu, Science). 4 (Accurate): Aligns with consensus. Secondary citations (News) exist. 3 (Unverified): Plausible “common knowledge” but lacks evidence/links. 2 (Suspect): Fringe theories, logic gaps, or context omissions. 1 (False): Demonstrably false, dangerous, or debunked. |
| 3. Objectivity & Tone | Bias Detection. Loru et al. (2025) identify “Language Manipulation” and emotional charging as key signals for low-credibility domains. | 5 (Neutral): Clinical/Academic. Multiple viewpoints. Zero emotional loading. 4 (Balanced): Professional journalistic tone. Acknowledges counter-arguments. 3 (Opinionated): Clear subjective bias/op-ed but grounded in facts. 2 (Inflammatory): Emotional triggers (“Shocking!”), excessive punctuation. 1 (Propaganda): Hate speech, fear-mongering, incoherent. |
| Group 2: Source & Metadata Assessment | ||
| 4. Information Freshness | Utility Blocker. Ouyang et al. (2025) prove “Timeliness Awareness” is critical for RAG utility. Waterfall Logic Implementation: 1. Priority 1: Check last_updated . If 1 year from Jan 2026 Score 5. 2. Priority 2: If P1 is null, use date . 3. Priority 3: If both null, use text markers (e.g., “RTX 50-series”). Exception: If last_updated (2025) date (2012), treat as Score 5. | 5 (Current): Dated within last 6–12 months. Data is current. 4 (Recent): 1–2 years old but valid. 3 (Static): Undated “evergreen”. References older context (e.g., “Windows 10”). 2 (Outdated): Clearly obsolete (old laws, discontinued products). 1 (Legacy): Ancient ( 10 years) and factually wrong due to time. |
Table 10: Revised Source Reliability Rubric with Citations & Guidance. Dimensions are scored on a 1–5 scale.
| Prompt Type | Content |
| System Prompt | You are a research evaluator specializing in Information Retrieval (IR). Your task is to perform an internal quality assessment of a cited website’s text content relative to a specific user query. |
| User Prompt Template | Rubrics: 1. Semantic Relevance (1-5) 5 (Fully Meets): Comprehensive, direct answer. No further searching needed. 4 (Highly Meets): Highly relevant but lacks minor details or slightly outdated. 3 (Moderately Meets): Broadly on topic but misses specific intent; requires more clicks. 2 (Slightly Meets): Tangential keyword match only. Misses core question. 1 (Fails to Meet): Irrelevant, wrong language, or technical failure. Implementation Guidance: Use text_content_length as a signal; if length is ¡300 characters, penalize relevance unless it is a perfectly concise factual answer. 2. Factual Accuracy & Citations (1-5) 5 (Verified): Claims cited with primary sources (Gov, Edu, Science). 4 (Accurate): Aligns with consensus. Secondary citations (News) exist. 3 (Unverified): Plausible “common knowledge” but lacks evidence/links. 2 (Suspect): Fringe theories, logic gaps, or context omissions. 1 (False): Demonstrably false, dangerous, or debunked. 3. Objectivity & Tone (1-5) 5 (Neutral): Clinical/Academic. Multiple viewpoints. Zero emotional loading. 4 (Balanced): Professional journalistic tone. Acknowledges counter-arguments. 3 (Opinionated): Clear subjective bias/op-ed but grounded in facts. 2 (Inflammatory): Emotional triggers (“Shocking!”), excessive punctuation. 1 (Propaganda): Hate speech, fear-mongering, incoherent. Few-Shot Calibration Examples Example 1: The Expert Document (Query 81 - Apache Mahout) Query: How do I evaluate AUC in Apache Mahout? Analysis: Relevance: 5 (Perfect intent/class alignment); Accuracy: 5 (Official docs); Tone: 3 (Clinical but saturated with boilerplate). Example 2: The GEO/Thin Content (Query 11 - Trenchcoats) Query: Reasonably-priced (under $50) short trenchcoat recommendations. Analysis: Relevance: 3 (Matches keyword but fails specific “short” constraint); Accuracy: 3 (Common knowledge, no links); Tone: 4 (Professional but slightly promotional). Example 3: The High-Quality Review (Query 1 - Laptops) Query: Smaller laptop for older games/word processing, SSD preferred. Analysis: Relevance: 4 (Highly relevant, explicit SSD mention missing in summary); Accuracy: 5 (Lab testing); Tone: 4 (Journalistic but contains affiliate links). Task & Output Format Provide a score (1-5) and a short justification for each of the three metrics above. Input Data Packet: { "query_id": "...", "query_text": "...", "source_url": "...", "source_text": "...", "text_length": "..." } Output Format (Return ONLY JSON): { "semantic_relevance": { "score": int, "justification": "string" }, "factual_accuracy": { "score": int, "justification": "string" }, "objectivity_tone": { "score": int, "justification": "string" } } |
| Prompt Type | Content |
|---|---|
| System Prompt | You are a digital forensic auditor. Your task is to evaluate the external credibility and timeliness of a source based on its metadata and structural markers. |
| User Prompt Template | Rubrics: 1. Information Freshness (1-5) 5 (Current): Dated within last 6-12 months. Data is current. 4 (Recent): 1-2 years old but valid. 3 (Static): Undated “evergreen”. References older context. 2 (Outdated): Clearly obsolete (old laws, discontinued products). 1 (Legacy): Ancient ( 10 years) and factually wrong. Implementation Guidance: 1. Priority 1: search_result_last_updated ( 1 yr from Jan 2026 = 5). 2. Priority 2: search_result_date . 3. Priority 3: Text Cues. |
| 2. Author Accountability (1-5) 5 (Expert): Named author with verifiable bio/credentials linked. 4 (Professional): Named author (Staff Writer) or reputable group. 3 (Opaque): “Admin” or “Team” authorship. 2 (Hidden): No author listed where expected. 1 (Deceptive): Fake persona or AI-generated face. | |
| 3. Ownership Transparency 5 (Fully Transparent): Physical address, phone, clear leadership. 4 (Partially Transparent): Clear “About”, leadership names, no physical info. 3 (Opaque): Contact form only; vague “About”. 2 (Anonymous Org): No “About” info or ownership details. 1 (Hidden): Intentional shell company signs. | |
| 4. Domain Authority (1-5) 5 (Authority): Gov/Edu, Major News, established brand. 4 (Trusted): Established commercial brand or niche expert. 3 (Unknown): Functional blog/small biz. 2 (Low Quality): Content farm, parasite SEO. 1 (Blacklisted): Known scam/disinformation. | |
| Few-Shot Examples for Calibration Ex 1 (Apache Mahout): Freshness: 5 (Dec 2025); Accountability: 4 (Apache Fdn); Transparency: 5; Domain: 5. Ex 2 (Democracy In Action): Freshness: 2 (2021); Accountability: 3; Transparency: 4; Domain: 4. Ex 3 (PolitiFact): Freshness: 3 (Active ecosystem); Accountability: 5; Transparency: 5; Domain: 4. Ex 4 (Yoga Detox): Freshness: 3 (Evergreen); Accountability: 5; Transparency: 5; Domain: 4. |
| Prompt Type | Content |
|---|---|
| System Prompt | You are a User Experience (UX) analyst. Your task is to evaluate the usability and monetization intensity of a website based on its extracted text structure. |
| User Prompt Template | Rubrics: Layout & Ad Density (1-5) 5 (Clean): Zero intrusive ads. High skimmability. 4 (Standard): Banners/sidebars that do not block content. 3 (Cluttered): Auto-playing video, sticky footers, heavy monetization. 2 (Obstructive): Pop-ups require dismissal. “Chumbox” fake articles at bottom. 1 (Spam/Unusable): Overlays, broken layout, malicious redirects. Implementation Guidance: Since you are viewing extracted text, look for “ad-interstitial text” (e.g., “ADVERTISEMENT” labels) or fragmented text structures. High text volume containing frequent repetitive affiliate links (e.g., “Check Price on Amazon”) suggests a Score 2 or 3. |
| Few-Shot Calibration Examples Example 1: The Obstructive Media (Query 1 - Laptops) Query: Best ultraportable laptops for 2026? Source: pcmag.com Analysis: Layout & Ad Density: 2 - Text is fragmented with frequent “ADVERTISEMENT” markers and ghost text from auto-play videos. Over 20 distinct commercial CTAs (“GET IT NOW”) exceed the threshold for high-intensity monetization. Example 2: The Cluttered Portal (Query 21 - Yoga Detox) Query: Can yoga really detoxify your body? Source: healthgrades.com Analysis: Layout & Ad Density: 2 - Heavily saturated with repetitive navigational artifacts (“Find a Doctor”, “Sign In”) and commercial CTAs. UX is secondary to user acquisition, triggering the “Obstructive” score. Example 3: The Clean Doc (Query 81 - Apache) Query: How do I evaluate AUC in Apache Mahout? Source: mahout.apache.org Analysis: Layout & Ad Density: 5 - Continuous, logically structured technical document. Functional navigation (“SKIP NAVIGATION LINKS”) is present but standard; zero commercial markers or pop-ups. | |
| Task & Output Requirement Provide a score (1-5) and a short justification for this metric. Input Data Packet: { "source_url": "...", "source_text": "...", "text_length": "..." } Output Requirement (Return ONLY JSON): { "layout_ad_density": { "score": int, "justification": "string" } } |
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers