Host-Aware Web Crawling for LLM Pretraining Data

Malte Ostendorff

cs.CLOct 9, 2026

Host-Aware Web Crawling for LLM Pretraining Data

Authors: Malte Ostendorff

Organizations: Common Crawl Foundation

Abstract

Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data. Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl's per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation. Finally, Common Crawl's observed per-host coverage has little correlation with the evaluated quality signals. We estimate that tens of millions of high-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts.

Figures & tables

Figure 1: The intraclass correlation ICC1 is the share of a classifier’s per-document score variance that lies between hosts rather than within them: Across all ten classifiers ICC1 is 0.43–0.62 (

N=532{,}628{}

hosts), so roughly half of the variation in page quality is between hosts. This suggests that a few sampled pages already characterize a host, enabling a host-level crawl policy.

Figure 2: Graph centrality is a weak signal for which hosts to crawl. When only 10% of pages are crawled, the share of fetched pages that are high quality: centrality ordering (current/ HC / PR ) captures only a fraction of the oracle (crawler knows host’s quality), and even trails random for DCLM.

Figure 3: Median host quality scores by language. English-trained classifiers differ in how they score English and non-English hosts: DCLM assigns higher scores to non-English hosts, while Nemotron-Q assigns higher scores to English hosts. Because centrality also varies by language, its correlation with classifier scores partly reflects these language differences.

Figure 4: The correlation between the quality of

k

sampled pages and the quality of the host’s remaining pages, where

1.0

is perfect prediction and

0

is none. The curve climbs with

k

and plateaus by

k\approx 15

–

20

, but a

\sim

3–5 page sample already captures most of the signal, so a few pages suffice to decide crawl depth cheaply. Shown for our FineWeb-Edu sample (hosts with

\geq 10

pages, so

k

reaches

9

) and full-crawl GneissWeb (

\geq 30

pages).

Figure 5: Crawl efficiency based on FineWeb-Edu, measured as high-quality pages obtained per page actually fetched, and for four ways of spending a fixed budget across hosts. Sample-then-deepen (our host-level crawl policy) nearly matches the oracle and collects several times more high-quality pages per fetch than crawling every host equally ( uniform breadth ) or biasing depth toward high-centrality hosts ( HC depth ).

Figure 6: Host-level quality density does not rise with crawl coverage (Spearman -0.08). The median quality density stays flat and low across coverage (from 20% at the least-crawled hosts to 4% at the near-fully-crawled majority). Grey bars show the share of hosts in each coverage bin: most are already almost fully crawled, which is why the overall rank correlation is near zero.

Figure 7: The missed high-quality budget concentrates on barely-crawled hosts. Each cell gives the share of the estimated $\approx$ 58M uncrawled high-quality pages that falls in one coverage $\times$ quality bin (columns: crawl coverage; rows: quality density; axes as in Figure 6 ). Half (50%) of the missed pages lie on hosts crawled below 20%, and the single densest “hidden-gem” bin holds 26% of missed high-quality pages alone.

Crawl	$\rho$ (cov, qual)	Gems	Uncrawled HQ (M)
2021-49	$-0.151$	339,430	20.3
2022-49	$-0.057$	337,224	28.9
2023-23	$-0.060$	325,545	20.1
2023-40	$-0.079$	390,185	38.0
2024-10	$-0.081$	369,534	35.7
2024-18	$-0.061$	353,728	24.1

Table 1: The coverage–quality disconnect is not a one-off. For each of 6 crawls (2021–2024) we report the Spearman correlation between crawl coverage and quality density (all near

0

and slightly negative, i.e. no positive alignment), the number of under-crawled high-quality “gem” hosts, and the estimated high-quality URLs those hosts leave uncrawled. Every crawl shows the same pattern and tens of millions of missed URLs.

Figure 8: Coverage-blindness does not depend on the GneissWeb filter. Per classifier, on one Spearman axis: its agreement with the GneissWeb quality density (green) versus the correlation of Common Crawl coverage with it (orange). Quality classifiers (Nemotron, FineWeb-Edu) agree substantially with GneissWeb (DCLM and the topic scorers less so), yet every coverage bar stays inside the negligible band (

|\rho|<0.1

): whichever quality signal we use, how fully a host is crawled does not track how good it is.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Later release	Gap	$N$ hosts	HC $\rho$	PR $\rho$
2024-may-jun-jul	$\sim$ 3-5mo after	531,054	$0.875$	$0.923$
2024-25-nov-dec-jan	$\sim$ 9-11mo after	527,432	$0.870$	$0.900$
2025-may-jun-jul	$\sim$ 15mo after	524,077	$0.851$	$0.885$

Appendix

Table 2: Spearman rank correlation between a host’s centrality rank in our reference graph and its rank in later Common Crawl graph releases, where

1.0

would mean an identical ordering of all hosts. Both harmonic centrality ( HC ) and PageRank ( PR ) stay above

0.85

even

\sim

15 months apart, so host rankings barely move over a year. This stability is what lets us pair one aligned graph with document snapshots from different crawls instead of rebuilding the graph each time.

Classifier	Current	HC	PR	Oracle	TLDs (HC/rnd)
DCLM	$0.75\times$	$0.67\times$	$0.73\times$	$\mathbf{3.98\times}$	223 / 422
FineWeb-Edu	$1.60\times$	$1.58\times$	$1.47\times$	$\mathbf{7.97\times}$	223 / 422
Nemotron Quality	$1.27\times$	$1.79\times$	$1.56\times$	$\mathbf{4.05\times}$	223 / 422

Appendix

Table 3: High-quality documents collected at a 10% crawl budget, as a multiple of a random crawl (

1.0=

random,

2\times=

twice as many). Centrality ordering ( HC , PR ) gains at most 1.8

\times

—and drops below

1

for DCLM—while the oracle reaches 4.0–8.0

\times

, so most of the achievable gain is left untaken. The last column shows centrality ordering also roughly halves the number of top-level domains represented (HC vs. random), narrowing corpus diversity. This is the per-classifier, fixed-budget companion to Figure 2 ; the full budget sweep is Figure 9 .

Metric	Classifier	Spearman $\rho$	Kendall $\tau$	Weighted $\rho$	$N$ hosts
$\log\textsc{HC}$	DCLM	$-0.083$	$-0.055$	$-0.087$	27,910
$\log\textsc{HC}$	GneissWeb Quality	$+0.021$	$+0.015$	$+0.018$	27,910
$\log\textsc{HC}$	GneissWeb Sci	$+0.031$	$+0.021$	$+0.030$	27,910
$\log\textsc{HC}$	GneissWeb Edu	$+0.039$	$+0.026$	$+0.037$	27,910
$\log\textsc{HC}$	GneissWeb Tech	$+0.036$	$+0.024$	$+0.038$	27,910
$\log\textsc{HC}$	GneissWeb Med	$-0.032$	$-0.021$	$-0.035$	27,910

Appendix

Table 4: Host-level correlations of harmonic centrality and PageRank with twelve classifier scores. Spearman

\rho

, Kendall

\tau

, and weighted Spearman

\rho

agree in sign for every metric–classifier pair.

Stratum	Classifier	Spearman $\rho$	$N$ hosts
English	FineWeb-Edu	$+0.237$	13,670
English	FineWeb2-HQ	$+0.233$	1,109
English	GneissWeb Sci	$+0.199$	13,670
English	Nemotron Edu	$+0.157$	13,670
English	GneissWeb Tech	$+0.154$	13,670
English	GneissWeb Edu	$+0.143$	13,670

Appendix

Table 5: Full language-stratified HC correlations. For JQL-Edu and FineWeb2-HQ, “English” rows reflect host-primary language classification; inference excluded documents mapped as English.

TLD Category	HC vs. DCLM $\rho$	$N$ hosts
Other	$-0.159$	659
Academic (.edu, .ac.*)	$-0.098$	766
Commercial (.com, .net)	$-0.028$	13,470
Country-code	$+0.004$	10,939
Nonprofit (.org)	$+0.032$	1,881
Government (.gov.*)	$+0.050$	195

Appendix

Table 6: HC–DCLM Spearman

\rho

by TLD macro-category. Negative within academic and commercial TLDs reflects DCLM’s language inflation of non-English domains.

Stratum	Classifier	Mean	Median	P10	P90
English	DCLM	0.041	0.001	0.000	0.065
English	FineWeb-Edu	0.220	0.203	0.041	0.408
English	Nemotron Quality	0.541	0.487	0.396	0.785
English	JQL-Edu	-4.850	-5.409	-8.946	0.190
English	FineWeb2-HQ	0.242	0.029	0.000	0.958
Non-English	DCLM	0.367	0.289	0.002	0.892

Appendix

Table 7: Score distributions by language stratum (selected classifiers). JQL-Edu scores are log-probabilities.

Classifier	$n_{\text{hosts}}$	$n_{\text{docs}}$	$\bar{m}$	$\sigma^{2}_{\text{btw}}$	$\sigma^{2}_{\text{wtn}}$	ICC1
Nemotron Quality	27,950	104,098	3.72	0.0195	0.0105	0.650
Nemotron Info	27,950	104,098	3.72	0.0379	0.0222	0.630
Nemotron Edu	27,950	104,098	3.72	0.0281	0.0190	0.596
GneissWeb Tech	27,950	104,098	3.72	0.0634	0.0461	0.579
DCLM	27,950	104,098	3.72	0.0596	0.0461	0.564
FineWeb-Edu	27,950	104,098	3.72	0.0131	0.0109	0.545

Appendix

Table 8: Full ICC1 table (

n\geq 3

docs per host). ICC1 stable across

n\geq 3/5/10

thresholds.

Dataset	Description and role (research question)
CC-MAIN-2024-10	Crawl (WARC) for primary crawl; we extract text (Trafilatura) and score it ourselves (224 WARCs, RQ1–RQ4).
CC-MAIN-2023-40	Crawl (WARC) for second snapshot and the language/robustness checks (32 WARCs, RQ2).
CC-MAIN-{2021-49, 2022-49, 2023-23, 2023-40, 2024-10, 2024-18}	Crawl (host stats from six crawls) for the temporal panel (RQ4).
Graph release (Sep 2023–Feb 2024)	Aligned host-level graph with harmonic centrality and PageRank per host (RQ2, App. A ).
Later graph releases (2024-May, 2024-Nov, 2025-May)	Host HC / PR ranks at $\sim$ 3–15 months later, for rank stability (App. A ).
CC host index	Per-host known/fetched URL counts and crawl coverage (RQ4).

Appendix

Table 9: Datasets used, by data regime; the research question each supports is noted in parentheses. WARC crawls are extracted and scored by us (classified-sample regime); the host index and GneissWeb annotations are Common Crawl’s precomputed full-crawl artifacts.

Classifier	Family	Hugging Face model	Notes
DCLM	fastText	mlfoundations/fasttext-oh-eli5	OH-2.5 + ELI5 vs. RefinedWeb, bigram
GneissWeb Quality	fastText	ibm-granite/GneissWeb.Quality_annotator	quality annotator
GneissWeb Sci	fastText	ibm-granite/GneissWeb.Sci_classifier	science topic
GneissWeb Edu	fastText	ibm-granite/GneissWeb.Edu_classifier	education topic
GneissWeb Tech	fastText	ibm-granite/GneissWeb.Tech_classifier	technology topic
GneissWeb Med	fastText	ibm-granite/GneissWeb.Med_classifier	medical topic

Appendix

Table 10: Exact Hugging Face models used to classify pages in the classified-sample regime. FineWeb-Edu is a compute-efficient fastText re-implementation ( Tsui and Nguyen, 2024 ) of the original Transformer classifier, at comparable accuracy. The Transformer classifiers (JQL-Edu, FineWeb2-HQ; the latter is the classifier of Messmer et al., 2025 ) are frozen-embedding models with lightweight per-language heads and are run on non-English documents only.

Figure 9: Full budget sweep behind Figure 2 : high-quality documents collected as the crawl budget grows, divided by what a random crawl collects at the same budget (log-scaled

y

-axis; the dashed line at

1

means “no better than random”). Ordering hosts by harmonic centrality ( HC ), PageRank ( PR ), or Common Crawl’s current allocation hugs the random line and stays far below the oracle ; the shaded band shows the gap between centrality-based yield and the oracle upper bound, e.g, widest at small budgets, where crawl order matters most, and closing as the budget approaches a full crawl. For DCLM, centrality is even worse than random.

Figure 10: The coverage–quality disconnect across 6 Common Crawl snapshots (2021–2024). (a) The Spearman correlation between crawl coverage and host quality density is near zero and negative in every crawl (-0.057 to -0.151). (b) Each crawl leaves tens of millions of high-quality URLs uncrawled on hidden-gem hosts. The misallocation is a persistent, not a single-crawl, artifact—the plotted form of Table 1 .

Figure 11: Where the missed high-quality budget sits, by (a) top-level domain and (b) content category (aggregates only; no host is named). Country-code publisher TLDs are over-represented relative to their host count— .uk alone holds 23% of the missed budget from just 6.1% of hosts at 34% median coverage. Among the hosts a keyword classifier can label, news/media and scholarly hosts dominate; the remaining

\sim

80% is uncategorised (a keyword lower bound), so panel (b) shows only the labelled share and omits the “Other” bucket. “Reference” denotes encyclopedic/reference sites (e.g. Wikipedia-family, dictionaries, encyclopedias); it is near zero because the major reference sites are already well crawled, so only a few dozen such hosts remain under-crawled gems.

Figure 12: Pairwise Spearman rank correlation between the twelve classifiers’ document scores (462,494 core documents; the diagonal is $1$ by construction). The scale runs from $-1$ (blue) through $0$ (white) to $+1$ (red). DCLM is largely orthogonal to the other quality signals and negative against the Nemotron heads, whereas the GneissWeb topic scorers and the Nemotron heads each form positively correlated blocks. This is the full pairwise view behind the single-classifier robustness of the coverage result (Figure 8 ).

Explore similar work

Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality
Jun 9, 2026Vedant Badoni, Danqi Chen, Xinyi WangLanguage Model Pretraining
Princeton Language and Intelligence
Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora
Sep 4, 2026Chirag Garg, Eelaaf Zahid, Farhan Ahmed +3Domain-Adaptive Pretraining
IBM Research
ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining
Sep 28, 2026Zichun Yu, Jiarui Yan, Shlok Sanghvi +2Language Model Pretraining Document Information Extraction
Language Technologies Institute, Carnegie Mellon University

Jun 9, 2026J/K move · Enter open · S save

Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

Vedant Badoni, Danqi Chen, Xinyi Wang

Princeton Language and Intelligence

Language Model Pretraining

Open paper AskPDF arXiv

The performance of modern language models depends critically on pretraining data composition. Yet existing data selection methods rely on auxiliary classifiers for document scoring or mixture optimization, adding computational overhead and dependence on labeled data. We propose WebGraphMix, a lightweight data selection framework that computes structural centrality scores over the Common Crawl host-level web graph and uses them to vary the proportion of central versus peripheral documents in the pretraining mixture. We hypothesize that central hosts expose models to reusable abstractions, while peripheral hosts encode specialized, long-tail knowledge. WebGraphMix computes centrality scores efficiently at web scale, requiring no model training, labeled data, or downstream supervision. We integrate WebGraphMix into the DataComp-LM pipeline and train models at 400M and 1B parameter scales with 8B and 28B tokens respectively, evaluating on 23 tasks ranging from factual knowledge to symbolic reasoning. Our experiments show that central and peripheral web regions encode complementary capabilities. Mixture combining both at a ratio of 1:1 achieves 41.4% on average, compared to 39.8% for uniform sampling. Combining structural scores with document-level quality classifier scores further improves performance to 43.8%. These findings demonstrate that web graph topology is a meaningful axis for pretraining data curation, capturing information that is largely orthogonal to existing content-based approaches.

Abstract

Explore similar work

Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora

ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining