Authors: Adam Visokay, Soumyakanti Pan, Molly Griston, Polina Iasakova, Mark Nepf, Denis Peskoff, Riya Sinha, Jikai Sun, Isabelle Thiara, Letao Wang, Yiwei Wu, Lauren Yan, Tyler H. McCormick
Organizations: School of Information, University of California, Berkeley · Department of Statistics, University of Washington · Department of Physics, University of Colorado, Boulder · Department of Statistics, Oregon State University · Evans School of Public Policy, University of Washington · Department of Information, University of Texas at Austin · Hubbard School of Journalism and Mass Communication, University of Minnesota · Department of Sociology, Washington University in St. Louis · Department of Family Medicine & Community Health, University of Kansas Medical Center · Department of Sociology, University of Washington
In less than five years, generative artificial intelligence (AI) has dramatically reshaped the public-facing web. Using a sample of more than 5 million crawlable webpages, we demonstrate that the share containing AI-generated text has increased precipitously, but unevenly. Our analyses reveal substantial differences in AI composition across language and domain categories (e.g., .edu, .com, or .gov). A higher fraction of English-language webpages contain AI-generated text compared to pages in other languages, and prevalence is substantially lower on .gov and .edu domains compared to .com. These distinctions persist even after accounting for compositional effects. We also find heterogeneity based on a page's position in the connectivity structure of the web. AI-generated text is most common and has increased most rapidly in pages that are the least central in the graph, which drives much of the overall growth in AI-generated text. A slower increase has taken place for more central pages, with the most central tier emerging as the least likely to contain AI-generated text. Methodologically, we use multiple AI detection tools, and generate estimates using a Bayesian hierarchical model that incorporates corrections for the detectors' statistical properties (both false positives and false negatives) and allows for different detector performance across groups, while propagating uncertainty in detector error rates into prevalence estimates. Our work suggests that AI is a prominent, but varied, contributor to public information and internet-based communication.
Figures & tables
Figure 2.1 : Estimated overall prevalence and its rate of change of pages with fully AI-generated or AI-edited text in Common Crawl. (a) The dashed gray line shows Open Pangram flags divided by analyzed pages in each crawl. The blue curve is the poststratified posterior mean, with cells weighted by their number of eligible pages, and pointwise 95% credible intervals shaded. The raw proportions are unweighted; their comparison with the poststratified estimate includes both detector correction and population weighting. (b) Gray segments show the change between consecutive crawl estimates divided by elapsed time, in percentage points per year (365.25 days). Blue segments give endpoint-based averages for the three comparison periods described in the text. Shading in panel b gives pointwise 95% credible intervals calculated from matched joint posterior draws. All 71 crawl estimates are used. Dates are crawl midpoints; prevalence describes content present in each snapshot.
Figure 2.2 : Estimated prevalence and growth of pages with fully AI-generated or AI-edited text across the Common Crawl’s link network. (a) Shares of domains in Common Crawl’s link graph, set by the PageRank tier cutoffs in Table 4.1 , and estimated shares of eligible pages in August 2026 using the model’s page-count weights. (b) Prevalence by PageRank tier, the share of the tier’s eligible pages with fully AI-generated or AI-edited text. Lines show posterior means and shaded bands show 95% credible intervals. (c) Each tier’s share of the overall prevalence increase from December 2021 to August 2026 (circles; 95% credible intervals), compared with its share of eligible pages in August 2026 (diamonds). With page share wjk and prevalence pjk of tier j in crawl k , and overall prevalence θˉk=∑jwjkpjk , the growth share is (wjk1pjk1−wjk0pjk0)/(θˉk1−θˉk0) for the first and last crawls k0 and k1 of the comparison, calculated in each posterior draw using the composition observed at each date. The difference denotes final page share subtracted from growth share in percentage points. These contributions combine within-tier prevalence change and changes in tier representation; the final page-share comparison does not separate them. Colors identify tiers from Top (most central) to Tail (least central); dates are crawl midpoints.
Figure 2.3 : Language differences in estimated prevalence and growth of pages with fully AI-generated or AI-edited text in Common Crawl. (a) Page-count-weighted prevalence of fully AI-generated or AI-edited text, with 95% credible intervals. Groups are English only, no English detected, and English plus another detected language (bilingual); pages with unknown language are excluded from this figure. (b) Average annual increases from February 2, 2023 to January 19, 2025, and from that date to August 13, 2026. Changes use elapsed days (365.25 days per year) and are averages between crawl snapshots, not the date growth changed. Intervals are the 2.5th and 97.5th percentiles of matched posterior-draw contrasts.
Figure 2.4 : Estimated prevalence of pages in Common Crawl with fully AI-generated or AI-edited text over time and across domain categories at a common page composition. (a) Estimated proportions of eligible archived pages containing fully AI-generated or AI-edited text, with posterior means and pointwise 95% credible intervals. Eligible pages contain at least 500 extracted text characters. Dates are crawl midpoints; prevalence is weighted by modeled page counts. (b) Prevalence in August 2026 under each category’s actual page composition (filled circles) and a common composition (open diamonds), with 95% credible intervals. Both series are model estimates. The common composition is the final crawl’s page-count distribution of centrality, language, and length, summed across all six categories; identical weights are used for every category within each posterior draw. Combinations absent from a category are predicted by the fitted model. Represented combinations cover 22.5% of the reference distribution for .gov and 16.6% for .edu .
PageRank
Nominal
Share of (%)
Extractable
Stratum
quantile
allocation
Census
Pages
Exposure
rate
Top
(0,10−5]
3,386
✓
4.2
30.2
0.608
Upper-middle
(10−5,10−4]
2,370
✓
5.1
10.1
0.562
Middle
(10−4,10−3]
2,740
✓
9.4
11.2
0.584
Lower-middle
(10−3,10−2]
3,000
15.0
11.8
0.617
Bottom
(10−2,10−1]
3,748
28.3
15.4
0.610
Table 4.1 : Sampling strata and the two estimand denominators. Strata are cut at fixed quantiles of each crawl’s PageRank ranking.
Figure 4.1 : This figure describes the full sample drawn from 71 Common Crawls. All quantities are population estimates formed from the per-crawl cell denominators, which scale the sample up by its inclusion probabilities; the dashed line marks ChatGPT’s public launch (2022-11-30). (a) Captures per crawl and the subset passing the 500-character extractable-text screen, which is the denominator of every rate we report. (b) Language composition of that denominator over time as a four-way partition. (c) Length composition on the word-count bins against which the primary detector’s error rates are published. (d) Share of extractable-text pages by PageRank stratum. Panels (b)–(d) are the three axes the measurement model conditions on, and the drift in (c) matters independently of prevalence: detectors are weakest on short text, so a shift toward longer pages raises measured sensitivity over the series.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Corrects for
Study
Corpus
Sampling
Detection
FP
FN
Uncertainty
Graphite (2026)
Common Crawl, English articles
Non-probability
Mean detector score
✓
—
None reported
Dolezal et al. (2026)
Internet Archive, English captures
One URL per host
Pangram v3
—
—
None reported
Pew Research Center (2026)
Common Crawl, English pages
Simple random, 10k pages per crawl
Open Pangram; Pangram 3.3 check
—
—
None reported
Liang et al. (2024) ; Liang et al. (2025)
Scientific corpora
Near-census
Distributional estimator
✓ a
✓ a
Validation-based
Elazar and Antoniak (2026)
arXiv
Near-census
Distributional + Pangram
✓
✓ b
95% CI
Appendix
Table A.2 : Comparison of existing studies and this study
Figure B.1 : Detector performance at deployed thresholds. (A) Share of AI-involved text flagged on each labeled corpus, with 95% Wilson intervals. (B) Share of human text flagged on the same corpora and on pre-2022 Common Crawl pages, which are the study population. Hollow markers indicate a corpus that overlaps the detector’s training data; those rows are excluded from calibration. Pangram 4 was run on the AI-edited EditLens texts only, and on a random subsample of 20,100 control pages. Binoculars is shown on non-English benchmark text for comparison but is not run on non-English pages.
Figure B.2 : Sensitivity by text length at deployed thresholds. Each panel is one labeled corpus and positive class; the horizontal axis is text length in words, in the three bins used for calibration (under 150, 150 to 300, over 300). Vertical bars are 95% Wilson intervals. Bins with fewer than 30 AI texts are omitted, which removes the shortest bin for AITDNA. Hollow markers indicate a corpus that overlaps the detector’s training data; those rows are excluded from calibration. Pangram 4 was run on the AI-edited EditLens texts only.
Figure B.3 : The COLING 2025 multilingual set by language, at deployed thresholds. (A) Share of AI-generated text flagged. (B) Share of human text flagged. Horizontal bars are 95% Wilson intervals. English texts are scored at each detector’s English threshold and all other languages at its non-English threshold. The set contains no human-written Italian text, so that row is empty in panel B. Binoculars is shown for comparison but is not run on non-English pages.
Figure B.4 : Detector flag rates by share of AI-generated tokens. Documents from the AITDNA benchmark are binned by the proportion of tokens written by an AI model, with human-written documents (share =0 ) at left; points give the percentage of documents in each bin that a detector flagged as fully AI-generated or AI-edited, bars give 95% intervals, and n is the number of documents per bin. Flag rates rise with the AI-generated share and the detectors diverge in where they err: the perplexity-based Binoculars flags fewer than half of the documents that are more than three-quarters fully AI-generated, while the supervised Pangram models flag a quarter or more of those with under a quarter AI-generated. Each detector measures a different operationalization of AI involvement, so none maps cleanly onto authorship.
Flagged by both
Data
Pair
n
Observed
Expected a
ϕ
ϕ within length b
Human text: pre-2022 control crawls (every flag is a false positive)
English, all pages
LL–FA
1,769,938
5,278
5,422
−0.002
—
English, Pangram 4 subsample
LL–P4
15,376
0
0.2
−0.004
—
Non-English, Pangram 4 subsample
LL–P4
4,724
0
0.0
—
—
AI-involved text: benchmarks
Appendix
Table B.3 : Dependence between detectors’ errors, within true class
Share of pages flagged
Context
Pages
Open Pangram
Binoculars-Falcon
All English-only pages
1,336,683
4.5%
3.0%
Top-level domain
.com
829,996
5.2%
3.0%
.org
198,951
4.1%
2.7%
.net
38,121
3.8%
3.9%
Appendix
Table B.4: False-positive rates on pre-2022 English pages, by context
Figure C.1 : Separating estimated growth in prevalence of pages in Common Crawl with fully AI-generated or AI-edited text from changes in the distribution of pages. (a) Each tier’s share of modeled eligible pages at the start and end of the December 2021 to August 2026 comparison. (b) Contribution to the overall prevalence increase, in percentage points. Blue bars show wˉj(pjk1−pjk0) , the component from prevalence change within tiers. Hatched bars show pˉj(wjk1−wjk0) , the component from changing page shares. Bars denote separate components measured from zero; diamonds show their sum wjk1pjk1−wjk0pjk0 . Bars and diamonds show posterior means. Error bars show 95% credible intervals from paired posterior draws. Overbars denote averages of the two endpoint values. The Bottom and Tail totals sum the two least-central tiers. Within-tier change includes changes in the pages and domains represented within each tier.
Figure C.2 : Change in the English advantage in annual growth under alternative compositions. Later-minus-earlier change in the English-only minus no-English difference in average annual increase, comparing February 2, 2023 to January 19, 2025 with January 19, 2025 to August 13, 2026, in percentage points per year; positive values indicate a larger English advantage in the later period. A profile is a combination of centrality tier, TLD and length. All pages : all represented pages. Shared profiles, own composition : the profiles observed for every language group at every endpoint, with each group keeping its own composition on them. Shared profiles, common composition : the same profiles, with every group given the same composition (the average over the six most recent crawls). Points are posterior means and bars are 2.5th to 97.5th percentiles of matched posterior-draw contrasts.
Figure C.3 : Prevalence of pages with fully AI-generated or AI-edited text by word-count level. (a) Posterior mean and 95% credible interval over the crawls. (b) Posterior mean and 95% credible interval in August 2026 (circles), and the share of eligible pages in each word-count level (diamonds).
Language level ( L )
Estimated prevalence
Standardized by
Centrality
TLD
Length
All three
English only
32.8
33.7
32.9
32.0
32.5
(32.2–33.4)
(33.1–34.3)
(32.3–33.4)
(31.4–32.6)
(31.9–33.0)
No English detected
17.0
16.5
17.3
17.0
16.8
(15.4–18.4)
(15.0–17.9)
(15.7–18.7)
(15.4–18.4)
(15.3–18.2)
English + another language
20.2
20.1
20.2
21.1
21.4
Appendix
Table C.5 : Estimated prevalence of pages in Common Crawl with fully AI-generated or AI-edited text (%) in the August 2026 snapshot by language level, unstandardized and standardized to the common distribution of centrality tier, TLD category and length level, each separately and jointly. The estimated prevalence uses each language level’s own composition of these attributes. Entries are posterior means with 95% credible intervals.
Model specification
Specificity model
Interactions
R
sδ,sp
sδ,se
εμ
sμ
Prevalence, %
Reported model
Full
No
12
0.5
0.25
0.01
1
23.6 (22.9–24.4)
Full sp , Youden constraint
Full
No
12
0.5
0.25
0.01
1
22.6 (21.9–23.3)
Full sp , with interactions
Full
Yes
12
0.5
0.25
0.01
1
23.9 (23.0–25.3)
Reduced sp
Reduced
No
12
0.5
0.25
0.01
1
22.9 (22.2–23.6)
sδ,sp=0.25
Reduced
No
12
0.25
0.25
0.01
1
23.1 (22.4–23.8)
sp , sδ,sp=0.75
Reduced
No
12
0.75
0.25
0.01
1
22.8 (22.2–23.5)
Appendix
Table D.6 : Settings and prevalence in the August 2026 snapshot under the reported model and 16 alternative specifications. Specificity effects are full (centrality tier, word count and TLD for both detectors) or reduced (word count for both detectors and TLD for Pangram); R is the number of basis functions of the spline in the temporal effect (– if there is no temporal effect); sδ,sp and sδ,se are the scales of the drifts in specificity and sensitivity after the control crawls (– if there is no drift), and εμ and sμ the location and scale of the prior of the intercept, μ∼N(logit(εμ),sμ2) . Prevalence is the posterior mean with its 95% credible interval. Table D.7 compares the fits by WAIC.
Model specification
Δlppd
p^WAIC
Δ WAIC
Reported model (full specificity effects)
0
77.9
0
Full specificity effects, Youden constraint
−274
75.9
+544
Full specificity effects, with interactions
+882
129.4
−1,662
Reduced specificity effects
−1,205
62.8
+2,380
Reduced specificity effects, sδ,sp=0.25
−1,221
64.1
+2,414
Reduced specificity effects, sδ,sp=0.75
−1,200
63.0
+2,370
Appendix
Table D.7 : Page-level WAIC under the reported model and sixteen alternative specifications, computed from all posterior draws. Δlppd and Δ WAIC are differences from the reported model, whose values are lppd=−2,157,455 , p^WAIC=77.9 and WAIC=4,315,065 ; p^WAIC is shown as is. Since ΔWAIC=−2(Δlppd−Δp^WAIC) , lower WAIC indicates a better fit and negative values of Δ WAIC indicate a better fit than the reported model.
Figure D.1 : Reported model and sixteen alternative specifications (settings in Table D.6 ). (a) Estimated prevalence of pages with fully AI-generated or AI-edited text in August 2026; points are posterior means and bars are 95% credible intervals. (b) Difference in WAIC from the reported model; points are differences and bars are ±1.96 standard errors of the difference. The horizontal axis is symmetric logarithmic, and a positive difference indicates a worse fit. Point color is the smallest tail effective sample size of the fit over the monitored quantities (Table D.10 ). The hollow marker is the fit without a temporal effect.
Model specification
English only
No English
Bilingual
English minus no English
Change in growth gap
Bilingual minus no English
Reported model (full specificity effects)
32.8
17.0
20.2
15.8
3.0
3.2
(32.2–33.4)
(15.4–18.4)
(19.3–21.0)
(14.4–17.4)
(2.8–3.2)
(1.6–4.9)
Full specificity effects, Youden constraint
32.7
14.5
19.9
18.2
3.0
5.4
(32.1–33.3)
(13.3–16.0)
(19.1–20.8)
(16.7–19.5)
(2.8–3.3)
(3.8–6.9)
Full specificity effects, with interactions
32.7
17.4
21.0
15.2
2.7
3.6
(32.1–33.3)
(15.2–20.5)
(20.1–22.0)
(12.2–17.4)
(2.4–3.0)
(0.5–5.9)
Appendix
Table D.8 : Language comparisons in the August 2026 snapshot under the reported model and sixteen alternative specifications (settings in Table D.6 ). Each entry is the posterior mean, with its 95% credible interval below, and prevalence is in percent. The English-only minus no-English difference is in percentage points; the change in the growth gap is the later minus the earlier annual growth rate of that difference (February 2023 to January 2025, January 2025 to August 2026), in percentage points per year; bilingual minus no English is the difference in percentage points. All intervals are computed within matched posterior draws.
Model specification
Mean difference non-negative
Credibly positive
Reported model (full specificity effects)
February 2026
April 2026
Full specificity effects, Youden constraint
October 2025
February 2026
Full specificity effects, with interactions
December 2025
May 2026
Reduced specificity effects
October 2025
January 2026
Reduced specificity effects, sδ,sp=0.25
October 2025
February 2026
Reduced specificity effects, sδ,sp=0.75
September 2025
December 2025
Appendix
Table D.9 : When bilingual prevalence first exceeds no-English prevalence under each specification (settings in Table D.6 ): the first crawl at which the posterior mean difference is non-negative, and the first crawl at which the difference is positive with posterior probability above 0.975.
Figure D.2 : Prior and posterior of the shifts in sensitivity and specificity after the control crawls (logit scale) in the reported model. The grey bar is the central 95% prior interval, and the blue point and bar are the posterior mean and 95% credible interval. The right column gives the posterior mean in units of the prior standard deviation.
Figure D.3 : False-positive rate implied by the control-period specificity, by centrality tier, word-count level and TLD category, for Pangram (top) and Binoculars (bottom). Dots are the posterior mean and 95% credible interval of the average over the English-only pages of the control crawls in each category, and diamonds are the observed flag rates of those pages.
Specification
Draws
R
ESS bulk
ESS tail
J , Pangram, English
J , Pangram, non-English
J , Binoculars , English
Reported model (full specificity effects)
8,000
1.005
880
736
0.58
0.37
− 0.06
(0.57, 0.59)
(0.35, 0.39)
( − 0.06, − 0.05)
Full specificity effects, Youden constraint
8,000
1.004
813
1,399
0.58
0.36
0.00
(0.57, 0.59)
(0.34, 0.37)
(0.00, 0.00)
Full specificity effects, with interactions
8,000
1.007
484
215
0.57
0.35
− 0.06
(0.56, 0.58)
(0.33, 0.37)
( − 0.06, − 0.05)
Appendix
Table D.10 : Convergence diagnostics and Youden index by fit. Draws are the posterior draws over four chains. R is the largest rank-normalized split- R , and ESS the smallest bulk and tail effective sample size, over all parameters and derived quantities. J is the posterior mean (95% credible interval) of the Youden index of the control-period calibration in the least specific observed cell of the stratum, for Pangram in english-stratum and non-English pages and for Binoculars in English-stratum pages (the constraint binds for Binoculars in the fit with a positive Youden index). No fit has divergent transitions, transitions at the maximum tree depth or a low energy Bayesian fraction of missing information.
The proliferation of AI-generated and AI-assisted text on the internet is feared to contribute to a degradation in semantic and stylistic diversity, factual accuracy, and other negative developments (sometimes subsumed under the Dead Internet Theory). What has hindered answering these questions is that it has not been understood just how much of the internet is actually AI-generated or AI-edited. To this end, we construct a representative sample of websites published on the internet between 2022 and 2025 using the Internet Archive, and apply a state-of-the-art AI text detector on them. We find that by mid-2025, roughly 35% of newly published websites were classified as AI-generated or AI-assisted, up from zero before ChatGPT's launch in late 2022. We also find statistically significant evidence for some of the identified hypotheses; for example, that increases in AI-generated text on the internet correlate negatively with semantic diversity and positively with the prevalence of positive sentiment. We do not, however, find statistically significant evidence supporting the hypothesis that an increased rate of AI-generated text on the internet decreases factual accuracy or stylistic diversity. Notably, this diverges from public perception, which we measure in a user study, where the majority of US adults turned out to believe in all four of the above-mentioned hypotheses. Individuals who do not use AI or use it infrequently tend to believe in these negative impacts more than those who use it frequently; similarly, individuals who hold negative views of AI tend to believe in these hypotheses more than those with favorable views of the technology.
Jonas Dolezal, Sawood Alam, Mark Graham +1
Imperial College London · Internet Archive · Stanford University
Many news reports have claimed that machine-generated text (MGT) from large language models (LLMs) is taking over the web. However, these claims are typically not based on a representative sample of the web, and the methodology underlying them is often opaque. Consequently, we lack an understanding of the true prevalence and characteristics of MGT on the web. We describe DeGenTWeb, which systematically identifies MGT-dominant websites: sites whose informational prose content is mostly MGT, with little human input. We show how to adapt detectors of MGT for use on web pages, and how to aggregate detection results from multiple pages on a site for accurate site-level categorization. DeGenTWeb mislabels only 0.12% of human-authored sites as MGT-dominant. We find that 20.9% of the sites archived by Common Crawl between January and July 2026 and 13.6% of the sites linked from Bing's search results in 2025 are MGT-dominant. These figures are lower bounds on the prevalence of MGT-dominant sites. We also find that the fraction of MGT-dominant sites is growing over time, and that many such sites share the same owner. We will release all of our code and data after publication.
Sichang Steven He, Calvin Ardi, Ramesh Govindan +1
Word-level detectors identify unedited AI-generated text almost perfectly, but the literature documents their brittleness under rewording, and a word-level score neither characterizes a text nor identifies which AI model wrote it. We ask whether AI-generated text can be identified one level deeper, from structural signatures: how information is presented, in what order, with what evidence, and in what voice. We replicate StoryScope (Russell et al., 2026), which showed such patterns for AI-generated fiction, on commercial content: 2,250 pre-ChatGPT human blog posts from 268 company domains against 11,250 AI mirrors from five frontier models. A 203-feature instrument, applied by an LLM and validated in a human gold-annotation session (human-human kappa 0.939, human-model 0.951), detects AI posts from its 176 structural features alone at 97.0 macro-F1 on held-out companies, nearly unchanged (96.1) when every AI post is reworded by its own model. The signal characterizes and attributes: AI posts share a tidy, self-announcing shape, 68.6% are attributed to the correct source against a 16.7% chance rate, and human posts occupy rare structural configurations. All effects replicate StoryScope's, consistent in direction and at least as large in magnitude. We release pipeline, instrument, prompts, code, and aggregate artifacts.