cs.AIOct 27, 2026
SaveThe uneven rise of AI-generated web content
Organizations: School of Information, University of California, Berkeley · Department of Statistics, University of Washington · Department of Physics, University of Colorado, Boulder · Department of Statistics, Oregon State University ·
Abstract
In less than five years, generative artificial intelligence (AI) has dramatically reshaped the public-facing web. Using a sample of more than 5 million crawlable webpages, we demonstrate that the share containing AI-generated text has increased precipitously, but unevenly. Our analyses reveal substantial differences in AI composition across language and domain categories (e.g., .edu, .com, or .gov). A higher fraction of English-language webpages contain AI-generated text compared to pages in other languages, and prevalence is substantially lower on .gov and .edu domains compared to .com. These distinctions persist even after accounting for compositional effects. We also find heterogeneity based on a page's position in the connectivity structure of the web. AI-generated text is most common and has increased most rapidly in pages that are the least central in the graph, which drives much of the overall growth in AI-generated text. A slower increase has taken place for more central pages, with the most central tier emerging as the least likely to contain AI-generated text. Methodologically, we use multiple AI detection tools, and generate estimates using a Bayesian hierarchical model that incorporates corrections for the detectors' statistical properties (both false positives and false negatives) and allows for different detector performance across groups, while propagating uncertainty in detector error rates into prevalence estimates. Our work suggests that AI is a prominent, but varied, contributor to public information and internet-based communication.
Figures & tables
Figure 2.1 : Estimated overall prevalence and its rate of change of pages with fully AI-generated or AI-edited text in Common Crawl. (a) The dashed gray line shows Open Pangram flags divided by analyzed pages in each crawl. The blue curve is the poststratified posterior mean, with cells weighted by their number of eligible pages, and pointwise 95% credible intervals shaded. The raw proportions are unweighted; their comparison with the poststratified estimate includes both detector correction and population weighting. (b) Gray segments show the change between consecutive crawl estimates divided by elapsed time, in percentage points per year (365.25 days). Blue segments give endpoint-based averages for the three comparison periods described in the text. Shading in panel b gives pointwise 95% credible intervals calculated from matched joint posterior draws. All 71 crawl estimates are used. Dates are crawl midpoints; prevalence describes content present in each snapshot.
Figure 2.2 : Estimated prevalence and growth of pages with fully AI-generated or AI-edited text across the Common Crawl’s link network. (a) Shares of domains in Common Crawl’s link graph, set by the PageRank tier cutoffs in Table 4.1 , and estimated shares of eligible pages in August 2026 using the model’s page-count weights. (b) Prevalence by PageRank tier, the share of the tier’s eligible pages with fully AI-generated or AI-edited text. Lines show posterior means and shaded bands show 95% credible intervals. (c) Each tier’s share of the overall prevalence increase from December 2021 to August 2026 (circles; 95% credible intervals), compared with its share of eligible pages in August 2026 (diamonds). With page share and prevalence of tier in crawl , and overall prevalence , the growth share is for the first and last crawls and of the comparison, calculated in each posterior draw using the composition observed at each date. The difference denotes final page share subtracted from growth share in percentage points. These contributions combine within-tier prevalence change and changes in tier representation; the final page-share comparison does not separate them. Colors identify tiers from Top (most central) to Tail (least central); dates are crawl midpoints.
Figure 2.3 : Language differences in estimated prevalence and growth of pages with fully AI-generated or AI-edited text in Common Crawl. (a) Page-count-weighted prevalence of fully AI-generated or AI-edited text, with 95% credible intervals. Groups are English only, no English detected, and English plus another detected language (bilingual); pages with unknown language are excluded from this figure. (b) Average annual increases from February 2, 2023 to January 19, 2025, and from that date to August 13, 2026. Changes use elapsed days (365.25 days per year) and are averages between crawl snapshots, not the date growth changed. Intervals are the 2.5th and 97.5th percentiles of matched posterior-draw contrasts.
Figure 2.4 : Estimated prevalence of pages in Common Crawl with fully AI-generated or AI-edited text over time and across domain categories at a common page composition. (a) Estimated proportions of eligible archived pages containing fully AI-generated or AI-edited text, with posterior means and pointwise 95% credible intervals. Eligible pages contain at least 500 extracted text characters. Dates are crawl midpoints; prevalence is weighted by modeled page counts. (b) Prevalence in August 2026 under each category’s actual page composition (filled circles) and a common composition (open diamonds), with 95% credible intervals. Both series are model estimates. The common composition is the final crawl’s page-count distribution of centrality, language, and length, summed across all six categories; identical weights are used for every category within each posterior draw. Combinations absent from a category are predicted by the fitted model. Represented combinations cover 22.5% of the reference distribution for .gov and 16.6% for .edu .
| PageRank | Nominal | Share of (%) | Extractable | |||
|---|---|---|---|---|---|---|
| Stratum | quantile | allocation | Census | Pages | Exposure | rate |
| Top | 3,386 | ✓ | 4.2 | 30.2 | 0.608 | |
| Upper-middle | 2,370 | ✓ | 5.1 | 10.1 | 0.562 | |
| Middle | 2,740 | ✓ | 9.4 | 11.2 | 0.584 | |
| Lower-middle | 3,000 | 15.0 | 11.8 | 0.617 | ||
| Bottom | 3,748 | 28.3 | 15.4 | 0.610 | ||
Table 4.1 : Sampling strata and the two estimand denominators. Strata are cut at fixed quantiles of each crawl’s PageRank ranking.
Figure 4.1 : This figure describes the full sample drawn from 71 Common Crawls. All quantities are population estimates formed from the per-crawl cell denominators, which scale the sample up by its inclusion probabilities; the dashed line marks ChatGPT’s public launch (2022-11-30). (a) Captures per crawl and the subset passing the 500-character extractable-text screen, which is the denominator of every rate we report. (b) Language composition of that denominator over time as a four-way partition. (c) Length composition on the word-count bins against which the primary detector’s error rates are published. (d) Share of extractable-text pages by PageRank stratum. Panels (b)–(d) are the three axes the measurement model conditions on, and the drift in (c) matters independently of prevalence: detectors are weakest on short text, so a shift toward longer pages raises measured sensitivity over the series.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Corrects for | ||||||
|---|---|---|---|---|---|---|
| Study | Corpus | Sampling | Detection | FP | FN | Uncertainty |
| Graphite (2026) | Common Crawl, English articles | Non-probability | Mean detector score | ✓ | — | None reported |
| Dolezal et al. (2026) | Internet Archive, English captures | One URL per host | Pangram v3 | — | — | None reported |
| Pew Research Center (2026) | Common Crawl, English pages | Simple random, 10k pages per crawl | Open Pangram; Pangram 3.3 check | — | — | None reported |
| Liang et al. (2024) ; Liang et al. (2025) | Scientific corpora | Near-census | Distributional estimator | ✓ a | ✓ a | Validation-based |
| Elazar and Antoniak (2026) | arXiv | Near-census | Distributional + Pangram | ✓ | ✓ b | 95% CI |
Table A.2 : Comparison of existing studies and this study
Figure B.1 : Detector performance at deployed thresholds. (A) Share of AI-involved text flagged on each labeled corpus, with 95% Wilson intervals. (B) Share of human text flagged on the same corpora and on pre-2022 Common Crawl pages, which are the study population. Hollow markers indicate a corpus that overlaps the detector’s training data; those rows are excluded from calibration. Pangram 4 was run on the AI-edited EditLens texts only, and on a random subsample of 20,100 control pages. Binoculars is shown on non-English benchmark text for comparison but is not run on non-English pages.
Figure B.2 : Sensitivity by text length at deployed thresholds. Each panel is one labeled corpus and positive class; the horizontal axis is text length in words, in the three bins used for calibration (under 150, 150 to 300, over 300). Vertical bars are 95% Wilson intervals. Bins with fewer than 30 AI texts are omitted, which removes the shortest bin for AITDNA. Hollow markers indicate a corpus that overlaps the detector’s training data; those rows are excluded from calibration. Pangram 4 was run on the AI-edited EditLens texts only.
Figure B.3 : The COLING 2025 multilingual set by language, at deployed thresholds. (A) Share of AI-generated text flagged. (B) Share of human text flagged. Horizontal bars are 95% Wilson intervals. English texts are scored at each detector’s English threshold and all other languages at its non-English threshold. The set contains no human-written Italian text, so that row is empty in panel B. Binoculars is shown for comparison but is not run on non-English pages.
Figure B.4 : Detector flag rates by share of AI-generated tokens. Documents from the AITDNA benchmark are binned by the proportion of tokens written by an AI model, with human-written documents (share ) at left; points give the percentage of documents in each bin that a detector flagged as fully AI-generated or AI-edited, bars give 95% intervals, and is the number of documents per bin. Flag rates rise with the AI-generated share and the detectors diverge in where they err: the perplexity-based Binoculars flags fewer than half of the documents that are more than three-quarters fully AI-generated, while the supervised Pangram models flag a quarter or more of those with under a quarter AI-generated. Each detector measures a different operationalization of AI involvement, so none maps cleanly onto authorship.
| Flagged by both | ||||||
| Data | Pair | Observed | Expected a | within length b | ||
| Human text: pre-2022 control crawls (every flag is a false positive) | ||||||
| English, all pages | LL–FA | 1,769,938 | 5,278 | 5,422 | — | |
| English, Pangram 4 subsample | LL–P4 | 15,376 | 0 | 0.2 | — | |
| Non-English, Pangram 4 subsample | LL–P4 | 4,724 | 0 | 0.0 | — | — |
| AI-involved text: benchmarks | ||||||
Table B.3 : Dependence between detectors’ errors, within true class
| Share of pages flagged | |||
| Context | Pages | Open Pangram | Binoculars-Falcon |
| All English-only pages | 1,336,683 | 4.5% | 3.0% |
| Top-level domain | |||
| .com | 829,996 | 5.2% | 3.0% |
| .org | 198,951 | 4.1% | 2.7% |
| .net | 38,121 | 3.8% | 3.9% |
Table B.4: False-positive rates on pre-2022 English pages, by context
Figure C.1 : Separating estimated growth in prevalence of pages in Common Crawl with fully AI-generated or AI-edited text from changes in the distribution of pages. (a) Each tier’s share of modeled eligible pages at the start and end of the December 2021 to August 2026 comparison. (b) Contribution to the overall prevalence increase, in percentage points. Blue bars show , the component from prevalence change within tiers. Hatched bars show , the component from changing page shares. Bars denote separate components measured from zero; diamonds show their sum . Bars and diamonds show posterior means. Error bars show 95% credible intervals from paired posterior draws. Overbars denote averages of the two endpoint values. The Bottom and Tail totals sum the two least-central tiers. Within-tier change includes changes in the pages and domains represented within each tier.
Figure C.2 : Change in the English advantage in annual growth under alternative compositions. Later-minus-earlier change in the English-only minus no-English difference in average annual increase, comparing February 2, 2023 to January 19, 2025 with January 19, 2025 to August 13, 2026, in percentage points per year; positive values indicate a larger English advantage in the later period. A profile is a combination of centrality tier, TLD and length. All pages : all represented pages. Shared profiles, own composition : the profiles observed for every language group at every endpoint, with each group keeping its own composition on them. Shared profiles, common composition : the same profiles, with every group given the same composition (the average over the six most recent crawls). Points are posterior means and bars are 2.5th to 97.5th percentiles of matched posterior-draw contrasts.
Figure C.3 : Prevalence of pages with fully AI-generated or AI-edited text by word-count level. (a) Posterior mean and 95% credible interval over the crawls. (b) Posterior mean and 95% credible interval in August 2026 (circles), and the share of eligible pages in each word-count level (diamonds).
| Language level ( ) | Estimated prevalence | Standardized by | |||
|---|---|---|---|---|---|
| Centrality | TLD | Length | All three | ||
| English only | 32.8 | 33.7 | 32.9 | 32.0 | 32.5 |
| (32.2–33.4) | (33.1–34.3) | (32.3–33.4) | (31.4–32.6) | (31.9–33.0) | |
| No English detected | 17.0 | 16.5 | 17.3 | 17.0 | 16.8 |
| (15.4–18.4) | (15.0–17.9) | (15.7–18.7) | (15.4–18.4) | (15.3–18.2) | |
| English + another language | 20.2 | 20.1 | 20.2 | 21.1 | 21.4 |
Table C.5 : Estimated prevalence of pages in Common Crawl with fully AI-generated or AI-edited text (%) in the August 2026 snapshot by language level, unstandardized and standardized to the common distribution of centrality tier, TLD category and length level, each separately and jointly. The estimated prevalence uses each language level’s own composition of these attributes. Entries are posterior means with 95% credible intervals.
| Model specification | Specificity model | Interactions | Prevalence, % | |||||
|---|---|---|---|---|---|---|---|---|
| Reported model | Full | No | 12 | 0.5 | 0.25 | 0.01 | 1 | 23.6 (22.9–24.4) |
| Full , Youden constraint | Full | No | 12 | 0.5 | 0.25 | 0.01 | 1 | 22.6 (21.9–23.3) |
| Full , with interactions | Full | Yes | 12 | 0.5 | 0.25 | 0.01 | 1 | 23.9 (23.0–25.3) |
| Reduced | Reduced | No | 12 | 0.5 | 0.25 | 0.01 | 1 | 22.9 (22.2–23.6) |
| Reduced | No | 12 | 0.25 | 0.25 | 0.01 | 1 | 23.1 (22.4–23.8) | |
| , | Reduced | No | 12 | 0.75 | 0.25 | 0.01 | 1 | 22.8 (22.2–23.5) |
Table D.6 : Settings and prevalence in the August 2026 snapshot under the reported model and 16 alternative specifications. Specificity effects are full (centrality tier, word count and TLD for both detectors) or reduced (word count for both detectors and TLD for Pangram); is the number of basis functions of the spline in the temporal effect (– if there is no temporal effect); and are the scales of the drifts in specificity and sensitivity after the control crawls (– if there is no drift), and and the location and scale of the prior of the intercept, . Prevalence is the posterior mean with its 95% credible interval. Table D.7 compares the fits by WAIC.
| Model specification | WAIC | ||
|---|---|---|---|
| Reported model (full specificity effects) | 0 | 77.9 | 0 |
| Full specificity effects, Youden constraint | 75.9 | ||
| Full specificity effects, with interactions | 129.4 | ||
| Reduced specificity effects | 62.8 | ||
| Reduced specificity effects, | 64.1 | ||
| Reduced specificity effects, | 63.0 |
Table D.7 : Page-level WAIC under the reported model and sixteen alternative specifications, computed from all posterior draws. and WAIC are differences from the reported model, whose values are , and ; is shown as is. Since , lower WAIC indicates a better fit and negative values of WAIC indicate a better fit than the reported model.
Figure D.1 : Reported model and sixteen alternative specifications (settings in Table D.6 ). (a) Estimated prevalence of pages with fully AI-generated or AI-edited text in August 2026; points are posterior means and bars are 95% credible intervals. (b) Difference in WAIC from the reported model; points are differences and bars are standard errors of the difference. The horizontal axis is symmetric logarithmic, and a positive difference indicates a worse fit. Point color is the smallest tail effective sample size of the fit over the monitored quantities (Table D.10 ). The hollow marker is the fit without a temporal effect.
| Model specification | English only | No English | Bilingual | English minus no English | Change in growth gap | Bilingual minus no English |
|---|---|---|---|---|---|---|
| Reported model (full specificity effects) | 32.8 | 17.0 | 20.2 | 15.8 | 3.0 | 3.2 |
| (32.2–33.4) | (15.4–18.4) | (19.3–21.0) | (14.4–17.4) | (2.8–3.2) | (1.6–4.9) | |
| Full specificity effects, Youden constraint | 32.7 | 14.5 | 19.9 | 18.2 | 3.0 | 5.4 |
| (32.1–33.3) | (13.3–16.0) | (19.1–20.8) | (16.7–19.5) | (2.8–3.3) | (3.8–6.9) | |
| Full specificity effects, with interactions | 32.7 | 17.4 | 21.0 | 15.2 | 2.7 | 3.6 |
| (32.1–33.3) | (15.2–20.5) | (20.1–22.0) | (12.2–17.4) | (2.4–3.0) | (0.5–5.9) |
Table D.8 : Language comparisons in the August 2026 snapshot under the reported model and sixteen alternative specifications (settings in Table D.6 ). Each entry is the posterior mean, with its 95% credible interval below, and prevalence is in percent. The English-only minus no-English difference is in percentage points; the change in the growth gap is the later minus the earlier annual growth rate of that difference (February 2023 to January 2025, January 2025 to August 2026), in percentage points per year; bilingual minus no English is the difference in percentage points. All intervals are computed within matched posterior draws.
| Model specification | Mean difference non-negative | Credibly positive |
|---|---|---|
| Reported model (full specificity effects) | February 2026 | April 2026 |
| Full specificity effects, Youden constraint | October 2025 | February 2026 |
| Full specificity effects, with interactions | December 2025 | May 2026 |
| Reduced specificity effects | October 2025 | January 2026 |
| Reduced specificity effects, | October 2025 | February 2026 |
| Reduced specificity effects, | September 2025 | December 2025 |
Table D.9 : When bilingual prevalence first exceeds no-English prevalence under each specification (settings in Table D.6 ): the first crawl at which the posterior mean difference is non-negative, and the first crawl at which the difference is positive with posterior probability above 0.975.
Figure D.2 : Prior and posterior of the shifts in sensitivity and specificity after the control crawls (logit scale) in the reported model. The grey bar is the central 95% prior interval, and the blue point and bar are the posterior mean and 95% credible interval. The right column gives the posterior mean in units of the prior standard deviation.
Figure D.3 : False-positive rate implied by the control-period specificity, by centrality tier, word-count level and TLD category, for Pangram (top) and Binoculars (bottom). Dots are the posterior mean and 95% credible interval of the average over the English-only pages of the control crawls in each category, and diamonds are the observed flag rates of those pages.
| Specification | Draws | ESS bulk | ESS tail | , Pangram, English | , Pangram, non-English | , Binoculars , English | |
|---|---|---|---|---|---|---|---|
| Reported model (full specificity effects) | 8,000 | 1.005 | 880 | 736 | 0.58 | 0.37 | 0.06 |
| (0.57, 0.59) | (0.35, 0.39) | ( 0.06, 0.05) | |||||
| Full specificity effects, Youden constraint | 8,000 | 1.004 | 813 | 1,399 | 0.58 | 0.36 | 0.00 |
| (0.57, 0.59) | (0.34, 0.37) | (0.00, 0.00) | |||||
| Full specificity effects, with interactions | 8,000 | 1.007 | 484 | 215 | 0.57 | 0.35 | 0.06 |
| (0.56, 0.58) | (0.33, 0.37) | ( 0.06, 0.05) |
Table D.10 : Convergence diagnostics and Youden index by fit. Draws are the posterior draws over four chains. is the largest rank-normalized split- , and ESS the smallest bulk and tail effective sample size, over all parameters and derived quantities. is the posterior mean (95% credible interval) of the Youden index of the control-period calibration in the least specific observed cell of the stratum, for Pangram in english-stratum and non-English pages and for Binoculars in English-stratum pages (the constraint binds for Binoculars in the fit with a positive Youden index). No fit has divergent transitions, transitions at the maximum tree depth or a low energy Bayesian fraction of missing information.