cs.CLFeb 23, 2026
SaveBeyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
Organizations: University of Washington · Apple · Stanford
Abstract
One of the first pre-processing steps for constructing web-scale LLM pretraining datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple intervention: by taking a Union over different extractors, we can increase the token yield of DCLM-Baseline by up to 71% while maintaining benchmark performance. We further show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance, with differences of up to 10 percentage points (p.p.) on WikiTQ and 3 p.p. on HumanEval.
Figures & tables
Figure 1: Different extractors lead to different final pages. The Venn diagram shows the overlap in resulting pages that come from applying the DCLM-Baseline pipeline to the outputs of three different initial extractors. 61% of pages are uniquely kept for just one extractor.
| Training Scale | Extractor | fastText Thresholds | Token Yield | Core_V2 | MMLU |
| 1B-1x (29B Tokens) | resiliparse | 0.11 | 39B | 28.5 | 24.7 |
| trafilatura | 0.11 | 27B | 29.6 | 25.7 | |
| jusText | 0.11 | 25B | 29.3 | 25.8 | |
| Union (Random) | (0.11, 0.11, 0.11) | 55B | 28.1 | 25.1 | |
| Union (Manual) | (0.11, 0.11, 0.11) | 57B | 29.2 | 24.1 | |
| 7B-1x (138B Tokens) | resiliparse | 0.11 | 193B | 42.6 | 39.5 |
Table 1: Using multiple extractors increases token yield while maintaining performance. We compare the token yields and Core_V2 performance of using different extractors in the DCLM-Baseline curation pipeline. “Union” refers to combining the resulting pages from the individual extractors. The “Random” and “Manual” variants refer to whether the pages in the intersection are selected randomly or according to a manually selected preference ordering: i.e., resiliparse , then trafilatura , then jusText . This is also the ordering of the percentiles for Union methods in the “ fastText Threshold” column. Note that while we show MMLU performance at all scales for completeness, we advise caution when drawing conclusions from it at 1B-1x and 7B-1x scales due to potential noise.
Figure 2: (Left) Analysis of extractor imbalance across domains. We group pages from all three extractors (Top 11%) by domain. For every domain with at least 50 pages, we compute the maximum ratio represented by any one extractor, plotting the distribution above. For 26.7% and 7.6% of domains respectively, at least 60% and 80% of surviving pages come from just one extractor. (Right) Does higher yield lead to better performance when data-constrained? We train 1B-5x models on smaller subsamples of data curated from the 1B-1x raw pool. The Union datasets that yield more tokens are able to achieve better performance. See Table 7 for extended results.
| Scale | Union Type | Token Yield | Core_V2 |
| 1B-1x | Random | 55B | 28.1 |
| Manual | 57B | 29.2 | |
| Random w/ Rededup | 51B | 28.2 | |
| Manual w/ Rededup | 53B | 27.4 | |
| 7B-1x | Random | 273B | 41.3 |
| Manual | 283B | 43.1 |
Table 2: Effect of reduplicating Union datasets. We compare the token yields and Core_V2 performance of our Union datasets with and without reduplication. “Rededup” refers to running another round of deduplication on the Union dataset (now across the pages from different extractors). Here we always take a Union over the (0.11, 0.11, 0.11) thresholds. All other nomenclature follows from Table 1 .
| CC-Tables Extraction | Model Scale | WikiTQ-Avg. |
| No CC-Tables | 7B-2x | 1.1 |
| resiliparse | 7B-1x | 11.9 |
| trafilatura | 7B-1x | 3.7 |
| jusText | 7B-1x | 1.6 |
Table 3: resiliparse is the most effective extractor for CC-Tables pages. We compare the performance of resiliparse , trafilatura , and jusText on pages from our CC-Tables dataset. For each extracted version of CC-Tables, we mix it with DCLM-Baseline in a 20%-80% mix, train models, and evaluate on WikiTableQuestions (WikiTQ). “No CC-Tables” refers to training on only DCLM-Baseline .
Figure 3: Extractor performance remains consistent across serialization formats. We plot the WikiTQ performance stratified across different test-time table serializations. Despite resiliparse and trafilatura producing tables that most closely match “concat” and “markdown” formats respectively, we observe a surprising degree of generalization across most serializations (with the exception of “json”).
| Model / Cooldown Dataset | Model Size | Tokens | WikiTQ-Avg. | Core_V2 | MMLU |
| LLama-3-8B | 8B | 15T | 29.3 | 56.5 | 66.2 |
| DCLM-7B-8k | 7B | 2.8T | 5.0 | 56.0 | 63.7 |
| DCLM-Bas. + CC-Tables ( = 1.00) | 7B | 2.7T | 22.7 | 53.7 | 62.3 |
| DCLM-Bas. + CC-Tables ( = 0.90) | 7B | 2.7T | 22.8 | 54.7 | 62.6 |
| DCLM-Bas. + CC-Tables ( = 0.85) | 7B | 2.7T | 22.4 | 55.7 | 62.7 |
| DCLM-Bas. + CC-Tables ( = 0.80) | 7B | 2.7T | 18.8 | 56.1 | 63.0 |
Table 4: CC-Tables- resiliparse closes 72% of the gap between the original DCLM-7B-8k model and Llama-3-8B. The top rows show results for existing models while the bottom rows are new models obtained by replacing the original DD-style cooldown for DCLM-7B-8K with one that mixes CC-Tables- resiliparse and DCLM-Baseline and performs WiSE-FT ( Wortsman et al., 2022 ) . We try different interpolation weights with providing a good balance between tabular and standard evaluations.
| Dataset | Python | Java | JS | C++ | Go | HE-Avg. | Core_V2 | MMLU |
| Alternative 200B token cooldown for DCLM-7B | ||||||||
| DCLM-Baseline (Top-7%) + Stack v2 + CC-Code- res. | 26.7 | 29.9 | 24.0 | 26.6 | 17.7 | 25.0 | 54.7 | 60.8 |
| DCLM-Baseline (Top-7%) + Stack v2 + CC-Code- traf. | 28.0 | 27.0 | 25.3 | 23.0 | 16.9 | 24.0 | 54.3 | 60.5 |
| DCLM-Baseline (Top-7%) + Stack v2 + CC-Code- jusT. | 25.0 | 24.1 | 21.7 | 20.3 | 15.8 | 21.4 | 54.0 | 58.9 |
| DCLM-Baseline (Top-7%) + Stack v2 | 28.1 | 23.6 | 24.6 | 20.9 | 15.2 | 22.5 | 55.4 | 62.4 |
| Trained from scratch, DCLM 7B-2x scale | ||||||||
Table 5: CC-Code extraction and code generation performance. We observe that using jusText leads to the additional CC-Code pages harming HumanEval performance. Meanwhile, resiliparse and trafilatura perform comparably. In each mix we train on 25% StackV2, 25% CC-Code (when present), and fill in the remainder with DCLM-Baseline . The top rows are for cooldown runs on top of the DCLM-7B ( Li et al., 2024 ) checkpoint prior to their original cooldown at 2T tokens. The bottom rows correspond to training 7B-2x models from scratch. All code generation metrics are reported as Pass@1 computed from 20 samples per problem.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Extractor | Run 1 | Run 2 | Run 3 | Mean (Std) |
| resiliparse | 28.5 | 27.5 | 28.8 | 28.3 (0.7) |
| trafilatura | 29.6 | 27.7 | 28.5 | 28.6 (1.0) |
Table 6: Run-to-run variation at 1B-1x scale. We report mean Core_V2 performance and standard deviation across three runs for resiliparse and trafilatura . Note that results in Table 1 were all single-runs since we did not try multiple seeds for all methods.
| Initial Pool | Extractor | fastText Thresh. | Repeats | Core_V2 |
| 1B-0.25x | resiliparse | 0.11 | 15.0x | 30.8 |
| Union (Random) | (0.11, 0.11, 0.11) | 10.5x | 32.3 | |
| Union (Manual) | (0.11, 0.11, 0.11) | 10.2x | 32.4 | |
| Union (Random) | (0.11, 0.15, 0.15) | 9.0x | 33.5 | |
| Union (Manual) | (0.11, 0.15, 0.15) | 8.7x | 33.5 | |
| 1B-0.33x | resiliparse | 0.11 | 11.2x | 33.1 |
Table 7: Using multiple extractors improves performance in settings which are data-constrained. We train 1B-5x models on progressively smaller subsamples of data curated from the 1B-1x raw pool. More heavy subsampling leads to more repetition and thus worse performance, but this can be mitigated by having a curation pipeline that yields more tokens (e.g., our Union datasets). A subset of these results are plotted in Figure 2 .
Figure 4: Extraction comparison for mutual fund data. We use difflib to visualize pairwise comparisons between resiliparse (left) and jusText (top) or trafilatura (bottom). For both tables in this page, jusText removes them while trafilatura applies markdown formatting.
Figure 5: Extraction comparison for an BioMed Central article. We use difflib to visualize pairwise comparisons between resiliparse (left) and jusText (top) or trafilatura (bottom). Note that for the table shown, jusText removes it while trafilatura applies markdown formatting. Here, resiliparse ends up splitting entries across line breaks instead of single spaces.
Figure 6: Extraction comparison for soccer recruiting statistics. We use difflib to visualize pairwise comparisons between resiliparse (left) and jusText (top) or trafilatura (bottom). Note that for both tables in this page, jusText removes them while trafilatura applies markdown formatting.
Figure 7: Extraction comparison for a Pokémon learnset table on Bulbapedia. We use difflib to visualize pairwise comparisons between resiliparse and trafilatura . Note that jusText removes the tables altogether so we do not show it. Meanwhile, resiliparse fails to cleanly split some columns while trafilatura fails to keep the actual contents.
| CC-Tables Extraction | DCLM-Baseline Extraction | CC-Tables Mixing Ratio | WikiTQ-Avg. |
| resiliparse | resiliparse | 0.1 | 10.1 |
| trafilatura | resiliparse | 0.1 | 2.2 |
| jusText | resiliparse | 0.1 | 1.9 |
| resiliparse | resiliparse | 0.2 | 11.9 |
| trafilatura | resiliparse | 0.2 | 3.7 |
| jusText | resiliparse | 0.2 | 1.6 |
Table 8: Conclusions about extractors hold regardless of mixing ratio or whether the DCLM-Baseline and CC-Tables extractors are consistent ( 7B-1x scale). We compare the performance of resiliparse , trafilatura , and jusText on pages from our CC-Tables dataset when it is mixed with different ratios and different extractions of DCLM-Baseline (bottom section). Overall, we observe that there is a slight increase to WikiTQ performance for resiliparse as the ratio of CC-Tables is larger but that conclusions between the extractors are the same across all settings.
Figure 8: Extraction comparison for a C# tutorial page. We use difflib to visualize pairwise comparisons between resiliparse (left) and jusText (top) or trafilatura (bottom). Note that for code blocks in this page, jusText removes them while trafilatura collapses whitespace formatting.
Figure 9: Extraction comparison for a Ruby Plasma tutorial. We use difflib to visualize pairwise comparisons between resiliparse (left) and jusText (top) or trafilatura (bottom). Note that for code blocks in this page, jusText removes them while trafilatura collapses whitespace formatting.
Figure 10: Extraction comparison for release note blog post. We use difflib to visualize pairwise comparisons between resiliparse (left) and jusText (top) or resiliparse (bottom). Note that for code blocks in this page, jusText removes them while trafilatura collapses whitespace formatting.