Data Darwinism (Part I) established a ten-level hierarchy for data processing, showing that stronger processing can unlock greater data value. However, that work relied on manually designed strategies for a single category. Modern pretraining corpora comprise hundreds of heterogeneous categories spanning domains and content types, each demanding specialized treatment. At this scale, manual strategy design becomes prohibitive. This raises a key question: can strategies evolve in an automated way? We introduce DataEvolve, a framework that enables strategies to evolve through iterative optimization rather than manual design. For each data category, DataEvolve operates in a closed evolutionary loop: it identifies quality issues, generates candidate strategies, executes them on sampled data, evaluates results, and refines approaches across generations. The process accumulates knowledge through an experience pool of discovered issues and a strategy pool tracking performance across iterations. Applied to 8 categories spanning 672B tokens from Nemotron-CC, DataEvolve produces Darwin-CC, a 504B-token dataset with strategies evolved through 30 iterations per category. Training 3B models on 500B tokens, Darwin-CC outperforms raw data (+3.96 points) and achieves a 44.13 average score across 18 benchmarks, surpassing DCLM, Ultra-FineWeb, and FineWeb-Edu, with strong gains on knowledge-intensive tasks such as MMLU. Analysis shows evolved strategies converge on cleaning-focused approaches: targeted noise removal and format normalization with domain-aware preservation, echoing the L4 (Generative Refinement) principles from Part I. Ablation studies confirm iterative evolution is essential: optimized strategies outperform suboptimal ones by 2.93 points, establishing evolutionary strategy design as feasible and necessary for pretraining-scale data curation.
Figures & tables
Figure 1: Performance comparison across pretraining datasets on 18 selected benchmarks.
Figure 2: Naively evaluating candidate curation strategies requires cleaning data at full scale and training a model to convergence for each candidate strategy—demanding thousands of GPU hours per evaluation. Across n candidate strategies and m categories, the total cost becomes computationally intractable.
Figure 3: Overview of the DataEvolve framework. The system enables strategies to evolve through an iterative feedback loop involving four core components: (1) the data observer identifies category-specific quality issues; (2) the strategy designer generates and refines cleaning strategies; (3) the data cleaner executes strategies on sample data; and (4) the quality judge provides scoring and diagnostic feedback. Discovered issues and evolved strategies are archived in the experience pool and strategy pool , enabling cross-generation knowledge transfer to guide evolutionary progression.
Content Type
Quality
Domain
Raw
Cleaned
Academic
High
Mathematics
7B
5B
Computer Science
32B
24B
Medicine
52B
29B
Other STEM
71B
51B
Not-High
Mathematics
6B
6B
Computer Science
107B
78B
Table 1: Category composition of Darwin-CC
Metrics
Raw
Sub-opt
Best
BBH
26.82
26.69 -0.12
26.16 -0.65
ARC-E
74.94
77.55 +2.61
78.59 +3.65
ARC-C
43.52
48.41 +4.90
49.32 +5.80
MMLU
27.49
32.55 +5.05
46.13 +18.64
AGIEval
18.15
18.30 +0.15
18.21 +0.07
HellaSwag
65.32
64.36 -0.96
62.21 -3.11
Table 2: Performance comparison across curation strategies. Models trained on raw data (Raw), data with suboptimal strategy (Sub-opt), data cleaned with DataEvolve ’s best strategy (Best), which generates Darwin-CC . All models are 3B parameters trained on 500B tokens.
Figure 6
Metrics
Fineweb-Edu
Ultra-Fineweb
DCLM
Darwin-CC
BBH
3.01
7.42
24.16
26.16
ARC-E
73.39
73.96
75.13
78.59
ARC-C
43.45
43.77
45.02
49.32
MMLU
28.38
25.53
28.54
46.13
AGIEval
16.96
17.72
17.90
18.21
HellaSwag
64.33
65.32
70.39
62.21
Table 3: Comparison of models trained on different pretraining corpora. All models are 3B parameters trained on 500B tokens.
Metric
Raw
Cleaned
Change
Self-ROUGE-2 ↓
0.0148
0.0116
-21.7%
L2 Distance ↑
1.376
1.374
-0.15%
Shannon Entropy ↑
10.84
10.87
+0.3%
Table 4: Diversity metrics for raw and cleaned datasets.
Figure 6: Representative benchmarks where cleaned data substantially outperforms raw data throughout training. The gap between cleaned (blue) and raw (red) widens consistently as training progresses.
Figure 7: Representative benchmarks where raw data outperforms cleaned data. The gap between raw (red) and cleaned (blue) is established early and remains stable throughout training.
Figure 8: Representative benchmarks where cleaned and raw data yield comparable performance, with the two curves interleaving throughout training without a consistent directional gap.
Figure 9: Semantic similarity distribution before and after processing. DataEvolve ’s cleaning strategies (Raw1-Cleaned) preserve substantially more semantic content than transformation-based methods (Raw1-Wiki, Raw1-QA), with similarity distributions approaching the identity baseline. The tight, right-skewed distribution indicates minimal content modification during cleaning.