Data Darwinism (Part I) established a ten-level hierarchy for data processing, showing that stronger processing can unlock greater data value. However, that work relied on manually designed strategies for a single category. Modern pretraining corpora comprise hundreds of heterogeneous categories spanning domains and content types, each demanding specialized treatment. At this scale, manual strategy design becomes prohibitive. This raises a key question: can strategies evolve in an automated way? We introduce DataEvolve, a framework that enables strategies to evolve through iterative optimization rather than manual design. For each data category, DataEvolve operates in a closed evolutionary loop: it identifies quality issues, generates candidate strategies, executes them on sampled data, evaluates results, and refines approaches across generations. The process accumulates knowledge through an experience pool of discovered issues and a strategy pool tracking performance across iterations. Applied to 8 categories spanning 672B tokens from Nemotron-CC, DataEvolve produces Darwin-CC, a 504B-token dataset with strategies evolved through 30 iterations per category. Training 3B models on 500B tokens, Darwin-CC outperforms raw data (+3.96 points) and achieves a 44.13 average score across 18 benchmarks, surpassing DCLM, Ultra-FineWeb, and FineWeb-Edu, with strong gains on knowledge-intensive tasks such as MMLU. Analysis shows evolved strategies converge on cleaning-focused approaches: targeted noise removal and format normalization with domain-aware preservation, echoing the L4 (Generative Refinement) principles from Part I. Ablation studies confirm iterative evolution is essential: optimized strategies outperform suboptimal ones by 2.93 points, establishing evolutionary strategy design as feasible and necessary for pretraining-scale data curation.
Figures & tables
Figure 1: Performance comparison across pretraining datasets on 18 selected benchmarks.
Figure 2: Naively evaluating candidate curation strategies requires cleaning data at full scale and training a model to convergence for each candidate strategy—demanding thousands of GPU hours per evaluation. Across n candidate strategies and m categories, the total cost becomes computationally intractable.
Figure 3: Overview of the DataEvolve framework. The system enables strategies to evolve through an iterative feedback loop involving four core components: (1) the data observer identifies category-specific quality issues; (2) the strategy designer generates and refines cleaning strategies; (3) the data cleaner executes strategies on sample data; and (4) the quality judge provides scoring and diagnostic feedback. Discovered issues and evolved strategies are archived in the experience pool and strategy pool , enabling cross-generation knowledge transfer to guide evolutionary progression.
Content Type
Quality
Domain
Raw
Cleaned
Academic
High
Mathematics
7B
5B
Computer Science
32B
24B
Medicine
52B
29B
Other STEM
71B
51B
Not-High
Mathematics
6B
6B
Computer Science
107B
78B
Table 1: Category composition of Darwin-CC
Metrics
Raw
Sub-opt
Best
BBH
26.82
26.69 -0.12
26.16 -0.65
ARC-E
74.94
77.55 +2.61
78.59 +3.65
ARC-C
43.52
48.41 +4.90
49.32 +5.80
MMLU
27.49
32.55 +5.05
46.13 +18.64
AGIEval
18.15
18.30 +0.15
18.21 +0.07
HellaSwag
65.32
64.36 -0.96
62.21 -3.11
Table 2: Performance comparison across curation strategies. Models trained on raw data (Raw), data with suboptimal strategy (Sub-opt), data cleaned with DataEvolve ’s best strategy (Best), which generates Darwin-CC . All models are 3B parameters trained on 500B tokens.
Figure 6
Metrics
Fineweb-Edu
Ultra-Fineweb
DCLM
Darwin-CC
BBH
3.01
7.42
24.16
26.16
ARC-E
73.39
73.96
75.13
78.59
ARC-C
43.45
43.77
45.02
49.32
MMLU
28.38
25.53
28.54
46.13
AGIEval
16.96
17.72
17.90
18.21
HellaSwag
64.33
65.32
70.39
62.21
Table 3: Comparison of models trained on different pretraining corpora. All models are 3B parameters trained on 500B tokens.
Metric
Raw
Cleaned
Change
Self-ROUGE-2 ↓
0.0148
0.0116
-21.7%
L2 Distance ↑
1.376
1.374
-0.15%
Shannon Entropy ↑
10.84
10.87
+0.3%
Table 4: Diversity metrics for raw and cleaned datasets.
Figure 6: Representative benchmarks where cleaned data substantially outperforms raw data throughout training. The gap between cleaned (blue) and raw (red) widens consistently as training progresses.
Figure 7: Representative benchmarks where raw data outperforms cleaned data. The gap between raw (red) and cleaned (blue) is established early and remains stable throughout training.
Figure 8: Representative benchmarks where cleaned and raw data yield comparable performance, with the two curves interleaving throughout training without a consistent directional gap.
Figure 9: Semantic similarity distribution before and after processing. DataEvolve ’s cleaning strategies (Raw1-Cleaned) preserve substantially more semantic content than transformation-based methods (Raw1-Wiki, Raw1-QA), with similarity distributions approaching the identity baseline. The tight, right-skewed distribution indicates minimal content modification during cleaning.
High-quality training data is essential to large language models (LLMs) and typically requires extensive and costly manual curation. Automatic data preparation systems support reusable operators and task-driven workflows, yet adapting these workflows to the desired data quality remains challenging. High-quality examples provide a practical specification for guiding this adaptation. We introduce DataEvolver, a data preparation system that automatically constructs pipelines through multi-level self-evolution. At the operator level, it expands the operator library and refines logical plans to resolve dependency conflicts. At the pipeline level, it instantiates these plans into code and uses trial-output feedback to refine the pipeline toward seed-specified quality requirements. Experiments on seven benchmarks show that DataEvolver substantially improves data quality and achieves an average gain of approximately 10 score points in downstream LLM performance compared with training on original data, highlighting new opportunities for the iterative co-evolution of LLMs and data.
Domain adaptation of large language models increasingly depends on constructing high-quality training data, yet existing data-preparation pipelines typically address quality only after generation through post-hoc filtering. This creates a fundamental mismatch: data-quality issues often originate from the construction process itself, while quality control is applied only to its outputs. We introduce DataFoundry, a framework for evolving data preparators through recursive self-improvement before large-scale data production. DataFoundry represents a data preparator as an evolvable runtime specification and instantiates its evolution with a Skills-as-Modules architecture, in which a central Controller orchestrates modular skills to compile executable runtimes, diagnose deficiencies on small pilot sets using domain-appropriate criteria, and translate diagnostic feedback into adapters that revise individual preparation components while preserving stable interfaces. We evaluate DataFoundry on DataPrep-Bench across mathematics, finance, law, and medicine, and find that recursively evolved preparators produce training data with higher downstream utility than baselines. Experiments across different backbones further demonstrate that these improvements are not tied to a particular model, while analyses and case studies further reveal the framework's optimization dynamics and illustrate how its evolution unfolds in practice.
Cehao Yang, Xiaojun Wu, Xueyuan Lin +4
1IDEA Research, International Digital Economy Academy · 2Hong Kong University of Science and Technology (Guangzhou) · 3DataArc Tech Ltd.
Large language model (LLM) agents require post-training methods that can improve long-horizon decision making from environment feedback. However, existing agentic post-training pipelines often treat data curation as a fixed preprocessing step, focusing mainly on data augmentation while neglecting filtering, refinement, and adaptation to downstream failures. We propose CurateEvo, a failure-driven dynamic evolution framework for agentic post-training data curation. CurateEvo represents the curation strategy as executable code and iteratively rewrites it using failed trajectories from a held-out development set. At each epoch, the evolved strategy transforms a fixed raw corpus into supervised fine-tuning data, reinforcement learning data, and an inference-time memory bank. The evolution process first improves effectiveness by diagnosing recurring failure modes and augmenting, filtering, or refining data accordingly, and then improves efficiency by pruning redundant or low-utility training turns under a cost-aware objective. Experiments on ACEBench-Agent, BFCL-V4, and τ^2-Bench under both labeled and wild-data settings show that CurateEvo consistently outperforms prior curation methods, improving average scores by 3.2 and 2.7 points, respectively. Further analyses demonstrate that CurateEvo is compatible with different post-training recipes and substantially reduces curation overhead.