The foundational pretraining phase determines a model's capability ceiling, as post-training struggles to overcome capability foundations established during pretraining, yet it remains critically under-explored. This stems from a structural paradox: organizations with computational resources operate under commercial pressures that inhibit transparent disclosure, while academic institutions possess research freedom but lack pretraining-scale computational resources. daVinci-LLM occupies this unexplored intersection, combining industrial-scale resources with full research freedom to advance the science of pretraining. We adopt a fully-open paradigm that treats openness as scientific methodology, releasing complete data processing pipelines, full training processes, and systematic exploration results. Recognizing that the field lacks systematic methodology for data processing, we employ the Data Darwinism framework, a principled L0-L9 taxonomy from filtering to synthesis. We train a 3B-parameter model from random initialization across 8T tokens using a two-stage adaptive curriculum that progressively shifts from foundational capabilities to reasoning-intensive enhancement. Through 200+ controlled ablations, we establish that: processing depth systematically enhances capabilities, establishing it as a critical dimension alongside volume scaling; different domains exhibit distinct saturation dynamics, necessitating adaptive strategies from proportion adjustments to format shifts; compositional balance enables targeted intensification while preventing performance collapse; how evaluation protocol choices shape our understanding of pretraining progress. By releasing the complete exploration process, we enable the community to build upon our findings and systematic methodologies to form accumulative scientific knowledge in pretraining.
Figures & tables
Figure 1: Performance comparison of daVinci-LLM-3B against baseline models with score across three capability domains, and overall score comparable to OLMo-3-7B.
Figure 2: Evolution of pretraining research depth across institutional structures. The y-axis represents research depth from surface-level artifacts (API-only access) to fundamental scientific questions. The x-axis shows the temporal progression from 2022 to 2026. Commercial entities (blue) possess computational resources but remain constrained to API-level access due to competitive pressures. Open-weight releases (green, e.g., Llama, Qwen) provide model artifacts but withhold design rationale and negative results. Academic efforts (orange, e.g., OLMo) achieve transparency and research freedom but face severe scale limitations—making systematic exploration with 200+ configurations structurally infeasible. The top tier remains largely unexplored, as it requires the rare alignment of large-scale computational resources with the research freedom to publish comprehensive findings. daVinci-LLM (purple) occupies this intersection, conducting the extensive ablations and systematic disclosures necessary to advance the science of pretraining. By releasing the complete decision-making logic alongside the model weights, we bridge the structural gap between industrial scale and scientific transparency.
Dimension
Llama 3
Qwen 3
YuLan
OLMo 3
daVinci
Model Artifacts
Model Weights
Training Code
Training Logs
Intermediate Checkpoints
Data Openness
Data Composition
Processing Pipeline
Table 1: Transparency comparison across state-of-the-art LLMs. Unlike existing models, daVinci provides the complete scientific process, enabling the systematic investigation of pretraining dynamics. Symbols: fully open, patial disclosure or not released, not disclosed.
Figure 3: Mapping of our pretraining data sources onto the Data Darwinism L0–L9 taxonomy across different training stage.
Data Source
Pool Size
Level
Stage 1
Stage 2
Stage 1-1
Stage 1-2
Stage 2-1
Stage 2-2
General
4.28T
2.73T (68.2%)
1.11T (55.42%)
100B (10%)
188.4B (18.84%)
Nemotron-CC-v1
4.28T
L3
✓
✓
✓
✓
Code
598B
381B (9.53%)
233B (11.66%)
300 (30%)
26.1B (2.61%)
Self-Crawled GitHub
187B
L3
✓
✓
✓
✓
Nemotron-Pretraining-Code-v1-non-synthetic
220B
L3
✓
✓
✓
✓
Table 2: Data pool composition and token allocation across training stages. Shaded category rows show each category’s total pool size and the tokens allocated per stage; data source rows indicate whether the source is used (✓) or not used (–) in each stage.
Data Source
General
Code
Science
Nemotron-CC-v1 Synthetic QA (492B)
Nemotron-Pretraining-SFT-v1-General (12B)
Nemotron-Pretraining-SFT-v1-Code (21B)
Llama-Nemotron-Post-Training-Code (5B)
Nemotron-Pretraining-SFT-v1-Math (138B)
Llama-Nemotron-Post-Training-Math-RS (10B)
Table 3: Domain coverage of QA data sources. Each source is categorized into one of three domains: General, Code, and Science (including Mathematics).
Parameters
∼ 3.09B
MLP intermediate size
11008
Layers)
36
Activation function
SwiGLU
Hidden size
2048
Positional encoding
RoPE (base θ=10000 )
Attention heads
16
Max position embeddings
4096
KV heads (GQA)
2
Normalization
RMSNorm ( ϵ=10−6 )
Head dimension
128
Tokenizer
Qwen2 tokenizer
Vocab size
151936
Precision
bfloat16
Table 4: Architectural specification of daVinci-LLM .
Stage 1
Stage 2
Stage 1-1
Stage 1-2
Stage 2-1
Stage 2-2
Training Tokens
4 Trillion
2 Trillion
1 Trillion
1 Trillion
Global Batch Size
1024 → 2048 → 4096
4096
4096
4096
Rotatory Base
10000
10000
10000
10000
Sequence Length
4096
4096
4096
4096
Precision
bfloat16
bfloat16
bfloat16
bfloat16
Table 5: Training hyperparameters of different training stages. Note that the Global Batch Size refers to the total number of sequences per training step; the total number of tokens per step is calculated as Global Batch Size×Sequence Length .
Figure 4: Progressive training results across Stage 1-1 and Stage 1-2, with checkpoints evaluated every 5000 steps. The vertical dashed line indicate the boundary between two substages.
Figure 5: Stage 1 Training dynamics for the first 300k steps: (a) training loss curve demonstrating consistent convergence, and (b) gradient norm curve tracking optimization stability across the initial training phase.
Figure 6: Progressive training results across Stage 2-1 and Stage 2-2, with checkpoints evaluated every 5000 steps. The vertical dashed line indicate the boundary between two substages.
Fully-open Models
Open-weight Models
Domain
Benchmark
daVinci 3B
OLMO-3 7B
OLMO-2 7B
Yulan 2.4B
LLaMa-3.2 3B
Qwen-2.5 3B
Qwen-3 4B
Qwen-3.5 4B
General
MMLU
62.53
66.53
65.93
50.70
54.91
65.73
75.35
72.75
MMLU-Pro
43.50
35.70
28.40
23.90
24.50
39.00
54.10
53.50
AGIEval
26.77
33.75
31.78
28.22
22.72
37.15
45.87
44.89
HellaSwag
71.17
74.15
80.50
68.56
73.60
73.60
73.75
75.29
TriviaQA
49.90
55.45
68.01
27.64
55.22
51.20
47.44
49.80
Table 6: Comprehensive evaluation across diverse capability benchmarks. daVinci-LLM-3B is compared against state-of-the-art open-weight models.
Configuration
HumanEval
EvalPlus
MBPP
Avg Gen.
Avg Code
Avg Sci.
Overall
Rule-Based Filtering (L2)
54.43
48.58
42.40
51.12
48.47
40.14
46.66
Model-Based Filtering (L3)
54.70
48.39
45.80
51.19
49.63
40.28
46.93
Δ (L3 – L2)
+0.27
-0.19
+3.40
+0.07
+1.16
+0.14
+0.27
Table 7: Comparison of rule-based and model-based filtering approaches.
Configuration
GSM-8K
GSM-Plus
MATH
Avg Gen.
Avg Code
Avg Sci.
Overall
Baseline
64.06
40.58
38.00
51.69
49.80
40.52
47.27
Generative Refined (L4)
65.43
42.38
45.00
51.40
50.15
42.25
47.83
Δ (L4 – Baseline)
+1.37
+1.80
+7.00
-0.29
+0.35
+1.73
+0.56
Table 8: Comparison of baseline and generative refinement approaches.
Figure 7: Effectiveness of CC-QA and CodeQA synthetic data in Stage 2-2.
Figure 8: Stage 1 benchmark performance. Light curves show individual benchmarks; dark curves show category averages for general knowledge, code, and science. Two vertical dashed lines indicate the boundaries between substages 1-1, 1-2, and 1-3, respectively.
Figure 9: Performance comparison between continued data domain adjustment (stage 1-3) and QA introduction (Stage 2).
Figure 10: Stage 2 benchmark performance. Light curves show individual benchmarks; dark curves show category averages for general knowledge, code, and science. The vertical dashed lines indicate the boundaries between Substages 2-1, 2-2.
Figure 11: Code and science data ratio analysis. Notation: C-X-S-Y indicates X% code and Y% science. (a) Performance across code and science proportions (QA fixed at 30%, remainder allocated to CommonCrawl web data). (b) Data synergy effects across QA ratios in Stage 2-1.
Configuration
Training Tokens
General
Code
Science
Overall
QA-30%
84B
51.59
47.71
40.25
46.80
252B
51.66
46.79
40.03
46.61
419B
51.80
47.16
40.59
46.94
QA-50%
84B
51.72
49.86
41.60
47.70
252B
51.85
49.01
42.80
48.06
419B
51.83
49.89
43.42
48.43
Table 9: Impact of QA data ratio on model performance in Stage 2-2.
Figure 12: MMLU performance comparison across evaluation protocols. Panel (a) shows absolute scores under PPL-based and generative (CoT) evaluation. Panel (b) highlights the performance gap relative to Qwen-2.5-3B, demonstrating how models with extensive QA training exhibit amplified advantages in generative settings. The gap between OLMO-2-7B and Qwen-2.5-3B increases from -0.20% (PPL) to +2.90% (generative), a 3.10% swing that reverses the ranking.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Avg General
Avg Code
Avg Science
Overall Avg
Constant LR
52.06
52.32
45.44
49.66
Cosine decay
52.49
53.31
45.39
50.00
Δ (decay - constant)
+0.43
+0.99
-0.05
+0.34
Appendix
Table 10: Impact of learning rate decay strategy on Stage 2-2 performance. Constant LR maintains 3e-5 throughout, while cosine decay gradually reduces LR from 3e-5 to 3e-6.
Figure 13: Impact of QA masking policy on three capability dimensions across training steps in Stage 2-2. The comparison between masked and unmasked configurations shows marginal improvements, with consistent but modest gains across all capability clusters.
Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.
Mohnish Harwani, Yujia Zheng
Purdue University · University of Illinois Urbana–Champaign
Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent (ρ=.81 across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining (R2=.68-.84 across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.
Emmy Liu, Kaiser Sun, Millicent Li +4
Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University +1
Large language models provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered. Although progress is usually attributed to scale, data and architecture, we show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity. Reducing the initialization scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks. We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling. We further uncover a critical initialization that balances the reasoning and training. Mechanistically, small initialization drives a distinct developmental trajectory: parameters first condense into low-complexity structures and later expand into richer representations, giving concrete form to the idea that compression is intelligence. Token-level analyses show that the gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. These results motivate a simple γ-initialization rule: expose initialization rage as an explicit knob and use small initialization by default, an almost cost-free intervention that improves pretraining and strengthens reasoning across model scales.
Liangkai Hang, Junjie Yao, Zhiyu Li +3
School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China. · MemTensor (Shanghai) Technology Co., Ltd.. for Advanced Algorithms Research, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China.