daVinci-LLM:Towards the Science of Pretraining

Yiwei Qin; Yixiu Liu; Tiantian Mi; Muhang Xie; Zhen Huang; Weiye Si; Pengrui Lu; Siyuan Feng

cs.AIMar 28, 2026

daVinci-LLM:Towards the Science of Pretraining

Authors: Yiwei Qin, Yixiu Liu, Tiantian Mi, Muhang Xie, Zhen Huang, Weiye Si, Pengrui Lu, Siyuan Feng,

Organizations: SII · SJTU · GAIR

Abstract

The foundational pretraining phase determines a model's capability ceiling, as post-training struggles to overcome capability foundations established during pretraining, yet it remains critically under-explored. This stems from a structural paradox: organizations with computational resources operate under commercial pressures that inhibit transparent disclosure, while academic institutions possess research freedom but lack pretraining-scale computational resources. daVinci-LLM occupies this unexplored intersection, combining industrial-scale resources with full research freedom to advance the science of pretraining. We adopt a fully-open paradigm that treats openness as scientific methodology, releasing complete data processing pipelines, full training processes, and systematic exploration results. Recognizing that the field lacks systematic methodology for data processing, we employ the Data Darwinism framework, a principled L0-L9 taxonomy from filtering to synthesis. We train a 3B-parameter model from random initialization across 8T tokens using a two-stage adaptive curriculum that progressively shifts from foundational capabilities to reasoning-intensive enhancement. Through 200+ controlled ablations, we establish that: processing depth systematically enhances capabilities, establishing it as a critical dimension alongside volume scaling; different domains exhibit distinct saturation dynamics, necessitating adaptive strategies from proportion adjustments to format shifts; compositional balance enables targeted intensification while preventing performance collapse; how evaluation protocol choices shape our understanding of pretraining progress. By releasing the complete exploration process, we enable the community to build upon our findings and systematic methodologies to form accumulative scientific knowledge in pretraining.

Figures & tables

Figure 1: Performance comparison of daVinci-LLM-3B against baseline models with score across three capability domains, and overall score comparable to OLMo-3-7B.

Figure 2: Evolution of pretraining research depth across institutional structures. The y-axis represents research depth from surface-level artifacts (API-only access) to fundamental scientific questions. The x-axis shows the temporal progression from 2022 to 2026. Commercial entities (blue) possess computational resources but remain constrained to API-level access due to competitive pressures. Open-weight releases (green, e.g., Llama, Qwen) provide model artifacts but withhold design rationale and negative results. Academic efforts (orange, e.g., OLMo) achieve transparency and research freedom but face severe scale limitations—making systematic exploration with 200+ configurations structurally infeasible. The top tier remains largely unexplored, as it requires the rare alignment of large-scale computational resources with the research freedom to publish comprehensive findings. daVinci-LLM (purple) occupies this intersection, conducting the extensive ablations and systematic disclosures necessary to advance the science of pretraining. By releasing the complete decision-making logic alongside the model weights, we bridge the structural gap between industrial scale and scientific transparency.

	Dimension	Llama 3	Qwen 3	YuLan	OLMo 3	daVinci
Model Artifacts	Model Weights
	Training Code
	Training Logs
	Intermediate Checkpoints
Data Openness	Data Composition
	Processing Pipeline

Table 1: Transparency comparison across state-of-the-art LLMs. Unlike existing models, daVinci provides the complete scientific process, enabling the systematic investigation of pretraining dynamics. Symbols: fully open, patial disclosure or not released, not disclosed.

Figure 3: Mapping of our pretraining data sources onto the Data Darwinism L0–L9 taxonomy across different training stage.

Data Source	Pool Size	Level	Stage 1		Stage 2
Data Source	Pool Size		Stage 1-1	Stage 1-2	Stage 2-1	Stage 2-2
General	4.28T		2.73T (68.2%)	1.11T (55.42%)	100B (10%)	188.4B (18.84%)
Nemotron-CC-v1	4.28T	L3	✓	✓	✓	✓
Code	598B		381B (9.53%)	233B (11.66%)	300 (30%)	26.1B (2.61%)
Self-Crawled GitHub	187B	L3	✓	✓	✓	✓
Nemotron-Pretraining-Code-v1-non-synthetic	220B	L3	✓	✓	✓	✓

Table 2: Data pool composition and token allocation across training stages. Shaded category rows show each category’s total pool size and the tokens allocated per stage; data source rows indicate whether the source is used (✓) or not used (–) in each stage.

Data Source	General	Code	Science
Nemotron-CC-v1 Synthetic QA (492B)
Nemotron-Pretraining-SFT-v1-General (12B)
Nemotron-Pretraining-SFT-v1-Code (21B)
Llama-Nemotron-Post-Training-Code (5B)
Nemotron-Pretraining-SFT-v1-Math (138B)
Llama-Nemotron-Post-Training-Math-RS (10B)

Table 3: Domain coverage of QA data sources. Each source is categorized into one of three domains: General, Code, and Science (including Mathematics).

Parameters	$\sim$ 3.09B	MLP intermediate size	11008
Layers)	36	Activation function	SwiGLU
Hidden size	2048	Positional encoding	RoPE (base $\theta=10000$ )
Attention heads	16	Max position embeddings	4096
KV heads (GQA)	2	Normalization	RMSNorm ( $\epsilon=10^{-6}$ )
Head dimension	128	Tokenizer	Qwen2 tokenizer
Vocab size	151936	Precision	bfloat16

Table 4: Architectural specification of daVinci-LLM .

	Stage 1		Stage 2
	Stage 1-1	Stage 1-2	Stage 2-1	Stage 2-2
Training Tokens	4 Trillion	2 Trillion	1 Trillion	1 Trillion
Global Batch Size	1024 $\rightarrow$ 2048 $\rightarrow$ 4096	4096	4096	4096
Rotatory Base	10000	10000	10000	10000
Sequence Length	4096	4096	4096	4096
Precision	bfloat16	bfloat16	bfloat16	bfloat16

Table 5: Training hyperparameters of different training stages. Note that the Global Batch Size refers to the total number of sequences per training step; the total number of tokens per step is calculated as

\text{Global Batch Size}\times\text{Sequence Length}

Figure 4: Progressive training results across Stage 1-1 and Stage 1-2, with checkpoints evaluated every 5000 steps. The vertical dashed line indicate the boundary between two substages.

Panel of Figure 5 — Figure 5: Stage 1 Training dynamics for the first 300k steps: (a) training loss curve demonstrating consistent convergence, and (b) gradient norm curve tracking optimization stability across the initial training phase.

Figure 6: Progressive training results across Stage 2-1 and Stage 2-2, with checkpoints evaluated every 5000 steps. The vertical dashed line indicate the boundary between two substages.

		Fully-open Models				Open-weight Models
Domain	Benchmark	daVinci 3B	OLMO-3 7B	OLMO-2 7B	Yulan 2.4B	LLaMa-3.2 3B	Qwen-2.5 3B	Qwen-3 4B	Qwen-3.5 4B
General	MMLU	62.53	66.53	65.93	50.70	54.91	65.73	75.35	72.75
	MMLU-Pro	43.50	35.70	28.40	23.90	24.50	39.00	54.10	53.50
	AGIEval	26.77	33.75	31.78	28.22	22.72	37.15	45.87	44.89
	HellaSwag	71.17	74.15	80.50	68.56	73.60	73.60	73.75	75.29
	TriviaQA	49.90	55.45	68.01	27.64	55.22	51.20	47.44	49.80

Table 6: Comprehensive evaluation across diverse capability benchmarks. daVinci-LLM-3B is compared against state-of-the-art open-weight models.

Configuration	HumanEval	EvalPlus	MBPP	Avg Gen.	Avg Code	Avg Sci.	Overall
Rule-Based Filtering (L2)	54.43	48.58	42.40	51.12	48.47	40.14	46.66
Model-Based Filtering (L3)	54.70	48.39	45.80	51.19	49.63	40.28	46.93
$\Delta$ (L3 – L2)	+0.27	-0.19	+3.40	+0.07	+1.16	+0.14	+0.27

Table 7: Comparison of rule-based and model-based filtering approaches.

Configuration	GSM-8K	GSM-Plus	MATH	Avg Gen.	Avg Code	Avg Sci.	Overall
Baseline	64.06	40.58	38.00	51.69	49.80	40.52	47.27
Generative Refined (L4)	65.43	42.38	45.00	51.40	50.15	42.25	47.83
$\Delta$ (L4 – Baseline)	+1.37	+1.80	+7.00	-0.29	+0.35	+1.73	+0.56

Table 8: Comparison of baseline and generative refinement approaches.

Figure 7: Effectiveness of CC-QA and CodeQA synthetic data in Stage 2-2.

Figure 8: Stage 1 benchmark performance. Light curves show individual benchmarks; dark curves show category averages for general knowledge, code, and science. Two vertical dashed lines indicate the boundaries between substages 1-1, 1-2, and 1-3, respectively.

Figure 9: Performance comparison between continued data domain adjustment (stage 1-3) and QA introduction (Stage 2).

Figure 10: Stage 2 benchmark performance. Light curves show individual benchmarks; dark curves show category averages for general knowledge, code, and science. The vertical dashed lines indicate the boundaries between Substages 2-1, 2-2.

Panel of Figure 11 — Figure 11: Code and science data ratio analysis. Notation: C-X-S-Y indicates X% code and Y% science. (a) Performance across code and science proportions (QA fixed at 30%, remainder allocated to CommonCrawl web data). (b) Data synergy effects across QA ratios in Stage 2-1.

Configuration	Training Tokens	General	Code	Science	Overall
QA-30%	84B	51.59	47.71	40.25	46.80
	252B	51.66	46.79	40.03	46.61
	419B	51.80	47.16	40.59	46.94
QA-50%	84B	51.72	49.86	41.60	47.70
	252B	51.85	49.01	42.80	48.06
	419B	51.83	49.89	43.42	48.43

Table 9: Impact of QA data ratio on model performance in Stage 2-2.

Figure 12: MMLU performance comparison across evaluation protocols. Panel (a) shows absolute scores under PPL-based and generative (CoT) evaluation. Panel (b) highlights the performance gap relative to Qwen-2.5-3B, demonstrating how models with extensive QA training exhibit amplified advantages in generative settings. The gap between OLMO-2-7B and Qwen-2.5-3B increases from -0.20% (PPL) to +2.90% (generative), a 3.10% swing that reverses the ranking.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Avg General	Avg Code	Avg Science	Overall Avg
Constant LR	52.06	52.32	45.44	49.66
Cosine decay	52.49	53.31	45.39	50.00
$\Delta$ (decay - constant)	+0.43	+0.99	-0.05	+0.34

Appendix

Table 10: Impact of learning rate decay strategy on Stage 2-2 performance. Constant LR maintains 3e-5 throughout, while cosine decay gradually reduces LR from 3e-5 to 3e-6.

Figure 13: Impact of QA masking policy on three capability dimensions across training steps in Stage 2-2. The comparison between masked and unmasked configurations shows marginal improvements, with consistent but modest gains across all capability clusters.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

Sequential Pretraining Favors Large Models
Oct 7, 2026Mohnish Harwani, Yujia ZhengLanguage Model Pretraining Continued Pretraining
Purdue University · University of Illinois Urbana–Champaign
What do Language Models Learn and When? The Implicit Curriculum Hypothesis
Apr 9, 2026Emmy Liu, Kaiser Sun, Millicent Li +4Emergent Abilities in Language Models Language Model Pretraining
Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University +1
Small Initialization Matters for Large Language Models
Jun 16, 2026Liangkai Hang, Junjie Yao, Zhiyu Li +3Language Model Pretraining Neural Network Initialization
School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China. · MemTensor (Shanghai) Technology Co., Ltd.. for Advanced Algorithms Research, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China.

Oct 7, 2026J/K move · Enter open · S save

Sequential Pretraining Favors Large Models

Mohnish Harwani, Yujia Zheng

Purdue University · University of Illinois Urbana–Champaign

Language Model Pretraining Continued Pretraining

Open paper AskPDF arXiv

Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.

Abstract

Language Model Pretraining

Explore similar work

Sequential Pretraining Favors Large Models

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

Small Initialization Matters for Large Language Models