Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison

Marianna Nezhurina; Harsh Raj; Victor May; Jörg Franke; Taishi Nakamura; Timur Carstensen; Niccolò Ajroldi; Ville Komulainen

cs.LGSep 10, 2025

Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison

Authors: Marianna Nezhurina, Harsh Raj, Victor May, Jörg Franke, Taishi Nakamura, Timur Carstensen, Niccolò Ajroldi, Ville Komulainen,

Organizations: LAION · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ) · Open-Ψ (Open-Sci) Collective · OpenEuroLLM team ·

Abstract

Reliable progress in language modeling depends on knowing whether a new dataset, architecture, or training recipe is genuinely better than existing alternatives. To foster reproducible learning procedure comparison, we present open-sci-ref, a family of dense transformers trained as open reference baselines at four scales (0.13B-1.7B parameters) on nine main reference datasets, with token budgets up to 1T, using a single tuned and fully documented training procedure. To facilitate a shared experimentation and measurement infrastructure, we release model weights, intermediate checkpoints, training logs, code, and downstream evaluations. When comparing training procedures, we take into account total compute spent. Placing training procedures on a common compute axis helps identify reference points that are weak or strong for the compute they use, without requiring a full scaling law study for every comparison. We identify Nemotron-CC-HQ as the strongest reference dataset on our evaluation suite: our 1.7B model trained on 1T of its tokens matches SmolLM2-1.7B with roughly 11 $\times$ less compute. Shared baselines expose pitfalls of scaling trend comparisons: when a small-scale point is unusually weak, the resulting slope can overstate the scalability of a training procedure. We illustrate this diagnostically with EuroLLM-1.7B, which underperforms 300B token FineWeb-Edu baseline despite 13 $\times$ more training compute. As an additional case study, the suite supports controlled comparison between permissively licensed and mixed-license datasets: the permissive-first MixtureVitae dataset changes rank substantially with scale. This also illustrates why data ranking should be evaluated through multi-scale reference trajectories rather than single proxy runs. All artifacts are publicly released.

Figures & tables

Figure 1: Why shared reference baselines are needed. Left : published models differ on every axis at once, so an observed gap cannot be attributed to any single cause. Right : varying one factor (here the pretraining data) while matching the rest and placing runs on a common compute axis isolates its effect under a shared protocol. Rank order may not be preserved across scales.

Params (B) (Non-Emb + Emb)	Layers	Hidden	Heads	FFN Hidden	GPU Memory Usage	FLOPs/token (6N estimate)
0.1 + 0.03 = 0.13	22	512	8	2256	0.89GB	$7.8\cdot 10^{8}$
0.35 + 0.05 = 0.40	22	1024	16	3840	2.32GB	$2.4\cdot 10^{9}$
1.21 + 0.10 = 1.31	24	2048	32	5440	7.54GB	$7.9\cdot 10^{9}$
1.61 + 0.10 = 1.71	24	2048	32	8192	9.88GB	$1.0\cdot 10^{10}$

Table 2: open-sci-ref model architecture and scales.

Figure 2: Average downstream performance over training for full reference trajectories at four model scales. Rankings are broadly stable among conventional datasets, with gaps widening at larger scales. MixtureVitae is the main exception: it begins near the bottom at 0.13B but rises to the HPLT/FineWeb-Edu group at 1.7B, illustrating why single-scale dataset rankings can be misleading.

Figure 3: Downstream performance across 11 tasks of open-sci-ref models with 1.7B parameters trained on 300B tokens over different datasets. While some tasks yield clear dataset rankings (e.g., ARC, HellaSwag, LAMBADA), others provide insufficient signal for meaningful dataset comparison.

Figure 4: Downstream performance on 11 tasks for open-sci-ref 1.7B models trained for 1T tokens on the top three reference datasets. Nemotron-CC-HQ has the strongest average endpoint, while DCLM is strongest on LAMBADA and competitive on several continuation-style tasks.

Controlled open-sci-ref-0.02 endpoints (all open-sci-ref-1.7B )
Model / dataset	Tok.	Par.	FLOPs	Avg.	COPA	LAMB.	OBQA	Wino.	MMLU	ARC-C	ARC-E	BoolQ	CSQA	HSwag	PIQA
	(T)	(B)	( $10^{21}$ )
Nemotron-CC-HQ	1.0	1.7	10.2	.67	.86	.61	.44	.66	.52	.53	.82	.76	.60	.74	.80
DCLM	1.0	1.7	10.2	.64	.82	.70	.41	.66	.40	.48	.77	.77	.47	.74	.78
FineWeb-Edu	1.0	1.7	10.2	.60	.84	.58	.44	.64	.32	.50	.80	.75	.23	.70	.79
FineWeb-Edu	0.3	1.7	3.06	.57	.82	.56	.42	.62	.26	.48	.77	.65	.21	.67	.77

Table 3: Per-benchmark endpoint results for selected open-sci-ref-0.02 1.7B reference baselines and external models. All rows in the upper block are open-sci-ref-1.7B runs trained on the listed dataset, where architecture, tokenizer, training recipe, evaluation protocol, and compute accounting are fixed. Bold marks the best value per column within this controlled block (ties bolded jointly). The upper block covers the three 1T endpoints, the 300B FineWeb-Edu reference, and the three permissively licensed datasets. Figure 2 shows all nine datasets at 300B. The lower block reports external models for orientation, with their training dataset in parentheses where known. These rows are not controlled comparisons because they differ in data, tokenizer, architecture details, and training recipe, and are therefore not included in the bolding. Training tokens are given in trillions and training compute in units of

10^{21}

FLOPs, estimated as

6ND

from the parameter count

N

and the training token count

D

. Scores are accuracies reported without the leading zero. Avg. is the unweighted mean over the 11 listed evaluations, including CommonSenseQA. LAMB. denotes LAMBADA, OBQA denotes OpenBookQA, Wino. denotes WinoGrande, CSQA denotes CommonSenseQA, and HSwag denotes HellaSwag.

Figure 5: Scaling trends for open-sci-ref-0.02 on a common estimated training compute axis. Solid lines show controlled open-sci-ref runs, where architecture, tokenizer, training recipe, evaluation protocol, and compute accounting are fixed. Dashed lines show selected external model families: these are not controlled comparisons because the models differ in data, tokenizer, architecture details, and training recipe.

Figure 6: Performance of English-dominant open-sci-ref on MMMLU, which is a version of the MMLU benchmark translated to multiple languages by OpenAI employing human professional translators.

Figure 7: Average performance on Belebele for different languages while training a 1.7B model on Nemotron-CC-HQ.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Token budget	Global batch	Iterations	Learning rate	Warmup
50B	2.65M	18,839	$3\times 10^{-3}$	6,000
50B	4.03M	12,406	$6\times 10^{-3}$	1,000
300B	2.09M	143,052	$3\times 10^{-3}$	5,000

Appendix

Table 4: Cosine learning rate configurations used during recipe tuning. These runs were used to establish validation loss targets before switching to the WSD schedule.

Token budget	Global batch	Iterations	Learning rate	Warmup	Cooldown
50B	2.65M	18,839	$2\times 10^{-3}$	6,000	3,767
50B	4.12M	11,921	$4\times 10^{-3}$	1,000	2,384
300B	2.09M	143,052	$1\times 10^{-3}$	5,000	28,610
300B	4.12M	72,661	$4\times 10^{-3}$	25,000	14,532
1T	4.12M	242,204	$4\times 10^{-3}$	25,000	48,440

Appendix

Table 5: WSD learning rate configurations. Cooldown is 20% of the total training duration. The 4.12M-token global batch rows define the main reference recipe used for the longer 300B- and 1T-token runs.

Figure 8: Comparing training with different context length. Training run for open-sci-ref-0.02 1.7B model on 100B tokens of Nemotron-CC-HQ, using various context length of 4096 (RoPE base 100k), 8192 (RoPE base 500k) and 16384 (RoPE base 1M). We observe no difference in average performance, ensuring models with larger context length have no drop in performance compared to reference context length of 4096.

Table 7: open-sci-ref reference baseline distributed training runs. The TFLOP/s/GPU column follows the same accounting as Tab. 6 .

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 342 papers

OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research
Jul 18, 2026Tavish Mankash, Vardhaman Kalloli, Keshava Prasad +1Language Model Pretraining Small Language Models
FAIRC · Plaksha University
ScAn-Bench: Evaluating Scaling Analysis Methodology
Sep 28, 2026Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +4VLM Evaluation LLM Evaluation
University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Language Model Scaling Laws Efficient Language Model Training
University of Luxembourg · University of Twente · Eindhoven University of Technology +1

Jul 18, 2026J/K move · Enter open · S save

OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

Tavish Mankash, Vardhaman Kalloli, Keshava Prasad, Deepan Muthirayan

FAIRC · Plaksha University

Language Model Pretraining Small Language Models Efficient Language Model Training

Open paper AskPDF arXiv

OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models while keeping their machinery visible. In OLM, model code reads like the architecture: components are ordinary modules, while Block, Residual, Repeat, and Parallel describe how they are wired. The resulting model can move unchanged from a teaching notebook to a complete pretraining run or a research ablation. OLM connects this readable model layer to tokenizers, local and streaming datasets, optimization, mixed precision, callbacks, checkpoints, and hardware-aware CPU, single-GPU, and single-node multi-GPU execution. We demonstrate the full path by tracing GPT-2 from diagram to code, launching a FineWeb-Edu training script, replacing one attention component, and letting AutoTrainer configure the available machine. The package includes 27 presets across nine familiar model families and documentation that progresses from LM fundamentals to architecture research. Validation shows close agreement with independent reference implementations, 90.6% four-GPU weak-scaling efficiency for a 348M-parameter workload, compact architecture edits, and positive early usability results. OLM is MIT-licensed and available through PyPI, GitHub, and its documentation site.

Abstract

LLM Training

Explore similar work

OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

ScAn-Bench: Evaluating Scaling Analysis Methodology

When Data Is Scarce: Scaling Sparse Language Models with Repeated Training