EngGPT2: Sovereign, Efficient and Open Intelligence

G. Ciarfaglia; A. Rosanova; S. Cipolla; J. Bartoli; A. Di Domenico; C. Fioroni; A. Fontana; M. R. Scoleri

cs.CLMar 17, 2026

EngGPT2: Sovereign, Efficient and Open Intelligence

Authors: G. Ciarfaglia, A. Rosanova, S. Cipolla, J. Bartoli, A. Di Domenico, C. Fioroni, A. Fontana, M. R. Scoleri,

Abstract

EngGPT2-16B-A3B is the latest iteration of Engineering Group's Italian LLM and it's built to be a Sovereign, Efficient and Open model. EngGPT2 is trained on 2.5 trillion tokens - less than Qwen3's 36T or Llama3's 15T - and delivers performance on key benchmarks, including MMLU-Pro, GSM8K, IFEval and HumanEval, comparable to dense models in the 8B-16B range, while requiring one-fifth to half of the inference power, and between one-tenth to one-sixth of the training data and consequent needed training power. Designed as a trained-from-scratch Mixture-of-Experts (MoE) architecture, EngGPT2 features 16 billion parameters with 3 billion active per inference, with expert sizes positioned between those used in GPT-OSS and Qwen3. Approximately 25% of its training corpus consists of Italian-language data, to deliver strong capabilities for European and Italian NLP tasks among models of similar scale. This efficiency aims to position EngGPT2 as a key contributor to the growing portfolio of open-weight European models, combining performance and efficiency with full alignment to the EU AI Act. EngGPT2 is also a single model capable of multiple reasoning modes: non-reasoning, reasoning in Italian or English, and turbo-reasoning (a concise, bullet-point style reasoning available in both languages designed for real-time reasoning use cases). EngGPT2 aims to set a new standard for resource-conscious, high-performance LLMs tailored to European and Italian contexts.

Figures & tables

Figure 1: Comparison of model evaluation and efficiency. Top: model comparison under the Cost of Intelligence framework. Bottom: EngGPT2 average evaluation comparison against main open-weight comparable size models.

Feature	Value	Feature	Value
Number of Layers	$24$	Num attention heads	$32$
Total number of experts	$64$	Num key value heads	$4$
Experts activated/token	8	Context length	$32768$
Vocabulary size	$131084$

Table 1: Architectural Parameters

Model	# Params (Total / Active)	Single Expert Size	H/m	% Active Params	Layers	Experts (Total / Activated)	Heads (Q /KV)	Vocab size
EngGPT2-16B-A3B	16B / 3B	9.3M	2880 / 1080	$20.27\%$	24	64 / 8	32 / 4	131084
Qwen3-30B-A3B	30B / 3B	4.7M	2048 / 768	$10.98\%$	48	128 / 8	32 / 4	151936
Openai-gpt-oss-20B	21B / 3.6B	24.9M	2880 / 2880	$17.30\%$	24	32 / 4	64 / 8	201088

Table 2: MoE Comparison

Dataset	License	Stage 1 (600B token)	Stage 2 (1.5T token)	Stage 3 (400B token)
Fineweb-edu (english)	odc-by	52%	48%	21%
Fineweb2 - ita	odc-by	23%	12%	15%
Finepdf - ita	odc-by	0%	5%	5%
Fineweb2 - rom lang	odc-by	12%	12%	6%
Finemath	odc-by	1%	3%	0%
Nemotron-Pretraining (SFT-v1 - Math)	NVIDIA 1 1 1 Data governed by the NVIDIA Data Agreement for Model Training (https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-Dataset-sample/raw/main/LICENSE.md)	0%	4%	17%

Table 3: Pretrain datasets. The listed datasets are subject to different licensing regimes, including open license, usage-restricted agreements and curated collections of permissively licensed content.

Figure 3: Throughput during the first three stages of pre-training.

Figure 4: MMLU accuracy and perplexity across the first three pre-training phases as a function of the number of processed training tokens. MMLU evaluation is conducted on a fixed subset of the benchmark to monitor the emergence of general knowledge throughout training. English perplexity is measured on a fixed subset of WikiText-2, while Italian perplexity is computed on a fixed subset of a curated Wikimedia-based Italian corpus, following the same evaluation protocol used for English. The decreasing perplexity trend reflects progressively improved predictive performance on held-out data.

Figure 5: RULER long-context evaluation across increasing context lengths. Comparison between the final pre-training checkpoint (post stage 3) and the long-context-adapted model on the RULER benchmark. Both models are extended beyond 64k tokens using YARN to enable evaluation at long context lengths. Results are reported as average accuracy across the RULER task suite for context sizes ranging from 4k to 64k tokens.

Dataset	Licence	Token Composition (45B token)
Llama-Nemotron-Post-Training-Dataset_reasoning_r1	cc-by-4.0 3 3 3 The dataset is largely open-licensed; samples with more restrictive terms were excluded from our training subset.	42%
OpenThoughts3-1.2M	Apache-2.0	12%
Dolci-ThinK-SFT-32B	odc-by	46%

Table 4: Mid-Train datasets

Step	MMLU	$\Delta$
12288	65.68	+11.01
16800	67.02	+1.33
20800	67.02	+0.00
23841	68.46	+1.44

Table 5: MMLU accuracy across mid-training checkpoints.

Dataset	#Tokens (M)	Mode	Task Type

Policies Alignment	5.3	RI, RE, NR	Policies; Safety
Metrics Alignment	31.4	RE, NR	Metrics
General Purpose	23.6	RE, NR	Creative Writing; Text Processing;Information Extraction; Structured Output; SQL-like Generation; Code Generation; Code Migration; Code Related Generation; ActLike; Data Mng; Agentic Orchestration; Misc
Mathematical Problems	11.3	RE	Math
Reasoning Capabilities	186.7	RE	Problem Solving; Code Generation; Code Related Generation

Table 6: Training dataset mixture used for the SFT stage.

Stage	MMLU-Pro	MMLU-Redux	AIME26
Mid-Training	50.4	66.8	47
SFT Stage	50.7	73.2	53
APO Stage	57.3	75.5	70

Table 7: KPIs across post-training stages.

Group	Model	MMLU-Pro	MMLU-Redux	IFEval	HumanEval	AIME25	AIME26	GSM8K	BFCL
Comparable	EngGPT2-16B-A3B	57.3	75.5	72	64	60	70	88	48.5
	Moonlight-16B-A3B-Instruct	47.1	70.1	51.4	77.4	20.0	10.0	83.8	42.2
	Llama-3.1-8B-Instruct	49.3	68.8	79.5	73.8	10.0	0.1	82.7	28.9
	gemma-2-9b-it	49.2	74.7	75.2	–	0.0	0.0	82.3	46.3
Larger	Gpt-oss-20b-high	75.2	89.6	86.8	94.5	80	86.7	93.3	60.9
	Qwen3-14B	77.3	89.9	90.4	97	83.3	86.7	93	74.3

Table 8: Benchmark results obtained using a standardized generation configuration across all evaluated models to ensure strict cross-model comparability.

Group	Model	MMLU-Pro	MMLU-Redux	IFEval	HumanEval	AIME25	AIME26	GSM8K	BFCL
Comparable	EngGPT2-16B-A3B	57.3	75.5	72.0	64.0	60.0	70.0	88.0	48.5
	Moonlight-16B-A3B-Instruct	47.1	70.1	51.4	77.4	20.0	10.0	83.8	42.2
	Llama-3.1-8B-Instruct	49.2	68.4	81.9	75.0	3.3	0.0	82.7	37.4
	gemma-2-9b-it	48.2	74.8	72.0	–	0.0	0.0	82.3	44.4
Larger	Gpt-oss-20b-high	63.1	89.8	91.1	98.2	73.3	76.7	93.9	58.9
	Qwen3-14B	77.3	89.9	90.4	97	83.3	86.7	93.0	74.3

Table 9: Comparison among EngGPT2-16B-A3B and other comparable models, each evaluated under its optimal serving and decoding configuration (see Appendix E ), reflecting maximum achievable performance.

Group	Model	ARC-IT Standard-config	MMLU-IT Standard-config	ARC-IT Optimal-config	MMLU-IT Optimal-config
Comparable	EngGPT2-16B-A3B	85.6	65.5	85.6	65.5
	Moonlight-16B-A3B-Instruct	63.1	49.2	63.1	49.2
	Llama-3.1-8B-Instruct	81.4	60.6	80.0	60.6
	gemma-2-9b-it	88.4	67.2	87.6	66.5
Larger	Gpt-oss-20b-high	92.6	80.2	93.2	80.4
	Qwen3-14B	89.2	77.6	89.2	77.6

Table 10: Comparison between EngGPT2 - 16B - A3B and other comparable models on Italian benchmarks, evaluated under both standardized and optimal configurations (see Appendix E ).

Figure 6: Visual comparison of general knowledge, reasoning and Italian benchmarks across EngGPT2-16B-A3B and representative open-weight models.

Figure 7: Normalized average benchmark score per model, derived from row-wise normalized benchmark results.

Figure 8: Model comparison under the Cost of Intelligence framework. The horizontal axis represents training efficiency (normalized performance per training token), while the vertical axis represents inference efficiency (normalized performance per active parameter). Models positioned toward the upper-right region exhibit higher overall efficiency.

Benchmark	Reasoning (EN)	Reasoning (ITA)	Turbo (EN)	Turbo (ITA)
MMLU-Pro	57.3	57.2	41.3	40.9
MMLU-Redux	75.5	68.0	62.6	62.1
AIME25	60.0	60.0	16.7	30.0
AIME26	70.0	63.2	10.0	23.0
GSM8K	88.0	88.0	74.4	75.8

Table 11: Reasoning Modalities Evaluation

Figure 9: Benchmark performance across full and compressed reasoning configurations in English and Italian.

Benchmark	Avg Tokens (Reasoning)	Avg Tokens (Turbo)	Perf. Drop (%)	Tokens Drop (%)
MMLU-Pro	2990	249	16.0	91.67
MMLU-Redux	1245	132	12.9	89.40
AIME25	20061	746	43.3	96.28
AIME26	19590	1753	60.0	91.05
GSM8K	731	122	13.6	83.31

Table 12: Reasoning Efficiency Trade-off

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Link List
Media & News	European Newspapers Directory – Browse National & Regional News by Country 4 4 4 https://newspapers-europe.eu/
Academic	sciencedirect.com, nature.com, tandfonline.com
Broadcasting	rai.it, bbc.co.uk, zdf.de, rtve.es
Legal	wolterskluwer.it, chbeck.de, dalloz.fr
Literary	mondadori.it, hachette.fr, penguin.co.uk
Professional	ipsoa.it, de jure.it, beck-online.de

Appendix

Table 13: URL List

Metric	Percentage
Clean Records	$76.26\%$
Flagged records ( $\geq 1$ signal)	$23.74\%$
High-risk records ( $\geq 0.5$ signal)	$1.65\%$

Appendix

Table 14: Copyright Filtering Statistics

Category	Task Type	Sub-Tasks

Text	Creative Writing	storytelling; talk about a theme; email writing; trip planner; dialog generation; QA generation; document generation; brainstorming
	Text Processing	rephrase; summarization; translation; NER; text filtering; text masking; text classification; focus shift; style adaptation; text expansion; text projection
	Information Extraction	grounded QA; not-grounded QA; keyword extraction; keyphrase extraction; needleHaystack
	Structured Output	json-like output; markdown output

Appendix

Table 15: Task taxonomy.

	Temperature	Top_p	Top_k	Min_p
EngGPT2-16B-A3B	0.6	0.95	20	0
Qwen3-14B 5 5 5 https://huggingface.co/Qwen/Qwen3-14B#best-practices	0.6	0.95	20	0
Qwen3-30B-A3B 6 6 6 https://huggingface.co/Qwen/Qwen3-30B-A3B#best-practices	0.6	0.95	20	0
Gpt-oss-20b-high 7 7 7 https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune#recommended-settings	1	1	0	–
Llama-3.1-8B-Instruct 8 8 8 https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct . We evaluated the new model version: Llama-3.1-8B-Instruct [ 55 ] .	0.6	0.9	–	–
gemma-2-9b-it 9 9 9 https://huggingface.co/google/gemma-2-9b#advanced-usage	1	–	–	–

Appendix

Table 16: Optimal serving parameters and reference sources for all evaluated models, as used in the final evaluation. If no value is reported for a given parameter, the default setting from the lm-evaluation-harness framework was used.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
May 8, 2026Andrea Sassella, Andrea Chizzola, Tommaso Bianchi +2LLM Evaluation Mixture-of-Experts Language Models
2DEIB, Politecnico di Milano · 1AIRIC, Politecnico di Milano
Mellum2 Technical Report
May 29, 2026Marko Kojic, Ivan Bondyrev, Aral de Moor +6Mixture-of-Experts Language Models Open-Weight Language Models
1JetBrains · 2Constructor University, Bremen, Germany
A Sovereign, Open-Source Foundation Model for German and English
Jul 10, 2026The Soofi-Team, :, Benedikt Droste +29Language Model Pretraining Multilingual Language Models
ellamind · Fraunhofer IAIS · Lamarr +8

May 8, 2026J/K move · Enter open · S save

Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs

Andrea Sassella, Andrea Chizzola, Tommaso Bianchi, Luca Alessandrelli+1

2DEIB, Politecnico di Milano · 1AIRIC, Politecnico di Milano

LLM Evaluation Mixture-of-Experts Language Models Open-Weight Language Models Language Modeling

Open paper AskPDF arXiv

This report benchmarks the performance of ENGINEERING Ingegneria Informatica S.p.A.'s EngGPT2MoE-16B-A3B LLM, a 16B parameter Mixture of Experts (MoE) model with 3B active parameters. Performance is investigated across a wide variety of representative benchmarks, and is compared against comparably-sized open-source MoE and dense models. In comparison with popular Italian models, namely FastwebMIIA-7B, Minerva-7B, Velvet-14B, and LLaMAntino-3-ANITA-8B, EngGPT2MoE-16B-A3B performs as well or better on international benchmarks: ARC-Challenge, GSM8K, AIME24, AIME25, MMLU, and HumanEval (HE). It achieves the best performance for the longest context setting (32k) of the RULER benchmark. On the Italian benchmark dataset ITALIC, the model performs as well or better than the other models except for Velvet-14B, which outperforms it. Compared with popular MoE models of comparable size, the new model reports higher values than DeepSeek-MoE-16B-Chat on all considered benchmarks. It has higher values than Moonlight-16B-A3B on HE, MMLU, AIME24, AIME25, GSM8K, and the 32k RULER setting, but lower on BFCL and some ARC and ITALIC settings. Finally it has lower values than GPT-OSS-20B on most benchmarks, including HE, MMLU, AIME24, AIME25, GSM8K, ARC, BFCL, and the RULER 32k. When compared with popular dense models, EngGPT2MoE-16B-A3B reports higher values on AIME24 and AIME25 than Llama-3.1-8B-Instruct, Gemma-3-12b-it, and Ministral-3-8BInstruct-2512-BF16, but lower values on ITALIC, BFCL, and RULER with a 32k context. When performance is aggregated across all benchmark metrics, EngGPT2MoE-16B-A3B shows higher performance than the Italian models under evaluation while achieving lower results than some of the most performant international models, in particular GPT-5 nano and Qwen3-8B. Taken together, our findings find the new model to be a step forward for native Italian Large Language Models.

Abstract

Language Modeling

Explore similar work

Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs

Mellum2 Technical Report

A Sovereign, Open-Source Foundation Model for German and English