Arcee Trinity Large Technical Report

Varun Singh; Lucas Krauss; Sami Jaghouar; Matej Sirovatka; Charles Goddard; Fares Obied; Jack Min Ong; Jannik Straube

cs.LGFeb 19, 2026

Arcee Trinity Large Technical Report

Authors: Varun Singh, Lucas Krauss, Sami Jaghouar, Matej Sirovatka, Charles Goddard, Fares Obied, Jack Min Ong, Jannik Straube,

Organizations: Arcee AI · Prime Intellect · DatologyAI

Abstract

We present the technical report for Arcee Trinity Large, a sparse Mixture-of-Experts model with 400B total parameters and 13B activated per token. Additionally, we report on Trinity Nano and Trinity Mini, with Trinity Nano having 6B total parameters with 1B activated per token, Trinity Mini having 26B total parameters with 3B activated per token. The models' modern architecture includes interleaved local and global attention, gated attention, depth-scaled sandwich norm, and sigmoid routing for Mixture-of-Experts. For Trinity Large, we also introduce a new MoE load balancing strategy titled Soft-clamped Momentum Expert Bias Updates (SMEBU). We train the models using the Muon optimizer. All three models completed training with zero loss spikes. Trinity Nano and Trinity Mini were pre-trained on 10 trillion tokens, and Trinity Large was pre-trained on 17 trillion tokens. The model checkpoints are available at https://huggingface.co/arcee-ai.

Figures & tables

Figure 1: The training loss graph for Trinity Large, with no sub-sampling or smoothing. For clarity, we indicate where the batch size was increased to 128M (134,217,728) tokens, as well as the points where we switch data mixtures.

Figure 2: The architecture of the Trinity model family. ∗ RoPE is only present in local layers. † The grouped-query attention has a sliding window for the local layers.

Dataset	Trinity (200k)	DeepSeek R1 (128k)	Qwen 3 (152k)	Llama 3 (128k)	GPT-OSS (200k)
C4 (en) B/T	4.84	4.70	4.64	4.73	4.79
C4 (zh) C/T	1.38	1.54	1.45	1.29	1.47
C4 (ja) C/T	1.39	1.60	1.59	1.60	1.54
C4 (ko) C/T	1.26	1.47	1.47	1.68	1.72
C4 (fr) B/T	3.98	3.53	3.34	3.49	3.97
Reasoning B/T	3.67	3.61	3.51	3.59	3.61

Table 1: Comparison of Bytes-per-token (B/T) and Characters-per-token (C/T) across languages and domains.

	Trinity Nano	Trinity Mini	Trinity Large
Transformer layers	56	32	60
Initial dense layers	2	2	6
Model dim ( $d_{\text{model}}$ )	1024	2048	3072
FFN intermediate dim	3072	6144	12288
Attention heads ( $h_{q}$ )	8	32	48
Per-head dim ( $d_{h}$ )	128	128	128

Table 2: Model configurations for the Trinity model variants.

Figure 3: A comparison of Trinity Large Base to other similar open-weight base models.

Evaluation	Score
MBPP+	88.62
Minerva MATH500	65.20
HellaSwag (5-shot)	90.11
WinoGrande (5-shot)	80.82
MMLU (5-shot)	82.58
MMLU-Pro (5-shot)	66.02

Table 3: Trinity Large Base performance on our base model evaluation suite.

Evaluation	Score
MMLU	87.21
MMLU-Pro	75.25
GPQA Diamond	63.32
SimpleQA	23.92
AIME25	24.36

Table 4: Trinity Large Preview performance on our instruct model evaluation suite.

Figure 4: Throughput comparison of models. All tests were done with models quantized to FP8, using vLLM, on 8xH200.

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 342 papers

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Apr 28, 2026Fan Jiang, Yu Zhao, Chenyang Lyu +5Multilingual Language Models Mixture-of-Experts Language Models
Alibaba International Digital Commerce
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
Slicing and Dicing: Configuring Optimal Mixtures of Experts
May 12, 2026Margaret Li, Sneha Kudugunta, Danielle Rothermel +1Mixture-of-Experts Language Models Mixture of Experts
Paul G Allen School of Computer Science University of Washington Seattle, USA · Courant School of Data Science New York University New York, USA

Apr 28, 2026J/K move · Enter open · S save

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi+4

Alibaba International Digital Commerce

Multilingual Language Models Mixture-of-Experts Language Models Activation Sparsity Efficient Language Model Training Open-Weight Language Models Sparse Neural Networks

Open paper AskPDF arXiv

We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-Instruct variants, which surpass the performance of competing models possessing

3

14\times

more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.

Abstract

LLM Training

Explore similar work

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

Slicing and Dicing: Configuring Optimal Mixtures of Experts