Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet

Joel Lidin; Amir Sarfi; Erfan Miahi; Quentin Anthony; Shivam Chauhan; Evangelos Pappas; Benjamin Thérien; Eugene Belilovsky

cs.DCMar 9, 2026

Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet

Authors: Joel Lidin, Amir Sarfi, Erfan Miahi, Quentin Anthony, Shivam Chauhan, Evangelos Pappas, Benjamin Thérien, Eugene Belilovsky,

Organizations: Covenant AI · Mila

Abstract

Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-scale foundation models. However, existing models trained in a globally distributed manner are relatively small in scale and have only been trained with whitelisted participants. Therefore, they do not yet realize the full promise of democratized participation. In this report, we describe Covenant-72B, an LLM produced by the largest collaborative globally distributed pre-training run (in terms of both compute and model scale), which simultaneously allowed open, permissionless participation supported by a live blockchain protocol. We utilized a state-of-the-art communication-efficient optimizer, SparseLoCo, supporting dynamic participation with peers joining and leaving freely. Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run.

Figures & tables

Figure 1: Covenant-72B parallelism protocol. Each peer runs a SparseLoCo replica and communicates heavily compressed and 2-bit-quantized pseudo-gradients with other peers. Within each peer, $8\times$ B200 GPUs use dynamic FSDP to shard model parameters, gradients, and training states across local GPUs. During the computation phase (inner steps), GPU $i$ requires only the inner optimizer state shards InnerOpt State i while the error-feedback EF State i is offloaded. During the communication phase, InnerOpt State i is offloaded and swapped with EF State i to compute compressed pseudo-gradients and update the error-feedback buffer.

Figure 2: Learning rate schedule. Left: pre-training inner learning rate with linear warmup, cosine decay with a flatten window, followed by an annealing phase on higher-quality data. The cosine decay was flattened due to lower participation, which required a longer decay horizon. Right: Supervised fine-tuning schedule with a 4k-context cosine stage followed by an 8k-context cosine-then-linear stage.

Benchmarks (0-shot accuracy)
	INTELLECT-1	Psyche Consilience	Covenant-72B	LLM360 K2	LLaMA-2-7B	LLaMA-2-70B
Model size	10B	40B	72B	65B	7B	70B
Tokens	1T	1.2T	1.1T	1.4T	2T	2T
Training env.	Internet	Internet	Internet	Centralized	Centralized	Centralized
Permissionless	No	No	Yes	No	No	No
ARC-Challenge	44.8	31.1	56.8	53.8	45.1	57.4

Table 1: Pre-training benchmark comparison across centrally trained and decentralized baselines. We report zero-shot accuracy on ARC-C, ARC-E, PIQA, OpenBookQA, HellaSwag, WinoGrande, and MMLU. Among decentralized methods communicating over-the-internet, Covenant-72B uses permissionless participation, while INTELLECT-1 and Psyche Consilience use whitelisted participants; LLM360 K2 and the LLaMA models are trained in centralized clusters.

Figure 3: Compute–communication timelines over a two-hour window. Each row shows the breakdown of successive training rounds, with black segments denoting the compute window (inner-step training) and red segments denoting synchronization overhead. Despite training a

7.2\times

larger model, Covenant-72B incurs only 70 s of idle time per round, compared to the 8.3 min per-round synchronization overhead reported for DiLoCo-style training in INTELLECT-1.

Benchmarks
	LLaMA-2-7B-Chat	LLaMA-2-70B-Chat	K2-Chat (65B)	Covenant-72B-Chat
ARC-Challenge	53.2	65.4	62.0	64.2
ARC-Easy	80.6	85.3	85.8	85.5
GSM8K	22.6	52.2	79.0	63.9
HellaSwag	78.6	85.9	79.3	79.2
MMLU	47.2	63.1	67.9	67.4

Table 2: Benchmark results on chat models. Values are accuracy (%) with one decimal. We use 25-shot for ARC-Challenge, 10-shot for HellaSwag, 3-shot for BBH CoT, and 4-shot for MATH; all remaining benchmarks use 5-shot. Metrics are acc_norm where available (except MMLU and WinoGrande acc , and GSM8K strict ); additional benchmarks use exact_match (BBH CoT, MATH, MMLU-Pro), prompt_strict (IFEval), and acc_norm (MuSR).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Active and contributing peers over training. Active peers (red) are registered on the network and actively submitting pseudo-gradients; contributing peers (black) denote the number of peers whose pseudo-gradients are selected for aggregation each round. In our permissionless setting, not all submissions are selected (e.g., due to failing validation checks or low-quality pseudo-gradients).

	Pre-Anneal	Post-Anneal
ARC-C	56.4	56.8
ARC-E	81.2	80.9
PIQA	82.2	81.6
OpenBookQA	44.4	44.0
HellaSwag	81.4	80.6
WinoGrande	75.7	75.9

Appendix

Table 3: Base model performance before and after annealing. Zero-shot accuracy on the same benchmarks as Table 1 . The pre-anneal checkpoint corresponds to step

6{,}100

(

{\sim}1.09

T tokens) and the post-anneal checkpoint to step

6{,}190

(

{\sim}1.1

T tokens).

Field	Value
Parameters	72,747,327,488
Layers	80
Model width	8192
Query heads	64
KV heads	8
RoPE ( $\theta$ )	$500{,}000$

Appendix

Table 4: Model configuration for Covenant-72B .

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 441 papers

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead
Oct 23, 2026Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat +27LLM Training Distributed Training
The Ohio State University · University of Wisconsin–Madison · Rochester Institute of Technology +5
SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMs
Feb 20, 2026Jihun Kim, Dongyeop Lee, Namhoon LeeCommunication-Efficient Distributed Training Zeroth-Order Optimization
POSTECH, pohang, Republic of Korea.
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Jul 14, 2026Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9Communication-Efficient Distributed Training Fault-Tolerant Distributed Training
Pluralis Research

Oct 23, 2026J/K move · Enter open · S save

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat, Srijith Nair+26

The Ohio State University · University of Wisconsin–Madison · Rochester Institute of Technology+5

LLM Training Distributed Training Federated Learning Communication-Efficient Distributed Training Fault-Tolerant Distributed Training

Open paper AskPDF arXiv

The growing computational power of modern GPU data centers has let a few industrial giants train and deploy large language models (LLMs) on massive amounts of data. Yet the sheer scale of these models and their training data makes centralized development increasingly hard to sustain, and at the same time shuts out smaller institutions that hold relevant private data but lack comparable infrastructure. In this paper, we organize our positions on distributed LLM training around the pretraining and fine-tuning paradigms, covering federated learning, split learning, and decentralized approaches, and we assess their readiness for geo-distributed settings with heterogeneous hardware and unreliable connectivity. We then present a {\em unified} system architecture built around a cloud-hosted parameter server that coordinates heterogeneous clients over outbound gRPC connections, reusing one communication, compression, and fault-tolerance stack for both pretraining and a family of fine-tuning methods. We validate the design with real geo-distributed experiments on GPT-2 Medium and Llama3-1B pretraining across multiple university sites, showing stable convergence under realistic conditions. Finally, we chart the road ahead, identifying open problems that span frontier model support, elastic fault tolerance, adaptive compression, federated alignment, privacy and trust, incentive design, and system--algorithm co-design, all of which must be addressed to make democratized LLM training a reality.

Abstract

Language Model Pretraining

Explore similar work

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMs

Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models