Distributed LLM Pretraining During Renewable Curtailment Windows: A Feasibility Study

Philipp Wiesner; Soeren Becker; Brett Cornick; Dominik Scheinert; Alexander Acker; Odej Kao

cs.DCFeb 26, 2026

Distributed LLM Pretraining During Renewable Curtailment Windows: A Feasibility Study

Authors: Philipp Wiesner, Soeren Becker, Brett Cornick, Dominik Scheinert, Alexander Acker, Odej Kao

Organizations: Exalsius · TU Berlin · Deep Science Ventures

Abstract

Training large language models (LLMs) requires substantial compute and energy. At the same time, renewable energy sources regularly produce more electricity than the grid can absorb, leading to curtailment, the deliberate reduction of clean generation that would otherwise go to waste. These periods represent an opportunity: if training is aligned with curtailment windows, LLMs can be pretrained using electricity that is both clean and cheap. This technical report presents a system that performs full-parameter LLM training across geo-distributed GPU clusters during regional curtailment windows, elastically switching between local single-site training and federated multi-site synchronization as sites become available or unavailable. Our prototype trains a 561M-parameter transformer model across three clusters using the Flower federated learning framework, with curtailment periods derived from real-world marginal carbon intensity traces. Preliminary results show that curtailment-aware scheduling preserves training quality while reducing operational emissions to 5-12% of single-site baselines.

Figures & tables

Figure 1: Sites train only during curtailment windows (green), when renewable generation exceeds demand. If multiple sites are curtailed simultaneously, they train locally in parallel and periodically average model states.

Figure 2: Execution timeline of curtailment-aware pretraining, showing how training shifts across renewable curtailment windows in California, Texas, and South Australia.

Figure 3: Curtailment-aware training reaches comparable perplexity as centralized training.

Scenario	Runtime (hours)	Best train perplexity	Energy (kWh)
Centralized	17.8	14.7	36.0
2-Site FL	11.1	15.2	36.1
Ours	14.6	14.6	37.7

Table 1: Overhead from dynamic site provisioning slightly increases energy consumption.

Figure 4: Fraction of training energy drawn during curtailment windows.

Energy-Efficient ML

Energy-efficient machine learning reduces the energy and environmental costs of training and inference while preserving useful model performance.

See all 257 papers

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead
Oct 23, 2026Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat +27LLM Training Distributed Training
The Ohio State University · University of Wisconsin–Madison · Rochester Institute of Technology +5
DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
May 12, 2026Yuanqing Wang, Yuchen Zhang, Hao Lin +9Efficient Language Model Training Distributed Training
Peking University · Infinigence AI · Institute of Computing Technology, CAS +2
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
May 18, 2026Aaron DefazioLanguage Model Pretraining Learning Rate Scheduling
FAIR at Meta Super-Intelligence Labs

Oct 23, 2026J/K move · Enter open · S save

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat, Srijith Nair+26

The Ohio State University · University of Wisconsin–Madison · Rochester Institute of Technology+5

LLM Training Distributed Training Federated Learning Communication-Efficient Distributed Training Fault-Tolerant Distributed Training

Open paper AskPDF arXiv

The growing computational power of modern GPU data centers has let a few industrial giants train and deploy large language models (LLMs) on massive amounts of data. Yet the sheer scale of these models and their training data makes centralized development increasingly hard to sustain, and at the same time shuts out smaller institutions that hold relevant private data but lack comparable infrastructure. In this paper, we organize our positions on distributed LLM training around the pretraining and fine-tuning paradigms, covering federated learning, split learning, and decentralized approaches, and we assess their readiness for geo-distributed settings with heterogeneous hardware and unreliable connectivity. We then present a {\em unified} system architecture built around a cloud-hosted parameter server that coordinates heterogeneous clients over outbound gRPC connections, reusing one communication, compression, and fault-tolerance stack for both pretraining and a family of fine-tuning methods. We validate the design with real geo-distributed experiments on GPT-2 Medium and Llama3-1B pretraining across multiple university sites, showing stable convergence under realistic conditions. Finally, we chart the road ahead, identifying open problems that span frontier model support, elastic fault tolerance, adaptive compression, federated alignment, privacy and trust, incentive design, and system--algorithm co-design, all of which must be addressed to make democratized LLM training a reality.

Abstract

Energy-Efficient ML

Explore similar work

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models