Model Pretraining

Follow this topic Ask about this topic

Recent momentum

-38%

31 papers in the last 28 days · 0.5% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

12 new papers

A weekly snapshot of new work published in Model Pretraining.

Period ending 2026-09-14

11 new papers

A weekly snapshot of new work published in Model Pretraining.

Period ending 2026-09-07

13 new papers

A weekly snapshot of new work published in Model Pretraining.

402 papers

Latest in Model Pretraining

Browse all topics →

Life Cycle Assessment of Pre-training the Lucie 7B Open-Source Large Language Model on the Jean Zay Supercomputer
Jun 9, 2026Marc Léobet, Pierre-François Lavallée, Jean-Pierre LorréLarge Language High-Performance Computing
LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models
Jun 8, 2026Mingqi Yuan, Xiaoquan Sun, Shihao Luo +1Efficient Continual Learning Unseen-Task Adaptation
Multi-Hop Knowledge Composition is Bound by Pretraining Exposure
Jun 8, 2026Yannis Karmim, Luis Marti, Djamé Seddah +1Multi-Hop Reasoning Model Pretraining
Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention
Jun 8, 2026George Theodosiou, Loukas Ilias, Dimitris AskounisParkinson Discrete Speech Representations
Pretrained, Frozen, Still Leaking: Auditing Cross-Encoder Attribute Transfer in EEG Foundation Models
Jun 8, 2026Jianwei TaiElectroencephalography Foundation Models Leakage
The Spectral Dynamics and Noise Geometry of Muon
Jun 7, 2026Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso PoggioMuon Spectrum
How Much MRI Preprocessing Is Enough? A Cost-Utility Study for Brain MRI Foundation Models
Jun 6, 2026Jiangshuan Pang, Wangyang Tang, Jing Yan +5Brain Magnetic Resonance Imaging Model Pretraining
Mix, Don't Pick: Why Synthetic Corpus Composition Matters for Time Series Foundation Model Pretraining
Jun 6, 2026Aaryan Nagpal, Debdeep Sanyal, Murari Mandal +2Time Series Foundation Models Model Pretraining
MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models
Jun 6, 2026Kaixin Lan, Mu You, Tao Fang +3Large Language Model Pretraining Model Pretraining
GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution
Jun 5, 2026Yue Min, Ruining Chen, Yujun LiTraining Data Attribution Model Pretraining
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Jun 5, 2026Zhiwei Xu, Shihao Wu, Hanseul Cho +2Large Language Model Pretraining Model Pretraining
ActiveMimic: Egocentric Video Pretraining with Active Perception
Jun 4, 2026Xingyao Lin, Guojin Zhong, Tianyi Lu +4Egocentric Video Video Pretraining
Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails
Jun 4, 2026Marco Antonio Stranisci, A Pranav, Rossana Damiano +2Large Language Model Bias Large Language Model Safety
Causal Longitudinal Prior-Fitted Networks for Counterfactual Outcome Prediction
Jun 4, 2026Amirhossein Zare, Amirhessam Zare, Herlock Rahimi +2Causal Inferences Counterfactual Reasoning
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
Jun 4, 2026Yongwei Zhou, Juncheng Diao, Junlin Shang +2Hyperparameter Optimization Model Pretraining
TopoPult-SSL: Gland-Mask-Free Cross-Device Meibomian Gland Segmentation via Self-Distilled Weak Clinical Priors
Jun 3, 2026Nicolò Savioli, Luca Del TongoMedical Image Segmentation One-Shot Medical Landmark Detection
Towards Pretraining Text Encoders for TabPFN
Jun 3, 2026Mustafa Tajjar, Alexander Pfefferle, Lennart Purucker +1Tabular Foundation Models Tabular Prior-Data Fitted Network
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Jun 2, 2026Rachit Bansal, Clara Mohri, Tian Qin +2Model Pretraining Deep Reinforcement Learning
ADAPTOOD: Uncertainty-Aware Fine-Tuning for Out-of-Distribution ECG Time Series Models
Jun 2, 2026Sotirios Vavaroutas, Yu Yvonne Wu, Ali Etemad +1Out-Of-Distribution Distribution Shifts
q0: Primitives for Hyper-Epoch Pretraining
Jun 2, 2026Bishwas Mandal, Shmuel Berman, Akshay Vegesna +1Best-Epoch Metrics Model Pretraining
KletterMix: Climbing Toward High-Quality German Pretraining Data
Jun 2, 2026Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia +4Large Language Model Pretraining Text Corpora
Speedrunning Tabular Foundation Model Pretraining
Jun 2, 2026Salih Bora Ozturk, Alexander Pfefferle, Frank HutterTabular Foundation Models Model Pretraining
The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP
Jun 2, 2026Henry He, Johann Frei, Raphael SchmittBert-Based Models Multilingual Language Models
Provable Data Scaling Law for Meta Learning via Complexity Minimization
Jun 1, 2026Kazuto Fukuchi, Ryuichiro Hataya, Kota MatsuiMeta-Learning Model Pretraining
Parameter-Efficient Fine-Tuning of Large Pretrained Models for Instance Segmentation Tasks
Jun 1, 2026Nermeen Abou Baker, David Rohrschneider, Uwe HandmannParameter-Efficient Fine-Tuning Low-Rank Adaptation
Segment-driven Structural Induction and Semantic Alignment for Heterogeneous Tabular Representation
Jun 1, 2026Woojun Jung, Susik YoonModel Pretraining Key-Value
Feature to Dynamics: Feature-space to Autoregression strategy for Zero-shot Time Series Forecasting
May 31, 2026Yifan Wu, Junjie Wu, Kai Wu +2Zero-Shot Forecasting Zero-Shot Inference
CryoProt: A Protein Pretraining Framework with Cross-Box Interactions on Cryo-EM Density Maps
May 31, 2026Dan Luo, Xuan Lin, Peng Zhou +4Cryo-Electron Microscopy Protein Data Bank
OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models
May 30, 2026Ziling Lu, Zongsheng Li, Xinke Shen +11Electroencephalography Foundation Models Evaluation Benchmarks
Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
May 29, 2026Sanchit Ahuja, Terra BlevinsMultilingual Language Models Catastrophic Forgetting
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
May 29, 2026Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande +1Large Language Model Pretraining Model Pretraining
A Protocol-Language Model for Network Intrusion (Without Deep Packet Inspection)
May 29, 2026Vivek Kumar SharmaIntrusion Detection Singguard-Nsfa
Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them
May 29, 2026Kevin Zhou, Lisa Alazraki, Kris Cao +1Repetition Mixtures
ParaTool: Shifting Tool Representations from Context to Parameters
May 28, 2026Zekai Yu, Qi Meng, Qizhi Chu +3Tool Invocation Multi-Turn Tool-Use Agent
Continual Model Routing in Evolving Model Hubs
May 27, 2026Jack Bell, Giacomo Carfì, Gerlando Gramaglia +1Efficient Continual Learning Independent Anchor Network
Efficient Pre-Training of LLMs through Truncated SVD Layers
May 27, 2026Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad +3Singular Value Decomposition Large Language Model Pretraining
Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer
May 27, 2026Jungyong Son, Jinwook Jung, Minhee Park +1Transfer Learning Model Pretraining
Chreode: A Cell World Model for One-Step Temporal Dynamics and Perturbation Prediction
May 27, 2026Mufan Qiu, Genhui Zheng, Yinuo Xu +4Temporal Dynamics Model Pretraining
The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates
May 27, 2026Shaobo Wang, Guo Chen, Ziyue Wang +5Large Language Model Decoder Process Reward Modeling
Black-box Membership Inference Attacks on the Pre-training Data of Image-generation Models
May 26, 2026Tao Qi, Huili Wang, Yuanhong Huang +6Membership Inference Attacks Model Pretraining
Unicorn: Scaling High-Dimensional Time Series Forecasting via Universal Correlation Modeling
May 26, 2026Haochen Yuan, Yichen Song, Yunbo Wang +1Multivariate Forecasting Time Series
Time Series Causal Discovery via Context-Conditioned and Causality-Augmented Pretraining
May 26, 2026Biao Ouyang, Tengxue Zhang, Zhihao Zhuang +3Causal Foundation Models Temporal Dependence
The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training
May 26, 2026Hongtao Zhang, Wenjie Zhou, Chenxi Jia +2Large Language Model Pretraining Learning Rates
On the Role of Inductive Bias in Time-Series Pretraining: A Case Study in Learning Generalizable Representations for Clinical Time Series
May 25, 2026Sharmita Dey, Diego Paez-GranadosTime-Series Classification Model Pretraining
MDGMIX: Boundary-Aware Subgraph Mixing for Multi-Domain Graph Pre-Training
May 25, 2026Ziyu Zheng, Yaming Yang, Ziyu Guan +2Subgraphs Model Pretraining
TriDP-PTM: a three-stage distortion-perception tradeoff guides the pre-training model for radar cardiac sensing
May 25, 2026Jinye Li, Aidong Men, Yang Liu +1Distortion Model Pretraining
PowLU: An Activation Function for Stable Pre-Training of LLMs
May 25, 2026Peijie Jiang, Yuqi Feng, Cunyin Peng +5Activation Functions Model Pretraining
Certified Robustness from Approximate Gaussian Mixture Structures in Pretrained Latent Spaces
May 25, 2026Konstantinos Emmanouilidis, Tianjiao Ding, Nghia Nguyen +2Adversarial Transferability Gaussian Mixture Models
Temporal Concept Drift in Legal Judgment Prediction: Neural Baselines Across Three Epochs of Ukrainian Court Decisions
May 23, 2026Volodymyr OvcharovPredictive Law Legal Natural Language Processing Benchmarks
Plume Segmentation from MethaneSAT with Cross-Sensor Transfer Learning and Physics-Informed Postprocessing
May 22, 2026Manuel Pérez-Carrasco, Maya Nasr, Zhan Zhang +16Cross-Sensor Generalization Model Pretraining
An Open-Source Training Dataset for Foundation Models for Black-box Optimization
May 22, 2026Aaron Klein, Herilalaina Rakotoarison, Luca Thale-Bombien +1Black-Box Optimization Benchmark Datasets
An accurate nucleic acid-small molecule docking framework via geometric deep learning with large-scale pretraining
May 22, 2026Shi Li, Xujun Zhang, Mingquan Liu +5Structure-Based Drug Design Drug Design
RelPrism: A Multi-Faceted Pre-training Framework with Self-Generated Tasks for Relational Databases
May 22, 2026Jinyu Yang, Cheng Yang, Junze Chen +4Relational Deep Learning Relational
Understanding Data Temporality Impact on Large Language Models Pre-training
May 21, 2026Hippolyte Pilchen, Romain Fabre, Franck Signe Talla +2Large Language Model Pretraining Model Pretraining
Translating Signals to Languages for sEMG-Based Activity Recognition
May 21, 2026Ming Wang, Haoxuan Qu, Qiuhong Ke +3Surface Electromyography Human Activity Recognition
Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications
May 21, 2026Ziyi Tong, Feifei Sun, Le Minh NguyenMembership Inference Large Language Model Pretraining
Universal CT Representations from Anatomy to Disease Phenotype through Agglomerative Pretraining
May 21, 2026Yuheng Li, Yuan Gao, Haoyu Dong +5Computed Tomography Dataset Anatomy
SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Encoder Spike Sparsity
May 20, 2026Ting LiuSparse Expert Activation Binary Spikes
SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining
May 20, 2026Nassim Ait Ali Braham, Aaron Banze, Conrad M. Albrecht +3Hyperspectral Imaging Earth Observation
Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers
May 20, 2026Nikhil Nayak, Julia White, Urchade Zaratiana +7Large Language Model Pretraining Spectral Preconditioning

Jun 9, 2026 · cs.CYJ/K move · Enter open · S save

Life Cycle Assessment of Pre-training the Lucie 7B Open-Source Large Language Model on the Jean Zay Supercomputer

Marc Léobet, Pierre-François Lavallée, Jean-Pierre Lorré

Mens Data · CNRS / IDRIS · LINAGORA

The environmental impact of training large language models (LLMs) is increasingly scrutinised, yet most published estimates focus on operational energy and disclose little about manufacturing (embodied) emissions, water consumption, or the underlying highperformance computing (HPC) infrastructure. We present a life cycle assessment (LCA) of the pre-training of Lucie 7B, an open-source multilingual Foundation Model developed by the OpenLLM-France consortium and trained on the NVIDIA H100 partition of the Jean Zay supercomputer operated by IDRIS (CNRS). The assessment is framed by the AFNOR SPEC 2314 "Frugal AI" reference and applies the Labos 1point5 methodology for greenhouse gas(GHG) accounting in computing. The study scope extends from data preparation to model validation, and integrates the full life cycle of the hardware infrastructure: manufacturing (including raw-material extraction), use (compute, temporary storage, system administration, cooling), and end-of-life. We report (i) an annual footprint of 417.5 tCO2eq for the Jean Zay H100 partition, split almost equally between manufacturing and operation; (ii) an effective intensity of 36.7 gCO2eq per H100 GPU-hour; (iii) a total training footprint of 21 tCO2eq for Lucie 7B (574 564 H100 GPU-hours), inclusive of amortised hardware manufacturing; (iv) on-site water consumption of approximately 76m3 for the training campaign and an annual Water Usage Effectiveness (WUE) of 0.07 L/kWh for IDRIS; (v) a heat-reuse factor (ERF) of 0.37 thanks to waste-heat recovery into the urban heating network. The study contributes one of the few publicly documented LCAs of an LLM training campaign that explicitly couples operational data with embodied emissions decomposed by subsystem (compute, storage, power chain, cooling), and discusses the implications for the design of frugal-by-construction AI systems in Europe.

Large Language High-Performance Computing Lifecycle Model Pretraining Artificial Intelligence Systems

Open paper Ask about it PDF