Continued Pretraining

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 33

All topics

Sequential Pretraining Favors Large Models
Oct 7, 2026Mohnish Harwani, Yujia ZhengLanguage Model Pretraining Continued Pretraining
Purdue University · University of Illinois Urbana–Champaign
Task Vector Descent: Learning from Non-IID Batches
Oct 4, 2026Anton Baumann, Jonas Hübotter, Zeynep Akata +1Continual Learning for LLMs Task Arithmetic
ETH Zurich · Technical University of Munich
Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining
Sep 30, 2026Lukas Thede, Shengzhuang Chen, Stefan Winzeck +3Continual Learning for LLMs Replay-Based Continual Learning
University of Tübingen, Tübingen AI Center · Helmholtz Munich · Munich Center for Machine Learning (MCML) +3
What Pretraining and Midtraining Make Learnable from Rewards?
Sep 29, 2026Chiwun Yang, Xiaoyu LiRL for Language Model Reasoning Continued Pretraining
City University of Hong Kong · University of New South Wales
Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
Sep 20, 2026Fırat Öncel, Salman Hussain Ali, Mirco Ravanelli +2Continual Learning for LLMs Factual Knowledge in Language Models
Concordia University · Mila – Quebec AI Institute · Université de Montréal +3
Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
Sep 7, 2026Yongan Yu, Shantam Raj, Jingwei Ni +3Multi-Source Domain Adaptation Domain-Adaptive Pretraining
McGill University · Mila – Quebec Artificial Intelligence Institute · University of Zürich +2
DKL: Decoupled Knowledge Learning for Instruction-Tuned Language Models
Sep 2, 2026Kushagra Bhushan, Meghanadh Pulivarthi, Sai Krishna Reddy Sathi +7Knowledge Augmentation for Language Models Retrieval-Augmented Generation
1IBM · 2Indian Institute of Technology, Madras
KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training
Aug 31, 2026Meghanadh Pulivarthi, Kushagra Bhushan, Vineet Kumar +5Language Model Pretraining Knowledge Augmentation for Language Models
IBM · Work done while at IBM. Currently at Amazon Books
Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation
Aug 31, 2026Ran Zhang, Miryam de Lhoneux, Wessel PoelmanContinued Pretraining Low-Resource Language Processing
LAGOM·NLP, Department of Computer Science, KU Leuven
Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
Aug 26, 2026Qiankai Xu, Qiguang Chen, Zixin Su +4Long-Form Document Generation Synthetic Benchmark Generation
ByteDance Seed · Nanjing University · Evolvent AI
AgriField-40K: Adapting Vision Models to Agriculture With Efficient Continual Pretraining
Aug 8, 2026Vasileios Tzouras, Paraskevas Pegios, Lazaros NalpantidisAgricultural Image Analysis Masked Autoencoders
Technical University of Denmark (DTU), Kongens Lyngby, Denmark · Pioneer Centre for Artificial Intelligence, Copenhagen, Denmark
Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustness
Aug 6, 2026Aarohi Srivastava, David ChiangLanguage Model Pretraining Multilingual Language Models
Computer Science and Engineering University of Notre Dame Notre Dame, IN, USA
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
Aug 5, 2026Indraneil Paul, Falko Helm, Goran Glavaš +1Software Engineering Long-Context Language Modeling
UKP Lab & Center for Applied Cybersecurity ATHENE, TU Darmstadt · CAIDAS, University of Würzburg
Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
Aug 4, 2026Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan +1Efficient Language Model Training Continued Pretraining
NVIDIA
Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling
Aug 3, 2026Ravi Satya Durga Prasad YenugulaLLM Compression Continued Pretraining
Independent Researcher
Two-Stage Bengali Sentiment Classification: Domain Adaptation Through Continual Learning and Parameter-Efficient Fine-Tuning
Aug 2, 2026MD Shaikh Rahman, Syed Maudud E Rabbi, Muhammad Mahbubur RashidSentiment Analysis Domain-Adaptive Pretraining
Department of Computer Science Universiti Sains Malaysia Pulau Pinang, Malaysia · Department of Computer Science Baruch College New York New York, USA · Department of Mechatronics Engineering International Islamic University Malaysia Kuala Lumpur, Malaysia
Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT
Jul 27, 2026Cen Lu, Yung-Chen Tang, Andrea CavallaroLanguage Model Pretraining LLM Safety Alignment
1EPFL · 2Idiap Research Institute
From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognition
Jul 7, 2026Lukmal Ilyas, Nevidu JayatillekeContinued Pretraining Low-Resource Speech Recognition
School of Computing, Informatics Institute of Technology, Sri Lanka · Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
Jun 4, 2026Yongwei Zhou, Juncheng Diao, Junlin Shang +2Language Model Scaling Laws Hyperparameter Transfer
MeiTuan · University of Chinese Academy of Sciences · Harbin Institute of Technology
Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining
May 31, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +10Vision-Language Models Multimodal Pretraining
The Hong Kong Polytechnic University · Sun Yat-sen University · PolyU-Daya Bay Technology and Innovation Research Institute
Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT
May 25, 2026Duy Anh NguyenEducational Assessment Domain-Adaptive Pretraining
University of Greenwich, London, United Kingdom · FPT University, Can Tho, Vietnam
Universal CT Representations from Anatomy to Disease Phenotype through Agglomerative Pretraining
May 21, 2026Yuheng Li, Yuan Gao, Haoyu Dong +5Medical Imaging Foundation Models Medical VLMs
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA 30332, USA · Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC 27705, USA +2
TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale
May 14, 2026Anurup GanguliContinual Learning for LLMs Continued Pretraining
Independent Researcher
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
May 12, 2026Rian Touchent, Eric de la ClergerieLanguage Model Pretraining Masked Language Modeling
Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris
Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training
May 12, 2026Yu-Hang Wu, Qin-Yuan Liu, Qiu-Yang Zhao +3Fine-Tuning Continued Pretraining
Nanhu Research Institute of China Electronic Science and Technology · School of Electronic and Electrical Engineering, Shanghai University of Engineering Science
Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm
May 11, 2026Haoyu Wang, Yifan Shang, Zhongxiang Sun +3Continual Learning for LLMs Generative Replay
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Artificial Intelligence and Data Science, University of International Business and Economics, Beijing, China
Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing
May 11, 2026Jinchang Zhu, Jindong Li, Chengyu Zou +4Long-Context Language Modeling Adaptive Loss Weighting
1The Hong Kong University of Science and Technology (Guangzhou) · Institute of Artificial Intelligence (TeleAI), China Telecom
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST
Prototype Guided Post-pretraining for Single-Cell Representation Learning
May 8, 2026Sachini Weerasekara, Natasha Darras, Sagar Kamarthi +2Representation Learning Continued Pretraining
Sachini Weerasekara∗ · Natasha Darras · Sagar Kamarthi +2
Teaching LLMs Program Semantics via Symbolic Execution Traces
May 7, 2026Jonas Bayer, Stefan Zetzsche, Olivier Bouissou +3LLM Evaluation Continued Pretraining
University of Cambridge · Amazon Web Services

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

Sequential Pretraining Favors Large Models

Mohnish Harwani, Yujia Zheng

Purdue University · University of Illinois Urbana–Champaign

Language Model Pretraining Continued Pretraining

Open paper AskPDF arXiv

Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.

Momentum

Related topics

Latest papers 33