Continued Pretraining

Follow this topic Ask about this topic

Continued pretraining studies how to adapt an already pretrained model to new domains, languages, tasks, or longer contexts without starting from scratch. Common approaches train further with next-token prediction or masked-language modeling on curated or synthetic data, sometimes using adapters, selective layer updates, replay, or checkpoint merging. Researchers evaluate both target-domain performance and retention of general capabilities across language, code, speech, vision, and scientific or medical benchmarks. The central tension is how to gain useful new knowledge efficiently while avoiding catastrophic forgetting; data quality, training order, update strategy, and evaluation choices can matter as much as training volume.

Latest papers

56 papers15 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir Language
Oct 22, 2026Mullosharaf K. Arabov, Svetlana S. KhaybullinaLow-Resource Language Processing Parameter-Efficient Fine-Tuning
Department of Data Analysis and Programming Technologies, Institute of Computational Mathematics and Information Technologies, Kazan (Volga Region) Federal University, Kazan, Russia
Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law
Oct 21, 2026Ziqing Qiao, Xiaoyue Xu, Sikui Zhang +4Long-Context Language Modeling Continued Pretraining
Department of Computer Science and Technology, Tsinghua University, Beijing, China · OpenBMB
Data Schedules and Learning Rate Schedules Should Be Designed Together
Oct 21, 2026Haodong Wen, Kairong Luo, Tingkai Yan +2LLM Training Continued Pretraining
Tsinghua University · Peng Cheng Laboratory
How to Use Parallel Data for Multilingual Pretraining
Oct 20, 2026Zhihong Huang, Yao Lu, Derek F. Wong +2Language Model Pretraining Multilingual Language Models
University of Macau · University College London · UNSW Sydney +1
Scaling in Games: Continued Pre-Training For Embodied Agents In Diverse Virtual Worlds
Oct 20, 2026Kuan Zhang, Yukun Chen, Zhihao Yang +8Embodied AI Continued Pretraining
miHoYo Honkai AI R&D Team · College AI, Tsinghua University · University of Chinese Academy of Sciences +3
Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining
Oct 19, 2026Xinyi Wang, Danqi ChenRL for Language Model Reasoning Language Model Pretraining
Princeton Language and Intelligence
The Ephemeral Cost of Switching Optimizers Across LLM Training Stages
Oct 15, 2026Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma +2LLM Training LLM Post-Training
MIT · MIT-IBM Watson AI Lab
Continual pretraining Prostate MRI ViT Encoder for task alignment
Oct 14, 2026Bilal A. Sidiqi, Yipei Wang, Shaheer U. Saeed +3Medical Imaging Foundation Models Continued Pretraining
Department of Medical Physics and Biomedical Engineering, University College London, London, UK · Hawkes Institute, University College London, London, UK · Department of Computer Science, University College London, London, UK
Beyond the greenwash: do LLMs' ecologically sound decisions hold up under pressure?
Oct 12, 2026Trent R. Northen, Mingxun WangLLM Alignment Language Model Safety Evaluation
Bioaligned Labs
Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency
Oct 9, 2026Sheng Zhao, Yuhao ZhuContinual Learning for LLMs Continued Pretraining
Department of Computer Science University of Rochester
Sequential Pretraining Favors Large Models
Oct 7, 2026Mohnish Harwani, Yujia ZhengLanguage Model Pretraining Continued Pretraining
Purdue University · University of Illinois Urbana–Champaign
Task Vector Descent: Learning from Non-IID Batches
Oct 4, 2026Anton Baumann, Jonas Hübotter, Zeynep Akata +1Continual Learning for LLMs Task Arithmetic
ETH Zurich · Technical University of Munich
Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models
Oct 1, 2026Zhenghao Zhao, Gaowen Liu, Zhiling Lan +1Continued Pretraining Catastrophic Forgetting
University of Illinois Chicago · Cisco Research
Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining
Sep 30, 2026Lukas Thede, Shengzhuang Chen, Stefan Winzeck +3Continual Learning for LLMs Replay-Based Continual Learning
University of Tübingen, Tübingen AI Center · Helmholtz Munich · Munich Center for Machine Learning (MCML) +3
What Pretraining and Midtraining Make Learnable from Rewards?
Sep 29, 2026Chiwun Yang, Xiaoyu LiRL for Language Model Reasoning Continued Pretraining
City University of Hong Kong · University of New South Wales
Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
Sep 20, 2026Fırat Öncel, Salman Hussain Ali, Mirco Ravanelli +2Continual Learning for LLMs Factual Knowledge in Language Models
Concordia University · Mila – Quebec AI Institute · Université de Montréal +3
ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI
Sep 13, 2026Sai Krishna Reddy Sathi, Anuj TiwariEmbodied AI Embodied Reasoning
Indian Institute of Technology Madras, India
Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
Sep 7, 2026Yongan Yu, Shantam Raj, Jingwei Ni +3Multi-Source Domain Adaptation Domain-Adaptive Pretraining
McGill University · Mila – Quebec Artificial Intelligence Institute · University of Zürich +2
DKL: Decoupled Knowledge Learning for Instruction-Tuned Language Models
Sep 2, 2026Kushagra Bhushan, Meghanadh Pulivarthi, Sai Krishna Reddy Sathi +7Knowledge Augmentation for Language Models Retrieval-Augmented Generation
1IBM · 2Indian Institute of Technology, Madras
KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training
Aug 31, 2026Meghanadh Pulivarthi, Kushagra Bhushan, Vineet Kumar +5Language Model Pretraining Knowledge Augmentation for Language Models
IBM · Work done while at IBM. Currently at Amazon Books
Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation
Aug 31, 2026Ran Zhang, Miryam de Lhoneux, Wessel PoelmanContinued Pretraining Low-Resource Language Processing
LAGOM·NLP, Department of Computer Science, KU Leuven
Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
Aug 26, 2026Qiankai Xu, Qiguang Chen, Zixin Su +4Long-Form Document Generation Synthetic Benchmark Generation
ByteDance Seed · Nanjing University · Evolvent AI
When to Review: Spaced Repetition for Continual Pre-Training of Language Models
Aug 18, 2026Alankar Atreya, Devesh Batra, Yoages Kumar Mantri +3Continual Learning for LLMs Continued Pretraining
NatWest AI Research · University College London
Invariant Pretraining for Robust Code Representations
Aug 15, 2026Yifeng He, Yundi Xu, Christopher Castro Gaw Gonzalo +2Representation Learning Invariant Representation Learning
University of California at Davis, Davis, USA · The University of Hong Kong, China
Diffract: Spectral View of LLM Domain Adaptation
Aug 11, 2026Nikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain Adaptation Domain-Adaptive Pretraining
Risk AI Research Lab · Applied AI Institute
AgriField-40K: Adapting Vision Models to Agriculture With Efficient Continual Pretraining
Aug 8, 2026Vasileios Tzouras, Paraskevas Pegios, Lazaros NalpantidisAgricultural Image Analysis Masked Autoencoders
Technical University of Denmark (DTU), Kongens Lyngby, Denmark · Pioneer Centre for Artificial Intelligence, Copenhagen, Denmark
Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustness
Aug 6, 2026Aarohi Srivastava, David ChiangLanguage Model Pretraining Multilingual Language Models
Computer Science and Engineering University of Notre Dame Notre Dame, IN, USA
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
Aug 5, 2026Indraneil Paul, Falko Helm, Goran Glavaš +1Software Engineering Long-Context Language Modeling
UKP Lab & Center for Applied Cybersecurity ATHENE, TU Darmstadt · CAIDAS, University of Würzburg
Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
Aug 4, 2026Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan +1Efficient Language Model Training Continued Pretraining
NVIDIA
Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling
Aug 3, 2026Ravi Satya Durga Prasad YenugulaLLM Compression Continued Pretraining
Independent Researcher

Oct 22, 2026J/K move · Enter open · S save

Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir Language

Mullosharaf K. Arabov, Svetlana S. Khaybullina

Department of Data Analysis and Programming Technologies, Institute of Computational Mathematics and Information Technologies, Kazan (Volga Region) Federal University, Kazan, Russia

Low-Resource Language Processing Parameter-Efficient Fine-Tuning Domain Adaptation for LLMs Continued Pretraining

Open paper AskPDF arXiv

Bashkir is a low-resource agglutinative language written in Cyrillic and remains poorly represented in modern LLMs. We present an open resource suite and a systematic benchmark of adaptation methods: the Bashkir Web Corpus (71,567 documents, 46.9M tokens, 16 sources) and three labelled datasets for binary, multiclass and multilabel text classification; the benchmark covers the first two. Experiments use eight architectures (82M-8B: DistilGPT2, GPT-2 base/medium, Phi-3-mini, Mistral-7B, YandexGPT-5-Lite-8B, mBERT, XLM-R) and three adaptation groups (full fine-tuning, QLoRA, in-context learning), plus continual pretraining (CPT). (1) QLoRA on 3.8-8B models outperforms encoder-only fine-tuning on binary classification: YandexGPT-5-Lite-8B with CPT reaches macro-F1 0.748 vs 0.653 (XLM-R) and 0.650 (mBERT); pairwise differences are significant after Holm correction (p<0.001). (2) YandexGPT-5-Lite-8B significantly outperforms Mistral-7B and Phi-3-mini on multiclass classification, BPB and native-speaker ratings (mean 3.28 vs 2.52 and 2.22; n=100; alpha=0.69-0.77; p<0.05); on binary classification its QLoRA advantage is not significant (p>=0.084). (3) CPT effects depend on model and task: perplexity improves for Phi-3-mini (+0.58) and YandexGPT-5-Lite-8B (+0.33) but worsens for Mistral-7B (-1.15, p<0.001); on binary classification CPT helps Mistral-7B most (+4.0 macro-F1). (4) Tokeniser fertility on Bashkir varies 2.67x (2.92-7.80) and tracks adaptation quality, while perplexity across tokenisers is not comparable. Cyrillic tokenisers with Russian pretraining are preferable; LLM judges reproduce native ratings for grammar and coherence (rho=0.33-0.46) but not language correctness (rho=0.16, p=0.11); native evaluation remains necessary. All artefacts are open-licensed.