Domain-Adaptive Pretraining

Ask about this topic

Overview

Domain-adaptive pretraining studies how to specialize general-purpose language and vision models by continuing self-supervised training on domain-specific text, images, or other data. Common approaches include masked-language or next-token training, self-supervised vision methods, and mid-training, often paired with curated corpora, adapted vocabularies, replay, or parameter-efficient tuning. Evaluations span medical, legal, scientific, and technical tasks such as question answering, classification, information extraction, and image mapping, alongside checks of general capabilities and cross-domain transfer. Results often favor data closely matched to the target task, but gains are not guaranteed: specialization can trade off against broader capabilities, and benchmark performance may not reflect transfer to new settings.

Latest papers

CardsList

44 papers4 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. Robust Marathi Sentiment Analysis using Domain-Adaptive Pre-training and Data Augmentation Approaches

    Oct 11, 2026Tanvi Somani, Aryan Babare, Samyak Bora +2Sentiment AnalysisLow-Resource Language Processing

    Pune Institute of Computer Technology (PICT), Pune, Maharashtra, India · L3Cube Labs, Pune · Indian Institute of Technology Madras, Chennai, Tamil Nadu, India

  2. Domain adaptation of Russian ModernBERT for long legal documents

    Oct 5, 2026I. Litvak, D. Gvozdetsky, F. Lashkin +7Language Model PretrainingLegal NLP

    Independent Researcher · National Research Nuclear University MEPhI · Moscow Center for Advanced Studies +1

  3. OncoNoteBERT: A Foundation Representation Model for Natural Language Processing of Real-World Outpatient Oncology Notes

    Oct 2, 2026Wuraola Oyewusi, Eliana Vasquez Osorio, Goran Nenadic +1Clinical NLPLanguage Modeling

    Division of Cancer Sciences, The University of Manchester, Manchester, UK · The Christie NHS Foundation Trust, Manchester, UK · Department of Computer Science, The University of Manchester, Manchester, UK

  4. QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code

    Sep 30, 2026Alexey Chernysh, Orkhan Ekhtibarov, Dmitry ZmitrovichLLM Fine-TuningDomain-Adaptive Pretraining

  5. Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining

    Sep 22, 2026Mudi Zhai, Ruihong Qiu, Qingyun Zeng +3KG ConstructionDomain-Adaptive Pretraining

    UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales, Sydney, NSW 2052, Australia · School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, QLD 4072, Australia · Microsoft Copilot Studio AI, Redmond, WA 98052, United States +2

  6. MiST: Mid-Training LLMs for Cybersecurity

    Sep 16, 2026Oded Ovadia, Elad Ben Zaken, Elad Guttman +1Synthetic Data PretrainingDomain-Adaptive Pretraining

    Dream

  7. SALUTE: Benchmarking and Adapting LLMs for the Defense Domain

    Sep 14, 2026Hyeongcheol Park, Sumin In, Suyeon Myeong +5LLM EvaluationDomain-Adaptive Pretraining

    Korea University, Republic of Korea · Sungkyunkwan University, Republic of Korea · Hanwha Systems, Republic of Korea

  8. A Multi-Resolution Multi-Domain Pre-Training Framework for Universal Traffic Forecasting

    Sep 12, 2026Zhouyang Liu, Jindong Han, Hao Wang +4Traffic ForecastingSpatiotemporal Forecasting

    PDL Lab, National University of Defense Technology, Hunan, China · Shandong University, Shandong, China · HKUST (GZ) +2

  9. Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training

    Sep 8, 2026Yunpeng Xu, Kun ZhengLLM AlignmentDomain-Adaptive Pretraining

  10. Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

    Sep 7, 2026Yongan Yu, Shantam Raj, Jingwei Ni +3Multi-Source Domain AdaptationDomain-Adaptive Pretraining

    McGill University · Mila – Quebec Artificial Intelligence Institute · University of Zürich +2

  11. Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora

    Sep 4, 2026Chirag Garg, Eelaaf Zahid, Farhan Ahmed +3Domain-Adaptive Pretraining

    IBM Research

  12. Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training

    Aug 31, 2026Lukas Borggren, Jenny Kunz, Marco KuhlmannContinual Learning for LLMsLLM Evaluation

    Linköping University · Bonnier News

  13. TELLME: Test-Enhanced Learning for Language Model Enrichment

    Aug 12, 2026Minjun Kim, Inho Won, Hyeonseok Lim +6Domain-Adaptive PretrainingEfficient Language Model Training

    Korea Advanced Institute of Science and Technology · Seoul National University of Science and Technology · National Security Research Institute

  14. Diffract: Spectral View of LLM Domain Adaptation

    Aug 11, 2026Nikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain AdaptationDomain-Adaptive Pretraining

    Risk AI Research Lab · Applied AI Institute

  15. How Far Do Foundation Models Transfer to Infant Signals? A Cross-Dataset Transfer Audit with a Unified Need Ontology

    Aug 10, 2026Wu HangyuAudio Representation LearningDomain-Adaptive Pretraining

    Shenzhen Coddie Technology Co., Ltd.

  16. Two-Stage Bengali Sentiment Classification: Domain Adaptation Through Continual Learning and Parameter-Efficient Fine-Tuning

    Aug 2, 2026MD Shaikh Rahman, Syed Maudud E Rabbi, Muhammad Mahbubur RashidSentiment AnalysisDomain-Adaptive Pretraining

    Department of Computer Science Universiti Sains Malaysia Pulau Pinang, Malaysia · Department of Computer Science Baruch College New York New York, USA · Department of Mechatronics Engineering International Islamic University Malaysia Kuala Lumpur, Malaysia

  17. Improving Mental Health Screening and Early Risk Detection in Spanish

    Jul 30, 2026Andreu Casamayor-Segarra, Vicent Ahuir, Antonio Molina-Marco +1Social Media AnalysisMental Health

    VRAIN: Valencian Research Institute for Artificial Intelligence Universitat Polit`ecnica de Val`encia Val`encia, Spain · VRAIN: Valencian Research Institute for Artificial Intelligence Universitat Polit`ecnica de Val`encia ValgrAI: Valencian Graduate School and Research Network of Artificial Intelligence Val`encia, Spain

  18. Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality

    Jul 16, 2026Kunal Pratap Singh, Ali Garjani, Rishubh Singh +6Cross-Modal LearningCross-Modal Representation Learning

    Swiss Federal Institute of Technology Lausanne (EPFL) · Apple

  19. Customized Generative AI Agent for Transportation Engineering Practice: A Development and Continued Pre-training Guideline

    Jun 27, 2026Dianwei Chen, Yuan-Zheng Lei, Zifan Zhang +2Domain-Adaptive PretrainingLLM Agent Training

  20. Legal Domain Adaptation of Modern BERT Models

    Jun 26, 2026Dominik Stammbach, Peter HendersonLegal NLPBidirectional Encoder Representations from Transformers

    Princeton University Princeton, New Jersey, USA

  21. Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining

    Jun 20, 2026Bofeng Huang, Jacques Sun, Diane Bouchacourt +2Language Model PretrainingMasked Language Modeling

    Doctolib · Work done while at Doctolib.

  22. Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

    Jun 17, 2026Ikram Belmadani, Oumaima El Khettari, Carlos Ramisch +3LLM Fine-TuningDomain-Adaptive Pretraining

    Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N UMR 6004, 44000 Nantes, France · Grenoble Alpes Univ., CNRS, INRIA, Grenoble INP, LIG UMR 5217, 38000 Grenoble, France

  23. MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection

    Jun 10, 2026Fatimah Almalki, Areej Alhothali, Lulwah Alharigy +1Social Media AnalysisHierarchical Classification

    Department of Computer Science Faculty of Computing and Information Technology King Abdulaziz University Jeddah, Saudi Arabia · Department of Psychology College of Arts and Humanities King Abdulaziz University Jeddah, Saudi Arabia

  24. The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

    Jun 2, 2026Henry He, Johann Frei, Raphael SchmittDomain-Adaptive PretrainingClinical NLP

    School of Computation, Information and Technology, Technical University of Munich, Munich, Germany. · Chair of IT Infrastructure for Translational Medical Research, Faculty of Applied Computer Science, University of Augsburg, Augsburg, Germany. · Institute of General Practice, Faculty of Medicine and Medical Center, University of Freiburg, Germany.

  25. KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts

    Jun 1, 2026Christian Autenried, Cosimo PersiaLanguage Model PretrainingDomain-Adaptive Pretraining

    Helse Vest ICT

  26. Clustering Guided Domain-Specific Pretrained Foundation Model for Very High-Resolution Arctic Remote Sensing

    May 28, 2026Amal S. Perera, Chandi Witharana, Elias Manos +2Domain-Adaptive PretrainingVision Foundation Model Adaptation

    Department of Natural Resources and the Environment, University of Connecticut, Storrs, CT · Woodwell Climate Research Center, Falmouth, MA.

  27. Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT

    May 25, 2026Duy Anh NguyenEducational AssessmentDomain-Adaptive Pretraining

    University of Greenwich, London, United Kingdom · FPT University, Can Tho, Vietnam

  28. Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization

    May 17, 2026Gunjan Balde, Soumyadeep Roy, Mainack Mondal +1Domain-Adaptive PretrainingText Summarization

    Dept. of Computer Science and Engg., IIT Kharagpur, Kharagpur, India · Dept. of Medicine (Biomedical Informatics), Stanford University, Stanford, CA, USA

  29. Decoupled and Divergence-Conditioned Prompt for Multi-domain Dynamic Graph Foundation Models

    May 13, 2026Haonan Yuan, Qingyun Sun, Junhua Shi +3Domain-Adaptive PretrainingPrompt Tuning

    School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin 541004, China · Department of Computer Science, University of Illinois at Chicago, Chicago 60607, USA

  30. From pre-training to downstream performance: Does domain-specific pre-training make sense?

    May 9, 2026Felix KronesSelf-Supervised Pre-TrainingDomain-Adaptive Pretraining

    Oxford Internet Institute, University of Oxford, Oxford, UK