Domain-Adaptive Pretraining

Ask about this topic

Overview

Domain-adaptive pretraining studies how to specialize general-purpose language and vision models by continuing self-supervised training on domain-specific text, images, or other data. Common approaches include masked-language or next-token training, self-supervised vision methods, and mid-training, often paired with curated corpora, adapted vocabularies, replay, or parameter-efficient tuning. Evaluations span medical, legal, scientific, and technical tasks such as question answering, classification, information extraction, and image mapping, alongside checks of general capabilities and cross-domain transfer. Results often favor data closely matched to the target task, but gains are not guaranteed: specialization can trade off against broader capabilities, and benchmark performance may not reflect transfer to new settings.

Latest papers

CardsList

44 papers4 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?

    Apr 21, 2026Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh +3Domain-Adaptive PretrainingDomain Adaptation for LLMs

    Fraunhofer IAIS · Lamarr Institute

  2. REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning

    Apr 19, 2026Seungmin Lee, Jeonghwan Lee, Hyunkuk Lim +2Text EmbeddingsRepresentation Learning

    PwC Korea GenAI Team, Seoul, South Korea

  3. Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlations

    Apr 16, 2026Rami Luisto, Liisa Petäinen, Tommi Grönholm +5HealthcareDomain-Adaptive Pretraining

    Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland. · Digital Workforce Services, Helsinki, Finland. · Heart and Lung Center, Helsinki University Hospital, Helsinki, Finland. +2

  4. Improving DNS Exfiltration Detection via Transformer Pretraining

    Apr 9, 2026Miloš Tomić, Aleksa Cvetanović, Predrag TadićNetwork Intrusion DetectionDomain-Adaptive Pretraining

    School of Electrical Engineering, University of Belgrade, Serbia

  5. Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus

    Apr 8, 2026Aidan Mannion, Cécile Macaire, Armand Violle +5Domain-Adaptive PretrainingDomain Adaptation for LLMs

    Université Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France · Sorbonne Université, LIMICS, 15 rue de l’École de Médecine, 75006 Paris, France

  6. HUKUKBERT: Domain-Specific Language Model for Turkish Law

    Apr 6, 2026Mehmet Utku Öztürk, Tansu Türkoğlu, Buse Buz-YalugLegal NLPLanguage Modeling

    Kalitte Inc., Türkiye · Aibrite Inc., Türkiye · University of Eastern Finland, Finland

  7. STRADAViT: Self-Supervised Domain Adaptation of Vision Transformer Backbones for Radio Astronomy

    Mar 31, 2026Andrea DeMarco, Ian Fenech Conti, Hayley Camilleri +2Radio AstronomyDomain-Adaptive Pretraining

    Institute of Space Sciences and Astronomy, University of Malta, Msida, MSD2080, Malta · School of Physics and Astronomy, Royal Observatory, University of Edinburgh, Edinburgh, United Kingdom · Istituto Nazionale di Astrofisica (INAF), Osservatorio Astrofisico di Catania, Catania, Italy

  8. Domain Adapted Large Language Models for Additive Manufacturing

    Mar 23, 2026Peter Pak, Amir Barati FarimaniMultimodal Large Language ModelsDomain Adaptation for LLMs

    Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA · Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA

  9. Optimal Splitting of Language Models from Mixtures to Specialized Domains

    Mar 19, 2026Skyler Seto, Pierre Ablin, Anastasiia Filippova +4Efficient Language Model TrainingDomain-Adaptive Pretraining

    Apple · National University of Singapore

  10. The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data

    Mar 17, 2026Christina Baek, Ricardo Pio Monti, David Schwab +31Language Model PretrainingDomain-Adaptive Pretraining

    DatologyAI

  11. On the Value of Tokeniser Pretraining in Physics Foundation Models

    Mar 5, 2026Hadi Sotoudeh, Payel Mukhopadhyay, Ruben Ohana +4Physical Foundation ModelsPhysics-Based Simulation

    University of Cambridge · Flatiron Institute · New York University +1

  12. Improving Automatic Summarization of Radiology Reports through Mid-Training of Large Language Models

    Feb 28, 2026Mengxian Lyu, Cheng Peng, Ziyi Chen +3Text SummarizationMedical LLMs

    Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, FL, USA · Department of Engineering Education, University of Florida, Gainesville, FL, USA · Department of Health Outcomes and Biomedical Informatics, Preston A. Wells, Jr. Center for Brain Tumor Therapy, University of Florida, Gainesville, FL, USA

  13. The pretraining domain outweighs the training objective in setting the privacy-utility trade-off of differentially private medical image analysis

    Jan 27, 2026Soroosh Tayebi Arasteh, Mina Farajiamiri, Mahshad Lotfinia +7Privacy-Utility TradeoffMedical Image Classification

    Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany · Department of Urology, Stanford University, Stanford, CA, USA +5

  14. Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens

    Jun 30, 2025Salahuddin Salahuddin, Ahmed Hussain, Jussi Löppönen +1Language Model PretrainingDomain-Adaptive Pretraining

    SSH Communications Security, Helsinki, Finland · Aalto University, Espoo, Finland · Networked Systems Security (NSS) Group, KTH Royal Institute of Technology, Stockholm, Sweden