Domain-Adaptive Pretraining

Overview

Domain-adaptive pretraining studies how to specialize general-purpose language and vision models by continuing self-supervised training on domain-specific text, images, or other data. Common approaches include masked-language or next-token training, self-supervised vision methods, and mid-training, often paired with curated corpora, adapted vocabularies, replay, or parameter-efficient tuning. Evaluations span medical, legal, scientific, and technical tasks such as question answering, classification, information extraction, and image mapping, alongside checks of general capabilities and cross-domain transfer. Results often favor data closely matched to the target task, but gains are not guaranteed: specialization can trade off against broader capabilities, and benchmark performance may not reflect transfer to new settings.

Latest papers

CardsList

46 papers2 papers in the last four weeks, down 60% on the four weeks before. 0.0% of all new papers.

  1. Learning from Unlabeled Photographs for General Oral Image Analysis

    Nov 3, 2026Zhong-Shi Zhang, Zi-Li Yu, Xiao Xiao +4Medical Image Representation LearningMedical Image Analysis

  2. BanglishDepYT: Depression Detection in Native Bengali Social Media with Emotion-Preserving Preprocessing and Domain-Adaptive Pretraining

    Nov 1, 2026Ayon Roy, Sadman Sharif, Syed Shams IslamDepression DetectionBengali NLP

  3. Robust Marathi Sentiment Analysis using Domain-Adaptive Pre-training and Data Augmentation Approaches

    Oct 11, 2026Tanvi Somani, Aryan Babare, Samyak Bora +2Sentiment AnalysisLow-Resource Language Processing

  4. Domain adaptation of Russian ModernBERT for long legal documents

    Oct 5, 2026I. Litvak, D. Gvozdetsky, F. Lashkin +7Language Model PretrainingLegal NLP

  5. OncoNoteBERT: A Foundation Representation Model for Natural Language Processing of Real-World Outpatient Oncology Notes

    Oct 2, 2026Wuraola Oyewusi, Eliana Vasquez Osorio, Goran Nenadic +1Clinical NLPLanguage Modeling

  6. Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining

    Sep 22, 2026Mudi Zhai, Ruihong Qiu, Qingyun Zeng +3KG ConstructionDomain-Adaptive Pretraining

  7. MiST: Mid-Training LLMs for Cybersecurity

    Sep 16, 2026Oded Ovadia, Elad Ben Zaken, Elad Guttman +1Synthetic Data PretrainingDomain-Adaptive Pretraining

  8. SALUTE: Benchmarking and Adapting LLMs for the Defense Domain

    Sep 14, 2026Hyeongcheol Park, Sumin In, Suyeon Myeong +5LLM EvaluationDomain-Adaptive Pretraining

  9. A Multi-Resolution Multi-Domain Pre-Training Framework for Universal Traffic Forecasting

    Sep 12, 2026Zhouyang Liu, Jindong Han, Hao Wang +4Traffic ForecastingSpatiotemporal Forecasting

  10. Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

    Sep 7, 2026Yongan Yu, Shantam Raj, Jingwei Ni +3Multi-Source Domain AdaptationDomain-Adaptive Pretraining

  11. Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora

    Sep 4, 2026Chirag Garg, Eelaaf Zahid, Farhan Ahmed +3Domain-Adaptive Pretraining

  12. TELLME: Test-Enhanced Learning for Language Model Enrichment

    Aug 12, 2026Minjun Kim, Inho Won, Hyeonseok Lim +6Domain-Adaptive PretrainingEfficient Language Model Training

  13. Diffract: Spectral View of LLM Domain Adaptation

    Aug 11, 2026Nikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain AdaptationDomain-Adaptive Pretraining

  14. Two-Stage Bengali Sentiment Classification: Domain Adaptation Through Continual Learning and Parameter-Efficient Fine-Tuning

    Aug 2, 2026MD Shaikh Rahman, Syed Maudud E Rabbi, Muhammad Mahbubur RashidSentiment AnalysisDomain-Adaptive Pretraining

  15. Improving Mental Health Screening and Early Risk Detection in Spanish

    Jul 30, 2026Andreu Casamayor-Segarra, Vicent Ahuir, Antonio Molina-Marco +1Social Media AnalysisMental Health

  16. Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality

    Jul 16, 2026Kunal Pratap Singh, Ali Garjani, Rishubh Singh +6Cross-Modal LearningCross-Modal Representation Learning

  17. Legal Domain Adaptation of Modern BERT Models

    Jun 26, 2026Dominik Stammbach, Peter HendersonLegal NLPBidirectional Encoder Representations from Transformers

  18. Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining

    Jun 20, 2026Bofeng Huang, Jacques Sun, Diane Bouchacourt +2Language Model PretrainingMasked Language Modeling

  19. Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

    Jun 17, 2026Ikram Belmadani, Oumaima El Khettari, Carlos Ramisch +3LLM Fine-TuningDomain-Adaptive Pretraining

  20. MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection

    Jun 10, 2026Fatimah Almalki, Areej Alhothali, Lulwah Alharigy +1Social Media AnalysisHierarchical Classification

  21. The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

    Jun 2, 2026Henry He, Johann Frei, Raphael SchmittDomain-Adaptive PretrainingClinical NLP

  22. Clustering Guided Domain-Specific Pretrained Foundation Model for Very High-Resolution Arctic Remote Sensing

    May 28, 2026Amal S. Perera, Chandi Witharana, Elias Manos +2Domain-Adaptive PretrainingVision Foundation Model Adaptation

  23. Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization

    May 17, 2026Gunjan Balde, Soumyadeep Roy, Mainack Mondal +1Domain-Adaptive PretrainingText Summarization