Domain-Adaptive Pretraining

Ask about this topic

Momentum

6 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 30

All topics
CardsList

  1. Domain adaptation of Russian ModernBERT for long legal documents

    Oct 5, 2026I. Litvak, D. Gvozdetsky, F. Lashkin +7Language Model PretrainingLegal NLP

    Independent Researcher · National Research Nuclear University MEPhI · Moscow Center for Advanced Studies +1

  2. QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code

    Sep 30, 2026Alexey Chernysh, Orkhan Ekhtibarov, Dmitry ZmitrovichLLM Fine-TuningDomain-Adaptive Pretraining

  3. Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining

    Sep 22, 2026Mudi Zhai, Ruihong Qiu, Qingyun Zeng +3KG ConstructionDomain-Adaptive Pretraining

    UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales, Sydney, NSW 2052, Australia · School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, QLD 4072, Australia · Microsoft Copilot Studio AI, Redmond, WA 98052, United States +2

  4. MiST: Mid-Training LLMs for Cybersecurity

    Sep 16, 2026Oded Ovadia, Elad Ben Zaken, Elad Guttman +1Synthetic Data PretrainingDomain-Adaptive Pretraining

    Dream

  5. SALUTE: Benchmarking and Adapting LLMs for the Defense Domain

    Sep 14, 2026Hyeongcheol Park, Sumin In, Suyeon Myeong +5LLM EvaluationDomain-Adaptive Pretraining

    Korea University, Republic of Korea · Sungkyunkwan University, Republic of Korea · Hanwha Systems, Republic of Korea

  6. Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training

    Sep 8, 2026Yunpeng Xu, Kun ZhengLLM AlignmentDomain-Adaptive Pretraining

  7. Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

    Sep 7, 2026Yongan Yu, Shantam Raj, Jingwei Ni +3Multi-Source Domain AdaptationDomain-Adaptive Pretraining

    McGill University · Mila – Quebec Artificial Intelligence Institute · University of Zürich +2

  8. Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training

    Aug 31, 2026Lukas Borggren, Jenny Kunz, Marco KuhlmannContinual Learning for LLMsLLM Evaluation

    Linköping University · Bonnier News

  9. TELLME: Test-Enhanced Learning for Language Model Enrichment

    Aug 12, 2026Minjun Kim, Inho Won, Hyeonseok Lim +6Domain-Adaptive PretrainingEfficient Language Model Training

    Korea Advanced Institute of Science and Technology · Seoul National University of Science and Technology · National Security Research Institute

  10. How Far Do Foundation Models Transfer to Infant Signals? A Cross-Dataset Transfer Audit with a Unified Need Ontology

    Aug 10, 2026Wu HangyuAudio Representation LearningDomain-Adaptive Pretraining

    Shenzhen Coddie Technology Co., Ltd.

  11. Two-Stage Bengali Sentiment Classification: Domain Adaptation Through Continual Learning and Parameter-Efficient Fine-Tuning

    Aug 2, 2026MD Shaikh Rahman, Syed Maudud E Rabbi, Muhammad Mahbubur RashidSentiment AnalysisDomain-Adaptive Pretraining

    Department of Computer Science Universiti Sains Malaysia Pulau Pinang, Malaysia · Department of Computer Science Baruch College New York New York, USA · Department of Mechatronics Engineering International Islamic University Malaysia Kuala Lumpur, Malaysia

  12. Improving Mental Health Screening and Early Risk Detection in Spanish

    Jul 30, 2026Andreu Casamayor-Segarra, Vicent Ahuir, Antonio Molina-Marco +1Social Media AnalysisMental Health

    VRAIN: Valencian Research Institute for Artificial Intelligence Universitat Polit`ecnica de Val`encia Val`encia, Spain · VRAIN: Valencian Research Institute for Artificial Intelligence Universitat Polit`ecnica de Val`encia ValgrAI: Valencian Graduate School and Research Network of Artificial Intelligence Val`encia, Spain

  13. Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality

    Jul 16, 2026Kunal Pratap Singh, Ali Garjani, Rishubh Singh +6Cross-Modal LearningCross-Modal Representation Learning

    Swiss Federal Institute of Technology Lausanne (EPFL) · Apple

  14. Customized Generative AI Agent for Transportation Engineering Practice: A Development and Continued Pre-training Guideline

    Jun 27, 2026Dianwei Chen, Yuan-Zheng Lei, Zifan Zhang +2Domain-Adaptive PretrainingLLM Agent Training

  15. Legal Domain Adaptation of Modern BERT Models

    Jun 26, 2026Dominik Stammbach, Peter HendersonLegal NLPTransformer

    Princeton University Princeton, New Jersey, USA

  16. Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining

    Jun 20, 2026Bofeng Huang, Jacques Sun, Diane Bouchacourt +2Language Model PretrainingMasked Language Modeling

    Doctolib · Work done while at Doctolib.

  17. Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

    Jun 17, 2026Ikram Belmadani, Oumaima El Khettari, Carlos Ramisch +3LLM Fine-TuningDomain-Adaptive Pretraining

    Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N UMR 6004, 44000 Nantes, France · Grenoble Alpes Univ., CNRS, INRIA, Grenoble INP, LIG UMR 5217, 38000 Grenoble, France

  18. MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection

    Jun 10, 2026Fatimah Almalki, Areej Alhothali, Lulwah Alharigy +1Social Media AnalysisArabic NLP

    Department of Computer Science Faculty of Computing and Information Technology King Abdulaziz University Jeddah, Saudi Arabia · Department of Psychology College of Arts and Humanities King Abdulaziz University Jeddah, Saudi Arabia

  19. The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

    Jun 2, 2026Henry He, Johann Frei, Raphael SchmittDomain-Adaptive PretrainingClinical NLP

    School of Computation, Information and Technology, Technical University of Munich, Munich, Germany. · Chair of IT Infrastructure for Translational Medical Research, Faculty of Applied Computer Science, University of Augsburg, Augsburg, Germany. · Institute of General Practice, Faculty of Medicine and Medical Center, University of Freiburg, Germany.

  20. KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts

    Jun 1, 2026Christian Autenried, Cosimo PersiaLanguage Model PretrainingDomain-Adaptive Pretraining

    Helse Vest ICT

  21. Clustering Guided Domain-Specific Pretrained Foundation Model for Very High-Resolution Arctic Remote Sensing

    May 28, 2026Amal S. Perera, Chandi Witharana, Elias Manos +2Domain-Adaptive PretrainingVision Foundation Model Adaptation

    Department of Natural Resources and the Environment, University of Connecticut, Storrs, CT · Woodwell Climate Research Center, Falmouth, MA.

  22. Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT

    May 25, 2026Duy Anh NguyenEducational AssessmentDomain-Adaptive Pretraining

    University of Greenwich, London, United Kingdom · FPT University, Can Tho, Vietnam

  23. Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization

    May 17, 2026Gunjan Balde, Soumyadeep Roy, Mainack Mondal +1Domain-Adaptive PretrainingText Summarization

    Dept. of Computer Science and Engg., IIT Kharagpur, Kharagpur, India · Dept. of Medicine (Biomedical Informatics), Stanford University, Stanford, CA, USA

  24. Decoupled and Divergence-Conditioned Prompt for Multi-domain Dynamic Graph Foundation Models

    May 13, 2026Haonan Yuan, Qingyun Sun, Junhua Shi +3Domain-Adaptive PretrainingPrompt Tuning

  25. From pre-training to downstream performance: Does domain-specific pre-training make sense?

    May 9, 2026Felix KronesSelf-Supervised Pre-TrainingDomain-Adaptive Pretraining

    Oxford Internet Institute, University of Oxford, Oxford, UK

  26. Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?

    Apr 21, 2026Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh +3Domain-Adaptive PretrainingDomain Adaptation for LLMs

    Fraunhofer IAIS · Lamarr Institute

  27. REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning

    Apr 19, 2026Seungmin Lee, Jeonghwan Lee, Hyunkuk Lim +2Text EmbeddingsRepresentation Learning

    PwC Korea GenAI Team, Seoul, South Korea

  28. Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlations

    Apr 16, 2026Rami Luisto, Liisa Petäinen, Tommi Grönholm +5HealthcareDomain-Adaptive Pretraining

    Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland. · Digital Workforce Services, Helsinki, Finland. · Heart and Lung Center, Helsinki University Hospital, Helsinki, Finland. +2

  29. Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens

    Jun 30, 2025Salahuddin Salahuddin, Ahmed Hussain, Jussi Löppönen +1Language Model PretrainingDomain-Adaptive Pretraining

    SSH Communications Security, Helsinki, Finland · Aalto University, Espoo, Finland · Networked Systems Security (NSS) Group, KTH Royal Institute of Technology, Stockholm, Sweden

  30. Diffract: Spectral View of LLM Domain Adaptation

    Date pendingNikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain AdaptationDomain-Adaptive Pretraining

    Risk AI Research Lab · Applied AI Institute