Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations

Rami Luisto; Liisa Petäinen; Tommi Grönholm; Jan Böhm; Maarit Ahtiainen; Tomi Lilja; Ilkka Pölönen; Sami Äyrämö

cs.CLApr 16, 2026

Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations

Authors: Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö

Organizations: Faculty of Information Technology, University of Jyvaskyla, Jyvaskyla,2026 Finland. · 2Digital Workforce Services, Helsinki, Finland. · 3Heart and Lung Center, Helsinki University Hospital, Helsinki, Finland.Apr · 4Central Finland Biobank, Jyvaskyla, Finland. · 5Wellbeing Services County of Central Finland, Jyvaskyla, Finland.

Abstract

In NLP classification tasks where little labeled data exists, domain fine-tuning of transformer models on unlabeled data is an established approach. In this paper we have two aims. (1) We describe our observations from fine-tuning the Finnish BERT model on Finnish medical text data. (2) We report on our attempts to predict the benefit of domain-specific pre-training of Finnish BERT from observing the geometry of embedding changes due to domain fine-tuning. Our driving motivation is the common\situation in healthcare AI where we might experience long delays in acquiring datasets, especially with respect to labels.

Explore similar work

The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP
Jun 2, 2026Henry He, Johann Frei, Raphael SchmittBert-Based Models Multilingual Language Models
Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?
Apr 21, 2026Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh +3Large Language Model Adaptation Model Pretraining
Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization
Jun 11, 2026Gaurav KumarSmall Language Models Finetuning

Jun 2, 2026 · cs.CLJ/K move · Enter open · S save

The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

Henry He, Johann Frei, Raphael Schmitt

Digital healthcare generates vast amounts of clinical text that can support AI-assisted applications, yet German biomedical language models remain limited by older architectures or restricted training data. We present ChristBERT (Clinical- and Healthcare-Related Issues and Subjects Tuned BERT), a family of domain-specific German RoBERTa-based language models trained on a 13.5GB corpus of scientific publications, clinical texts, health-related web content, and translated clinical resources. To investigate the impact of domain adaptation strategies in German clinical NLP, we compare continued pre-training, training from scratch, and domain-specific vocabulary adaptation. The resulting models are evaluated on three medical named entity recognition tasks and two text classification tasks. ChristBERT consistently outperforms existing general-purpose and medical German language models on four of five benchmarks and establishes a new state of the art for German clinical language modeling. Our results show that the optimal adaptation strategy is task-dependent: in our evaluation, training from scratch is particularly effective for highly specialized clinical texts, whereas continued pre-training performs well on more commonly written medical texts. All models are publicly released to support future research and applications in German medical NLP.

Bert-Based Models Multilingual Language Models Model Pretraining

Open paper Ask about it PDF

Abstract

Explore similar work

The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?

Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization