Paper ID: 2204.03951

RuBioRoBERTa: a pre-trained biomedical language model for Russian language biomedical text mining

Alexander Yalunin, Alexander Nesterov, Dmitriy Umerenkov

This paper presents several BERT-based models for Russian language biomedical text mining (RuBioBERT, RuBioRoBERTa). The models are pre-trained on a corpus of freely available texts in the Russian biomedical domain. With this pre-training, our models demonstrate state-of-the-art results on RuMedBench - Russian medical language understanding benchmark that covers a diverse set of tasks, including text classification, question answering, natural language inference, and named entity recognition.

Submitted: Apr 8, 2022