CardioLM: A multilingual suite of small language models for the cardiology domain

Manuela-Daniela Danu; Miguel Rodríguez-Ortega; Alberto Becerra-Tome; Pablo Arancibia; Fernando Gallego-Donoso; Lorenzo Pratesi; Xi Bai; Andrei-Damian Dinescu

cs.CLNov 5, 2026

CardioLM: A multilingual suite of small language models for the cardiology domain

Authors: Manuela-Daniela Danu, Miguel Rodríguez-Ortega, Alberto Becerra-Tome, Pablo Arancibia, Fernando Gallego-Donoso, Lorenzo Pratesi, Xi Bai, Andrei-Damian Dinescu, Salvador Lima-López, Andrea Guala, Saskia Haitjema, Jonas Mack, Michele Miranda, Wesam Al Nabki, Jan Rodríguez-Miret, Vasile George Marica, Vladimír Kincl, Kateřina Dostálová, Michael Andrej, Alicia Ujil, Corinne Allaart, Machteld Boonstra, Malin Overmars, Anoop Shah, Giovanna Ceroni, Jonathan Kossoff, Domenico D'Amario, Renzo Laborante, Antonio Iaconelli, Isabella Fumarulo, Jacopo Lenkowicz, Livia Lilli, Emma Weiss, Alex Scafa, Felix Lindberg, Milou Gamage, Eulàlia Farré-Maduell, Sébastien Bratières, Martin Krallinger, Bram van Es

Organizations: Siemens SRL, Brasov, Romania. · Automation and Information Technology, Transilvania University of Brasov, Brasov, Romania. · NLP4BIA, Barcelona Supercomputing Center, Barcelona, Spain. · Translated, Rome, Italy. · University College London, London, United Kingdom. · Vall d’Hebron Institute of Research, Barcelona, Spain. · CIBER-CV, Instituto de Salud Carlos III, Madrid, Spain. · University Medical Center Utrecht, Utrecht, The Netherlands. · Sapienza University of Rome, Rome, Italy. · International Clinical Research Center, St. Anne ’s University Hospital, Faculty of Medicine, Masaryk University, Brno, Czech Republic. · Department of Internal Medicine/Cardioangiology, University Hospital Hradec, Králové, Czech Republic. · Department of Cardiology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands. · Amsterdam Cardiovascular Sciences Research Institute, Amsterdam, the Netherlands. · Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy. · Data Science Unit, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy. · Bucharest Emergency Clinical Hospital, Bucharest, Romania. · Department of Clinical Science and Education, Södersjukhuset, Karolinska Institutet, Stockholm, Sweden.

Abstract

Background: Cardiovascular care generates large volumes of clinical free-text documentation in local languages. However, clinical natural language processing (NLP) resources remain concentrated in English, and no existing benchmark is cardiology-specific, multilingual, and broad in entity coverage. Methods: We present two contributions. The first is CardioCCC, a corpus of 508 cardiology case reports originally published in Spanish and extended to Czech, Dutch, English, Italian, Romanian, and Swedish through machine translation, annotation projection, and clinician review, with annotations covering diseases, symptoms, procedures, and medications. The second is CardioLM, comprising CardioBERTa, a set of language-specific encoders obtained by continued pretraining on cardiology and biomedical corpora, and CardioNER, a set of named entity recognition (NER) models fine-tuned on CardioCCC. Results: Cardiology-specific pretraining reduced pseudo-perplexity by up to a factor of eight, yet changed strict span-level F1 by at most 0.04, with benefits concentrated in languages whose backbones lacked biomedical coverage. The jointly trained multilingual XLM-RoBERTa-large models achieved the best strict F1 in nearly every language and entity type, exceeding the strongest monolingual system by up to 0.13 F1. A large share of residual errors concerned boundary detection: relaxed matching recovered 0.10-0.23 F1 for disease, procedure, and symptom, but only 0.01-0.07 for medication. Clinicians rated CardioNER well above a rule- and dictionary-based baseline, with the mean rating across entity types rising from 26% to 73%. Conclusions: Specialty-specific pretraining is not the decisive factor for downstream entity recognition, whereas multilingual joint training yields the strongest results in nearly all settings and a simpler deployment path for multi-country clinical information extraction.

Figures & tables

Figure 1: End-to-end pipeline from cardiology-domain continued pretraining to NER system development.

Language	Cardiology	Biomedical	General	Total
Czech	150	2,330	403	2,883
Dutch	1,400	4,073	0	5,473
English	771	8	163	942
Italian	266	4,562	0	4,828
Romanian	184	4,211	0	4,395
Spanish	207	3,578	0	3,785

Table 1: Size of the continued pretraining corpus per language, expressed in millions of words.

Dataset characteristics
Total word count (source; translations; total)	51,079; 343,368; 394,447
Total number of documents	658
Demographics
Age (average; range)	58.2; 20–110
Female sex (n; %)	343 (52%)
Document types

Table 2: Characteristics of the ParaCliTe dataset.

Figure 2: Illustrative English CardioCCC sample showing projected entity annotations for diseases, symptoms and clinical findings, procedures, and medications prior to manual clinical review.

Language	DISEASE	MEDICATION	PROCEDURE	SYMPTOM	TOTAL
Czech	17,788	4,290	16,477	18,729	57,284
Dutch	17,646	4,317	16,195	18,590	56,748
English	17,097	4,231	15,708	18,694	55,730
Italian	18,175	4,385	16,521	19,467	58,548
Romanian	17,546	4,239	16,379	18,068	56,232
Spanish	18,232	4,227	17,073	19,983	59,515

Table 3: Statistics of the multilingual, annotated CardioCCC dataset.

Figure 3: Illustrative comparison of the lexical baseline (left) and the Transformer-based CardioNER system (right) on an English ParaCliTe report.

Language	Base-model name	Domain	Architecture	Pretraining task	Model size (# trainable parameters)
Czech	ufal/robeczech-base	generic	RoBERTa	MLM	125M
	Straka et al. (2021)
Dutch	CLTL/MedRoBERTa.nl	biomedical	RoBERTa	MLM	125M
	Verkijk and Vossen (2021)
English	allenai/biomed_roberta_base	biomedical	RoBERTa	MLM	125M
	Gururangan et al. (2020)

Table 4: Overview of the base models selected for language-specific continued pretraining.

Figure 4: Shard-wise continued pretraining: the checkpoint produced on each corpus shard initialises training on the next, until all shards have been consumed.

Figure 5: Two-stage token-to-span decoding, comprising (1) token-level BIO prediction and (2) span estimation and post-processing.

Language	Model	PPL minicons median(geometric mean)	PPL ours median(geometric mean)
Czech	robeczech-base	56.49 (56.17)	22.68 (23.77)
Czech	CardioBERTa.cs	6.94 ( 6.96 )	3.22 ( 3.53 )
Dutch	RobBERT2023-base	12.70 (12.50)	6.40 ( 6.20 )
	MedRoBERTa.nl	44.70 (45.40)	30.70 (30.60)
	CardioBERTa.nl	9.80 ( 9.40 )	6.80 (6.80)
English	biomed_roberta_base	29.76 (29.88)	16.13 (16.69)

Table 5: Pseudo-perplexity aggregation over all spans .

Language	Model	PPL minicons median(geometric mean)	PPL ours median(geometric mean)
Czech	robeczech-base	38.88 (49.41)	4.90 (10.42)
Czech	CardioBERTa.cs	9.09 ( 14.37 )	1.71 ( 4.25 )
Dutch	RobBERT2023-base	55.4 (62.1)	9.8 (17.4)
	MedRoBERTa.nl	65.2 (87.5)	24.9 (43.6)
	CardioBERTa.nl	17.6 ( 30.3 )	8.0 ( 17.2 )
English	biomed_roberta_base	12.77 (21.64)	2.43 (6.69)

Table 6: Pseudo-perplexity aggregation over all medical spans .

		Micro			Macro
Language	Model	F1	Recall	Precision	F1	Recall	Precision
Language	Model	strict (relaxed)	strict (relaxed)	strict (relaxed)	strict (relaxed)	strict (relaxed)	strict (relaxed)
Czech	robeczech-base	0.68 (0.84)	0.67 (0.82)	0.70 (0.85)	0.72 (0.85)	0.70 (0.83)	0.73 (0.87)
	CardioBERTa.cs	0.70 (0.84)	0.68 (0.83)	0.71 ( 0.86 )	0.72 (0.86)	0.71 (0.84)	0.74 ( 0.88 )
	xlm-roberta-large	0.71 ( 0.86 )	0.70 ( 0.85 )	0.71 ( 0.86 )	0.74 ( 0.87 )	0.73 ( 0.86 )	0.74 ( 0.88 )
Dutch	RobBERT2023-base	0.63 (0.81)	0.64 (0.82)	0.63 (0.80)	0.68 (0.83)	0.68 (0.83)	0.68 (0.82)

Table 7: Span-level NER performance of multilabel models on the CardioCCC held-out test set, averaged across the 10 fold models. Predictions are decoded using a single-label protocol (argmax over softmax-normalised logits ) and aggregated into entity spans. Micro- and macro-averages are computed over the four entity types, with strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 reported. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

		F1 strict (relaxed)				Recall strict (relaxed)				Precision strict (relaxed)
Language	Model	Dis	Med	Proc	Sym	Dis	Med	Proc	Sym	Dis	Med	Proc	Sym
Czech	robeczech-base	0.67 (0.86)	0.86 (0.90)	0.72 (0.83)	0.62 (0.80)	0.67 (0.85)	0.83 (0.87)	0.69 (0.80)	0.61 (0.79)	0.68 (0.86)	0.89 ( 0.94 )	0.75 (0.87)	0.62 (0.80)
	CardioBERTa.cs	0.70 ( 0.87 )	0.84 (0.90)	0.73 (0.85)	0.63 (0.81)	0.70 (0.87)	0.83 (0.88)	0.69 (0.80)	0.63 (0.80)	0.70 ( 0.87 )	0.86 (0.92)	0.77 ( 0.89 )	0.64 (0.81)
	xlm-roberta-large	0.71 ( 0.87 )	0.86 ( 0.91 )	0.74 ( 0.86 )	0.64 ( 0.82 )	0.71 ( 0.88 )	0.86 ( 0.91 )	0.72 ( 0.84 )	0.64 ( 0.83 )	0.71 ( 0.87 )	0.87 (0.92)	0.77 ( 0.89 )	0.63 ( 0.82 )
Dutch	RobBERT2023-base	0.62 (0.82)	0.87 (0.89)	0.65 (0.81)	0.57 (0.78)	0.62 (0.83)	0.85 (0.87)	0.64 (0.80)	0.61 (0.82)	0.62 (0.82)	0.88 (0.90)	0.67 (0.83)	0.54 (0.73)
	CardioBERTa.nl	0.64 (0.84)	0.83 (0.87)	0.67 (0.82)	0.61 (0.79)	0.65 (0.85)	0.82 (0.86)	0.61 (0.76)	0.59 (0.78)	0.63 (0.83)	0.84 (0.88)	0.72 ( 0.90 )	0.61 (0.80)

Table 8: Span-level NER performance of multilabel models on the CardioCCC held-out test set, broken down by entity type and averaged across the 10 fold models. Predictions are decoded using a single-label protocol (argmax over softmax-normalised logits ) and aggregated into entity spans. Strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 are reported for each of the four entity types. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

		F1 strict (relaxed)				Recall strict (relaxed)				Precision strict (relaxed)
Language	Model	Dis	Med	Proc	Sym	Dis	Med	Proc	Sym	Dis	Med	Proc	Sym
Czech	robeczech-base	0.67 (0.85)	0.86 (0.91)	0.74 (0.86)	0.61 (0.79)	0.68 (0.87)	0.85 (0.90)	0.72 (0.83)	0.61 (0.79)	0.65 (0.84)	0.87 (0.92)	0.76 (0.88)	0.61 (0.80)
	CardioBERTa.cs	0.69 (0.86)	0.85 ( 0.92 )	0.75 (0.87)	0.62 (0.80)	0.72 ( 0.89 )	0.85 ( 0.92 )	0.73 (0.85)	0.65 (0.83)	0.67 (0.84)	0.84 (0.91)	0.77 (0.89)	0.61 (0.77)
	xlm-roberta-large	0.72 ( 0.88 )	0.87 ( 0.92 )	0.77 ( 0.88 )	0.68 ( 0.83 )	0.73 ( 0.89 )	0.87 ( 0.92 )	0.75 ( 0.86 )	0.69 ( 0.85 )	0.72 ( 0.87 )	0.88 ( 0.93 )	0.78 ( 0.90 )	0.66 ( 0.81 )
Dutch	RobBERT2023-base	0.63 (0.85)	0.88 (0.90)	0.67 (0.82)	0.62 (0.77)	0.61 (0.82)	0.87 (0.88)	0.62 (0.76)	0.63 (0.81)	0.64 (0.85)	0.90 ( 0.92 )	0.72 (0.88)	0.60 (0.77)
	CardioBERTa.nl	0.66 (0.86)	0.86 (0.90)	0.68 (0.82)	0.65 (0.80)	0.64 (0.83)	0.85 (0.89)	0.62 (0.75)	0.64 (0.79)	0.68 ( 0.89 )	0.87 (0.91)	0.74 ( 0.90 )	0.65 (0.81)

Table 9: Span-level NER performance of multiclass models on the CardioCCC held-out test set, averaged across the 10 fold models. A separate model is fine-tuned and evaluated for each entity type; thus, results for the four entity types come from four independently trained models rather than a single multi-entity model. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 are reported for each entity type. XLM-RoBERTa-large is trained jointly on all seven languages as four separate entity-specific models and evaluated on the full multilingual test set and separately by language.

Figure 6: Mean clinician ratings for the baseline and Transformer-based NER systems, reported for each entity type (disease, symptom, procedure, and medication) and averaged across entity types.

Figure 7: Clinician-reported usefulness of baseline and Transformer-based clinical entity extraction systems for potential hospital-oriented implementation.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Model	F1	Recall	Precision
Language	Model	macro (micro)	macro (micro)	macro (micro)
Czech	robeczech-base	0.84 (0.83)	0.83 (0.82)	0.85 (0.83)
	CardioBERTa.cs	0.85 ( 0.84 )	0.85 (0.84)	0.86 ( 0.84 )
	xlm-roberta-large	0.85 ( 0.84 )	0.86 ( 0.85 )	0.85 ( 0.84 )
Dutch	RobBERT2023-base	0.80 (0.79)	0.81 (0.80)	0.80 (0.79)
	CardioBERTa.nl	0.79 (0.77)	0.78 (0.76)	0.80 (0.77)

Appendix

Table A.1: Token-level NER performance of multilabel models on the CardioCCC held-out test set, averaged across the 10 fold models. Averages are computed over the eight B-/I- labels, with the O class excluded from both micro- and macro-averages. Labels are decoded independently using sigmoid (

>0.5

), allowing a token to receive multiple labels or none. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

		Micro			Macro
Language	Model	F1	Recall	Precision	F1	Recall	Precision
Language	Model	strict (relaxed)	strict (relaxed)	strict (relaxed)	strict (relaxed)	strict (relaxed)	strict (relaxed)
Czech	robeczech-base	0.68 (0.83)	0.66 (0.80)	0.71 (0.87)	0.72 (0.85)	0.69 (0.81)	0.75 (0.88)
	CardioBERTa.cs	0.70 (0.84)	0.67 (0.81)	0.73 ( 0.88 )	0.73 (0.85)	0.70 (0.82)	0.76 ( 0.89 )
	xlm-roberta-large	0.71 ( 0.86 )	0.70 ( 0.84 )	0.72 (0.87)	0.74 ( 0.87 )	0.73 ( 0.85 )	0.75 (0.88)
Dutch	RobBERT2023-base	0.64 (0.81)	0.63 (0.80)	0.65 (0.82)	0.68 (0.83)	0.67 (0.81)	0.70 (0.84)

Appendix

Table A.2: Span-level NER performance of multilabel models on the CardioCCC held-out test set, averaged across the 10 fold models. Predictions are decoded using per-label sigmoid confidence scores for thresholding, matching the training objective, and then aggregated into entity spans. Because argmax label selection is invariant to sigmoid versus softmax scaling, each token receives a single label. Micro- and macro-averages are computed over the four entity types, with strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 reported. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

		F1 strict (relaxed)				Recall strict (relaxed)				Precision strict (relaxed)
Language	Model	Dis	Med	Proc	Sym	Dis	Med	Proc	Sym	Dis	Med	Proc	Sym
Czech	robeczech-base	0.68 (0.86)	0.85 (0.89)	0.72 (0.84)	0.62 (0.79)	0.66 (0.83)	0.80 (0.85)	0.68 (0.79)	0.60 (0.77)	0.69 (0.88)	0.90 ( 0.95 )	0.77 (0.89)	0.65 (0.82)
	CardioBERTa.cs	0.70 ( 0.87 )	0.84 (0.89)	0.73 (0.85)	0.64 (0.80)	0.69 (0.85)	0.81 (0.86)	0.68 (0.79)	0.61 (0.76)	0.72 ( 0.89 )	0.87 (0.93)	0.79 ( 0.91 )	0.67 ( 0.83 )
	xlm-roberta-large	0.71 ( 0.87 )	0.87 ( 0.91 )	0.74 ( 0.87 )	0.64 ( 0.82 )	0.70 ( 0.87 )	0.86 ( 0.90 )	0.71 ( 0.83 )	0.64 ( 0.82 )	0.72 (0.88)	0.88 (0.92)	0.77 (0.90)	0.65 (0.82)
Dutch	RobBERT2023-base	0.63 (0.82)	0.86 (0.89)	0.66 (0.81)	0.58 (0.78)	0.61 (0.80)	0.84 (0.86)	0.63 (0.77)	0.59 (0.80)	0.64 (0.84)	0.89 ( 0.92 )	0.70 (0.86)	0.57 (0.76)
	CardioBERTa.nl	0.64 (0.83)	0.81 (0.85)	0.66 (0.80)	0.61 (0.78)	0.61 (0.79)	0.76 (0.80)	0.58 (0.71)	0.56 (0.72)	0.67 (0.87)	0.86 (0.90)	0.76 ( 0.92 )	0.66 ( 0.84 )

Appendix

Table A.3: Span-level NER performance of multilabel models on the CardioCCC held-out test set, broken down by entity type and averaged across the 10 fold models. Predictions are decoded using per-label sigmoid confidence scores for thresholding, matching the training objective, and then aggregated into entity spans. Because argmax label selection is invariant to sigmoid versus softmax scaling, each token receives a single label. Strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 are reported for each of the four entity types. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

Language	Model	F1	Recall	Precision
Language	Model	strict (relaxed)	strict (relaxed)	strict (relaxed)
Czech	robeczech-base	0.67 (0.85)	0.68 (0.87)	0.65 (0.84)
	CardioBERTa.cs	0.69 (0.86)	0.72 ( 0.89 )	0.67 (0.84)
	xlm-roberta-large	0.72 ( 0.88 )	0.73 ( 0.89 )	0.72 ( 0.87 )
Dutch	RobBERT2023-base	0.63 (0.85)	0.61 (0.82)	0.64 (0.85)
	CardioBERTa.nl	0.66 (0.86)	0.64 (0.83)	0.68 ( 0.89 )

Appendix

Table A.4: Span-level NER performance of multiclass models on the CardioCCC held-out test set for disease recognition, averaged across the 10 fold models. Each model is trained only on the disease entity type. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Precision, recall, and F1 are reported under strict (exact boundary) and relaxed (overlapping boundary) matching. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

Language	Model	F1	Recall	Precision
Language	Model	strict (relaxed)	strict (relaxed)	strict (relaxed)
Czech	robeczech-base	0.86 (0.91)	0.85 (0.90)	0.87 (0.92)
	CardioBERTa.cs	0.85 ( 0.92 )	0.85 ( 0.92 )	0.84 (0.91)
	xlm-roberta-large	0.87 ( 0.92 )	0.87 ( 0.92 )	0.88 ( 0.93 )
Dutch	RobBERT2023-base	0.88 (0.90)	0.87 (0.88)	0.90 ( 0.92 )
	CardioBERTa.nl	0.86 (0.90)	0.85 (0.89)	0.87 (0.91)

Appendix

Table A.5: Span-level NER performance of multiclass models on the CardioCCC held-out test set for medication recognition, averaged across the 10 fold models. Each model is trained only on the medication entity type. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Precision, recall, and F1 are reported under strict (exact boundary) and relaxed (overlapping boundary) matching. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

Language	Model	F1	Recall	Precision
Language	Model	strict (relaxed)	strict (relaxed)	strict (relaxed)
Czech	robeczech-base	0.74 (0.86)	0.72 (0.83)	0.76 (0.88)
	CardioBERTa.cs	0.75 (0.87)	0.73 (0.85)	0.77 (0.89)
	xlm-roberta-large	0.77 ( 0.88 )	0.75 ( 0.86 )	0.78 ( 0.90 )
Dutch	RobBERT2023-base	0.67 (0.82)	0.62 (0.76)	0.72 (0.88)
	CardioBERTa.nl	0.68 (0.82)	0.62 (0.75)	0.74 ( 0.90 )

Appendix

Table A.6: Span-level NER performance of multiclass models on the CardioCCC held-out test set for procedure recognition, averaged across the 10 fold models. Each model is trained only on the procedure entity type. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Precision, recall, and F1 are reported under strict (exact boundary) and relaxed (overlapping boundary) matching. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

Language	Model	F1	Recall	Precision
Language	Model	strict (relaxed)	strict (relaxed)	strict (relaxed)
Czech	robeczech-base	0.61 (0.79)	0.61 (0.79)	0.61 (0.80)
	CardioBERTa.cs	0.62 (0.80)	0.65 (0.83)	0.61 (0.77)
	xlm-roberta-large	0.68 ( 0.83 )	0.69 ( 0.85 )	0.66 ( 0.81 )
Dutch	RobBERT2023-base	0.62 (0.77)	0.63 (0.81)	0.60 (0.77)
	CardioBERTa.nl	0.65 (0.80)	0.64 (0.79)	0.65 (0.81)

Appendix

Table A.7: Span-level NER performance of multiclass models on the CardioCCC held-out test set for symptom recognition, averaged across the 10 fold models. Each model is trained only on the symptom entity type. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Precision, recall, and F1 are reported under strict (exact boundary) and relaxed (overlapping boundary) matching. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
ufal/robeczech-base	Czech	93.08 (135.79)	90.76 (154.51)
	Dutch	69.16 (34.04)	69.77 (44.00)
	English	43.09 (20.78)	40.85 (26.07)
	Italian	36.07 (24.75)	35.60 (31.30)
	Romanian	56.92 (58.68)	58.11 (67.13)

Appendix

Table A.8: Full-text (medical-span) pseudo-perplexity scores for Czech models evaluated on Czech ParaCliTe texts, broken down by document source language (cz = originally authored in Czech; all others = translated into Czech). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
ufal/robeczech-base	Czech	43.70 (19.27)	42.96 (33.03)
	Dutch	30.20 (4.79)	31.77 (9.65)
	English	18.39 (2.56)	18.13 (5.96)
	Italian	15.01 (3.28)	14.58 (7.08)
	Romanian	19.71 (6.71)	19.31 (12.18)

Appendix

Table A.9: Full-text (medical-span) pseudo-perplexity scores for Czech models evaluated on Czech ParaCliTe texts, broken down by document source language (cz = originally authored in Czech; all others = translated into Czech). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
CLTL/MedRoBERTa.nl	Czech	34.47 (74.7)	34.65 (101)
	Dutch	27.71 (8.11)	24.79 (16.2)
	English	33.23 (29.3)	31.62 (41.6)
	Italian	57.70 (207.7)	56.27 (242.4)
	Romanian	65.18 (239.8)	65.22 (259.8)

Appendix

Table A.10: Full-text (medical-span) pseudo-perplexity scores for Dutch models evaluated on Dutch ParaCliTe texts, broken down by document source language (nl = originally authored in Dutch; all others = translated into Dutch). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
CLTL/MedRoBERTa.nl	Czech	25.93 (30.2)	24.88 (51.7)
	Dutch	20.55 (4.9)	19.79 (11)
	English	21.28 (14.8)	22.51 (25.7)
	Italian	37.76 (62.5)	37.12 (107.7)
	Romanian	40.96 (78.4)	39.56 (117.7)

Appendix

Table A.11: Full-text (medical-span) pseudo-perplexity scores for Dutch models evaluated on Dutch ParaCliTe texts, broken down by document source language (nl = originally authored in Dutch; all others = translated into Dutch). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
allenai/biomed_roberta_base	Czech	25.83 (18.09)	25.42 (34.45)
	Dutch	33.15 (17.21)	36.89 (27.05)
	English	20.36 (6.28)	20.58 (10.34)
	Italian	16.67 (7.47)	16.52 (15.57)
	Romanian	27.29 (20.38)	27.23 (38.58)

Appendix

Table A.12: Full-text (medical-span) pseudo-perplexity scores for English models evaluated on English ParaCliTe texts, broken down by document source language (en = originally authored in English; all others = translated into English). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
allenai/biomed_roberta_base	Czech	16.02 (4.14)	15.34 (12.21)
	Dutch	18.10 (2.97)	21.34 (7.33)
	English	13.31 (1.50)	12.77 (3.41)
	Italian	9.22 (1.92)	9.31 (5.33)
	Romanian	14.05 (4.11)	13.57 (11.81)

Appendix

Table A.13: Full-text (medical-span) pseudo-perplexity scores for English models evaluated on English ParaCliTe texts, broken down by document source language (en = originally authored in English; all others = translated into English). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
IVN-RIN/bioBIT	Czech	6.30 (6.71)	6.01 (11.74)
	Dutch	9.49 (9.21)	9.18 (14.50)
	English	3.86 (2.83)	3.89 (4.84)
	Italian	11.14 (15.52)	11.08 (25.40)
	Romanian	8.88 (14.69)	8.99 (21.03)

Appendix

Table A.14: Full-text (medical-span) pseudo-perplexity scores for Italian models evaluated on Italian ParaCliTe texts, broken down by document source language (it = originally authored in Italian; all others = translated into Italian). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the median and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
IVN-RIN/bioBIT	Czech	3.34 (1.55)	3.29 (3.63)
	Dutch	4.60 (1.61)	4.62 (3.80)
	English	2.35 (1.11)	2.28 (1.85)
	Italian	4.87 (2.51)	4.92 (6.60)
	Romanian	4.15 (2.65)	4.10 (5.91)

Appendix

Table A.15: Full-text (medical-span) pseudo-perplexity scores for Italian models evaluated on Italian ParaCliTe texts, broken down by document source language (it = originally authored in Italian; all others = translated into Italian). Scores were computed following Salazar et al. (2020) ( our implementation ), with the median and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
FacebookAI/xlm-roberta-base	Czech	16.97 (37.82)	16.14 (52.58)
	Dutch	14.01 (32.38)	14.30 (41.19)
	English	8.22 (16.19)	7.89 (20.19)
	Italian	17.61 (62.95)	17.91 (82.48)
	Romanian	30.17 (118.19)	30.17 (131.46)

Appendix

Table A.16: Full-text (medical-span) pseudo-perplexity scores for Romanian models evaluated on Romanian ParaCliTe texts, broken down by document source language (ro = originally authored in Romanian; all others = translated into Romanian). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
FacebookAI/xlm-roberta-base	Czech	6.07 (4.42)	5.83 (11.60)
	Dutch	5.19 (3.68)	5.37 (9.00)
	English	3.47 (2.52)	3.33 (4.96)
	Italian	6.25 (6.27)	6.22 (15.48)
	Romanian	9.84 (14.80)	9.84 (26.21)

Appendix

Table A.17: Full-text (medical-span) pseudo-perplexity scores for Romanian models evaluated on Romanian ParaCliTe texts, broken down by document source language (ro = originally authored in Romanian; all others = translated into Romanian). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL
Model	Source language	Median	Geometric mean
PlanTL-GOB-ES/	Czech	34.07 (32.66)	32.87 (146.24)
roberta-base-biomedical-clinical-es	Dutch	42.32 (25.87)	51.74 (122.79)
	English	28.38 (5.69)	25.36 (23.39)
	Italian	23.17 (15.19)	25.49 (68.51)
	Romanian	25.78 (21.68)	24.49 (100.40)

Appendix

Table A.18: Full-text (medical-span) pseudo-perplexity scores for Spanish models evaluated on Spanish ParaCliTe texts, broken down by document source language (es = originally authored in Spanish; all others = translated into Spanish). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the median and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL
Model	Source language	Median	Geometric mean
PlanTL-GOB-ES/	Czech	26.81 (18.09)	25.90 (86.57)
roberta-base-biomedical-clinical-es	Dutch	32.90 (12.03)	42.22 (67.18)
	English	23.79 (3.51)	21.55 (15.73)
	Italian	16.39 (5.71)	18.34 (37.18)
	Romanian	17.97 (9.53)	17.79 (58.55)

Appendix

Table A.19: Full-text (medical-span) pseudo-perplexity scores for Spanish models evaluated on Spanish ParaCliTe texts, broken down by document source language (es = originally authored in Spanish; all others = translated into Spanish). Scores were computed following Salazar et al. (2020) ( our implementation ), with the median and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
KB/bert-base-swedish-cased	Czech	56.25 (256.15)	53.23 (241.11)
	Dutch	43.59 (156.39)	39.40 (143.75)
	English	36.39 (114.18)	32.48 (102.8)
	Italian	35.29 (178.09)	35.05 (157.31)
	Romanian	50.79 (229.7)	54.10 (239.2)

Appendix

Table A.20: Full-text (medical-span) pseudo-perplexity scores for Swedish models evaluated on Swedish ParaCliTe texts, broken down by document source language (sv = originally authored in Swedish; all others = translated into Swedish). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

Model	Source language	Full-text PPL (Medical-span PPL)
Model	Source language	Median	Geometric mean
KB/bert-base-swedish-cased	Czech	39.74 (144)	38.21 (126)
	Dutch	27.94 (62.29)	25.18 (67.46)
	English	21.80 (43.91)	20.31 (47.91)
	Italian	25.42 (92.24)	24.96 (87.25)
	Romanian	34.10 (134.04)	35.74 (135.88)

Appendix

Table A.21: Full-text (medical-span) pseudo-perplexity scores for Swedish models evaluated on Swedish ParaCliTe texts, broken down by document source language (sv = originally authored in Swedish; all others = translated into Swedish). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.

		DISEASE		MEDICATION		PROCEDURE		SYMPTOM
Language	Model	Median	Geo. mean	Median	Geo. mean	Median	Geo. mean	Median	Geo. mean
Czech	robeczech-base	22.32	27.97	187.03	191.19	32.05	41.36	45.45	58.37
Czech	CardioBERTa.cs	5.23	8.40	39.69	51.40	7.57	11.90	11.39	17.12
Dutch	RobBERT2023-base	38.64	44.6	180.78	165.6	48.5	51.5	59.1	66.6
	MedRoBERTa.nl	66.01	87.4	232.9	277.70	58.8	85.1	33.75	53.2
	CardioBERTa.nl_clinical	18.04	29.7	71.9	111.9	13.4	21.5	12.5	22.6

Appendix

Table A.22: Medical-span pseudo-perplexity on the ParaCliTe dataset, reported per entity type for each language and model. Scores are computed following Kauf and Ivanova (2023) using Minicons .

		DISEASE		MEDICATION		PROCEDURE		SYMPTOM
Language	Model	Median	Geo. mean	Median	Geo. mean	Median	Geo. mean	Median	Geo. mean
Czech	robeczech-base	2.63	5.68	46.03	48.53	3.53	7.54	6.46	13.31
Czech	CardioBERTa.cs	1.27	2.68	3.89	11.21	1.39	3.18	2.62	5.75
Dutch	RobBERT2023-base	5.379	10.471	48.811	52.573	7.924	13.004	14.570	23.195
	MedRoBERTa.nl	26.855	45.383	57.573	70.015	22.299	38.932	21.042	36.950
	CardioBERTa.nl_clinical	7.962	16.798	22.286	41.282	5.851	12.115	7.580	15.788

Appendix

Table A.23: Medical-span pseudo-perplexity on the ParaCliTe dataset, reported per entity type for each language and model. Scores are computed following Salazar et al. (2020) using our implementation .

Language	Corpus name	Corpus type	Words (M)
Czech	EMEA CS	Biomedical	12.21
Czech	Wikipedia Life Science Cs	Biomedical	112.01
Czech	Wikipedia Cardiology Cs	Cardiology	0.10
Czech	ABSTRACTS_CS	General	0.21
Czech	CorEtVasa	Cardiology	10.09
Dutch	BioLORD definitions	Biomedical	300.00

Appendix

Table B.1: Overview of collected corpora. Word counts are expressed in millions.

Clinical NLP

Clinical natural language processing (NLP) develops methods to extract, interpret, and generate information from healthcare text and conversations.

See all 145 papers

Multilingual Artificial Datasets for AI-Driven Cardiology: the CliTe and ParaCliTe corpora
Nov 6, 2026Lorenzo Pratesi, Eulalia Farre-Maduell, Machteld Boonstra +21Synthetic Clinical Data Generation Clinical NLP
Translated SRL, Rome, Italy · Barcelona Supercomputing Center, Barcelona, Spain · Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands +14
CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios
Jul 28, 2026Xiao Li, Mouxiao Bian, Zhaodi Wu +8Healthcare Clinical Language Model Evaluation
Department of Cardiology, Zhongshan Hospital Fudan University , Shanhai, China · Shanghai Institute of Cardiovascular Diseases, National Clinical , Shanghai, China · Research Center for Interventional Medicine , Shanghai, China +2
ClinicalEncoder26AM: A Multlilingual Diagnosable ColBERT Model; Evidences from the MultiClinNER Shared Task
May 27, 2026François RemyNamed Entity Recognition Healthcare
Parallia AI

Nov 6, 2026J/K move · Enter open · S save

Multilingual Artificial Datasets for AI-Driven Cardiology: the CliTe and ParaCliTe corpora

Lorenzo Pratesi, Eulalia Farre-Maduell, Machteld Boonstra, Michele Miranda+20

Translated SRL, Rome, Italy · Barcelona Supercomputing Center, Barcelona, Spain · Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands+14

Synthetic Clinical Data Generation Clinical NLP

Open paper AskPDF arXiv

Background and Aims. Comparable, multilingual, validated cardiology corpora are critical to advance trustworthy natural language processing (NLP) models for cardiology but such resources are scarce due to privacy concerns. This study aimed to cover this gap by presenting suitable artificial corpus creation guidelines and by releasing 3 multilanguage corpora. Methods. Following purpose-specific publicly-available corpus construction guidelines, cardiologists from 7 European hospitals generated artificial clinical records in Czech, Dutch, English, Italian, Romanian, Spanish and Swedish, ensuring clinical diversity and sex balance, resulting in the CliTe corpus. A CliTe subset underwent a multi-stage manual translation into the other 6 languages, resulting in the parallel ParaCliTe corpus, while the remaining reports were automatically translated, constituting ParaCliTe Silver. Results. CliTe is a collection of 699 artificial cardiology reports, mostly outpatient notes (340 documents), discharge reports (263), and inpatient notes (89), describing fictional patients with ischaemic heart disease (199 patients), heart failure (188), and valvular disease (79), among others. ParaCliTe contains reports from 94 fictitious patients, each available in seven languages with sentence-level alignment, resulting in 658 documents. The corpus mainly comprises discharge reports (322 documents), and outpatient (196) and inpatient (140) notes, describing patients with heart failure (399 documents), ischaemic heart disease (287), and valvular disease (238). Empirical results illustrate the utility of ParaCliTe and highlight the limitations of state-of-the-art NLP models adapted to the cardiology domain. Conclusions. The creation of multilingual, equitable and realistic corpora of artificial patients capable of strengthening NLP performance for cardiology is feasible when supported by detailed guidelines.

Abstract

Clinical NLP

Explore similar work

Multilingual Artificial Datasets for AI-Driven Cardiology: the CliTe and ParaCliTe corpora

CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios

ClinicalEncoder26AM: A Multlilingual Diagnosable ColBERT Model; Evidences from the MultiClinNER Shared Task