cs.CLNov 5, 2026

CardioLM: A multilingual suite of small language models for the cardiology domain

Authors: Manuela-Daniela Danu, Miguel Rodríguez-Ortega, Alberto Becerra-Tome, Pablo Arancibia, Fernando Gallego-Donoso, Lorenzo Pratesi, Xi Bai, Andrei-Damian Dinescu, 

Organizations: Siemens SRL, Brasov, Romania. · Automation and Information Technology, Transilvania University of Brasov, Brasov, Romania. · NLP4BIA, Barcelona Supercomputing Center, Barcelona, Spain. · Translated, Rome, Italy. · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Background: Cardiovascular care generates large volumes of clinical free-text documentation in local languages. However, clinical natural language processing (NLP) resources remain concentrated in English, and no existing benchmark is cardiology-specific, multilingual, and broad in entity coverage. Methods: We present two contributions. The first is CardioCCC, a corpus of 508 cardiology case reports originally published in Spanish and extended to Czech, Dutch, English, Italian, Romanian, and Swedish through machine translation, annotation projection, and clinician review, with annotations covering diseases, symptoms, procedures, and medications. The second is CardioLM, comprising CardioBERTa, a set of language-specific encoders obtained by continued pretraining on cardiology and biomedical corpora, and CardioNER, a set of named entity recognition (NER) models fine-tuned on CardioCCC. Results: Cardiology-specific pretraining reduced pseudo-perplexity by up to a factor of eight, yet changed strict span-level F1 by at most 0.04, with benefits concentrated in languages whose backbones lacked biomedical coverage. The jointly trained multilingual XLM-RoBERTa-large models achieved the best strict F1 in nearly every language and entity type, exceeding the strongest monolingual system by up to 0.13 F1. A large share of residual errors concerned boundary detection: relaxed matching recovered 0.10-0.23 F1 for disease, procedure, and symptom, but only 0.01-0.07 for medication. Clinicians rated CardioNER well above a rule- and dictionary-based baseline, with the mean rating across entity types rising from 26% to 73%. Conclusions: Specialty-specific pretraining is not the decisive factor for downstream entity recognition, whereas multilingual joint training yields the strongest results in nearly all settings and a simpler deployment path for multi-country clinical information extraction.

Figures & tables

Figure 1: End-to-end pipeline from cardiology-domain continued pretraining to NER system development.
LanguageCardiologyBiomedicalGeneralTotal
Czech1502,3304032,883
Dutch1,4004,07305,473
English7718163942
Italian2664,56204,828
Romanian1844,21104,395
Spanish2073,57803,785
Table 1: Size of the continued pretraining corpus per language, expressed in millions of words.
Dataset characteristics
Total word count (source; translations; total)51,079; 343,368; 394,447
Total number of documents658
Demographics
Age (average; range)58.2; 20–110
Female sex (n; %)343 (52%)
Document types
Table 2: Characteristics of the ParaCliTe dataset.
Figure 2: Illustrative English CardioCCC sample showing projected entity annotations for diseases, symptoms and clinical findings, procedures, and medications prior to manual clinical review.
LanguageDISEASEMEDICATIONPROCEDURESYMPTOMTOTAL
Czech17,7884,29016,47718,72957,284
Dutch17,6464,31716,19518,59056,748
English17,0974,23115,70818,69455,730
Italian18,1754,38516,52119,46758,548
Romanian17,5464,23916,37918,06856,232
Spanish18,2324,22717,07319,98359,515
Table 3: Statistics of the multilingual, annotated CardioCCC dataset.
Figure 3: Illustrative comparison of the lexical baseline (left) and the Transformer-based CardioNER system (right) on an English ParaCliTe report.
LanguageBase-model nameDomainArchitecturePretraining taskModel size (# trainable parameters)
Czechufal/robeczech-basegenericRoBERTaMLM125M
Straka et al. (2021)
DutchCLTL/MedRoBERTa.nlbiomedicalRoBERTaMLM125M
Verkijk and Vossen (2021)
Englishallenai/biomed_roberta_basebiomedicalRoBERTaMLM125M
Gururangan et al. (2020)
Table 4: Overview of the base models selected for language-specific continued pretraining.
Figure 4: Shard-wise continued pretraining: the checkpoint produced on each corpus shard initialises training on the next, until all shards have been consumed.
Figure 5: Two-stage token-to-span decoding, comprising (1) token-level BIO prediction and (2) span estimation and post-processing.
LanguageModelPPL minicons median(geometric mean)PPL ours median(geometric mean)
Czechrobeczech-base56.49 (56.17)22.68 (23.77)
CardioBERTa.cs6.94 ( 6.96 )3.22 ( 3.53 )
DutchRobBERT2023-base12.70 (12.50)6.40 ( 6.20 )
MedRoBERTa.nl44.70 (45.40)30.70 (30.60)
CardioBERTa.nl9.80 ( 9.40 )6.80 (6.80)
Englishbiomed_roberta_base29.76 (29.88)16.13 (16.69)
Table 5: Pseudo-perplexity aggregation over all spans .
LanguageModelPPL minicons median(geometric mean)PPL ours median(geometric mean)
Czechrobeczech-base38.88 (49.41)4.90 (10.42)
CardioBERTa.cs9.09 ( 14.37 )1.71 ( 4.25 )
DutchRobBERT2023-base55.4 (62.1)9.8 (17.4)
MedRoBERTa.nl65.2 (87.5)24.9 (43.6)
CardioBERTa.nl17.6 ( 30.3 )8.0 ( 17.2 )
Englishbiomed_roberta_base12.77 (21.64)2.43 (6.69)
Table 6: Pseudo-perplexity aggregation over all medical spans .
 MicroMacro
LanguageModelF1RecallPrecisionF1RecallPrecision
strict (relaxed)strict (relaxed)strict (relaxed)strict (relaxed)strict (relaxed)strict (relaxed)
Czechrobeczech-base0.68 (0.84)0.67 (0.82)0.70 (0.85)0.72 (0.85)0.70 (0.83)0.73 (0.87)
CardioBERTa.cs0.70 (0.84)0.68 (0.83)0.71 ( 0.86 )0.72 (0.86)0.71 (0.84)0.74 ( 0.88 )
xlm-roberta-large0.71 ( 0.86 )0.70 ( 0.85 )0.71 ( 0.86 )0.74 ( 0.87 )0.73 ( 0.86 )0.74 ( 0.88 )
DutchRobBERT2023-base0.63 (0.81)0.64 (0.82)0.63 (0.80)0.68 (0.83)0.68 (0.83)0.68 (0.82)
Table 7: Span-level NER performance of multilabel models on the CardioCCC held-out test set, averaged across the 10 fold models. Predictions are decoded using a single-label protocol (argmax over softmax-normalised logits ) and aggregated into entity spans. Micro- and macro-averages are computed over the four entity types, with strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 reported. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
  F1 strict (relaxed)Recall strict (relaxed)Precision strict (relaxed)
LanguageModelDisMedProcSymDisMedProcSymDisMedProcSym
Czechrobeczech-base0.67 (0.86)0.86 (0.90)0.72 (0.83)0.62 (0.80)0.67 (0.85)0.83 (0.87)0.69 (0.80)0.61 (0.79)0.68 (0.86)0.89 ( 0.94 )0.75 (0.87)0.62 (0.80)
CardioBERTa.cs0.70 ( 0.87 )0.84 (0.90)0.73 (0.85)0.63 (0.81)0.70 (0.87)0.83 (0.88)0.69 (0.80)0.63 (0.80)0.70 ( 0.87 )0.86 (0.92)0.77 ( 0.89 )0.64 (0.81)
xlm-roberta-large0.71 ( 0.87 )0.86 ( 0.91 )0.74 ( 0.86 )0.64 ( 0.82 )0.71 ( 0.88 )0.86 ( 0.91 )0.72 ( 0.84 )0.64 ( 0.83 )0.71 ( 0.87 )0.87 (0.92)0.77 ( 0.89 )0.63 ( 0.82 )
DutchRobBERT2023-base0.62 (0.82)0.87 (0.89)0.65 (0.81)0.57 (0.78)0.62 (0.83)0.85 (0.87)0.64 (0.80)0.61 (0.82)0.62 (0.82)0.88 (0.90)0.67 (0.83)0.54 (0.73)
CardioBERTa.nl0.64 (0.84)0.83 (0.87)0.67 (0.82)0.61 (0.79)0.65 (0.85)0.82 (0.86)0.61 (0.76)0.59 (0.78)0.63 (0.83)0.84 (0.88)0.72 ( 0.90 )0.61 (0.80)
Table 8: Span-level NER performance of multilabel models on the CardioCCC held-out test set, broken down by entity type and averaged across the 10 fold models. Predictions are decoded using a single-label protocol (argmax over softmax-normalised logits ) and aggregated into entity spans. Strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 are reported for each of the four entity types. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
  F1 strict (relaxed)Recall strict (relaxed)Precision strict (relaxed)
LanguageModelDisMedProcSymDisMedProcSymDisMedProcSym
Czechrobeczech-base0.67 (0.85)0.86 (0.91)0.74 (0.86)0.61 (0.79)0.68 (0.87)0.85 (0.90)0.72 (0.83)0.61 (0.79)0.65 (0.84)0.87 (0.92)0.76 (0.88)0.61 (0.80)
CardioBERTa.cs0.69 (0.86)0.85 ( 0.92 )0.75 (0.87)0.62 (0.80)0.72 ( 0.89 )0.85 ( 0.92 )0.73 (0.85)0.65 (0.83)0.67 (0.84)0.84 (0.91)0.77 (0.89)0.61 (0.77)
xlm-roberta-large0.72 ( 0.88 )0.87 ( 0.92 )0.77 ( 0.88 )0.68 ( 0.83 )0.73 ( 0.89 )0.87 ( 0.92 )0.75 ( 0.86 )0.69 ( 0.85 )0.72 ( 0.87 )0.88 ( 0.93 )0.78 ( 0.90 )0.66 ( 0.81 )
DutchRobBERT2023-base0.63 (0.85)0.88 (0.90)0.67 (0.82)0.62 (0.77)0.61 (0.82)0.87 (0.88)0.62 (0.76)0.63 (0.81)0.64 (0.85)0.90 ( 0.92 )0.72 (0.88)0.60 (0.77)
CardioBERTa.nl0.66 (0.86)0.86 (0.90)0.68 (0.82)0.65 (0.80)0.64 (0.83)0.85 (0.89)0.62 (0.75)0.64 (0.79)0.68 ( 0.89 )0.87 (0.91)0.74 ( 0.90 )0.65 (0.81)
Table 9: Span-level NER performance of multiclass models on the CardioCCC held-out test set, averaged across the 10 fold models. A separate model is fine-tuned and evaluated for each entity type; thus, results for the four entity types come from four independently trained models rather than a single multi-entity model. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 are reported for each entity type. XLM-RoBERTa-large is trained jointly on all seven languages as four separate entity-specific models and evaluated on the full multilingual test set and separately by language.
Figure 6: Mean clinician ratings for the baseline and Transformer-based NER systems, reported for each entity type (disease, symptom, procedure, and medication) and averaged across entity types.
Figure 7: Clinician-reported usefulness of baseline and Transformer-based clinical entity extraction systems for potential hospital-oriented implementation.
Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

LanguageModelF1RecallPrecision
macro (micro)macro (micro)macro (micro)
Czechrobeczech-base0.84 (0.83)0.83 (0.82)0.85 (0.83)
CardioBERTa.cs0.85 ( 0.84 )0.85 (0.84)0.86 ( 0.84 )
xlm-roberta-large0.85 ( 0.84 )0.86 ( 0.85 )0.85 ( 0.84 )
DutchRobBERT2023-base0.80 (0.79)0.81 (0.80)0.80 (0.79)
CardioBERTa.nl0.79 (0.77)0.78 (0.76)0.80 (0.77)
Appendix
Table A.1: Token-level NER performance of multilabel models on the CardioCCC held-out test set, averaged across the 10 fold models. Averages are computed over the eight B-/I- labels, with the O class excluded from both micro- and macro-averages. Labels are decoded independently using sigmoid ( >0.5>0.5 ), allowing a token to receive multiple labels or none. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
 MicroMacro
LanguageModelF1RecallPrecisionF1RecallPrecision
strict (relaxed)strict (relaxed)strict (relaxed)strict (relaxed)strict (relaxed)strict (relaxed)
Czechrobeczech-base0.68 (0.83)0.66 (0.80)0.71 (0.87)0.72 (0.85)0.69 (0.81)0.75 (0.88)
CardioBERTa.cs0.70 (0.84)0.67 (0.81)0.73 ( 0.88 )0.73 (0.85)0.70 (0.82)0.76 ( 0.89 )
xlm-roberta-large0.71 ( 0.86 )0.70 ( 0.84 )0.72 (0.87)0.74 ( 0.87 )0.73 ( 0.85 )0.75 (0.88)
DutchRobBERT2023-base0.64 (0.81)0.63 (0.80)0.65 (0.82)0.68 (0.83)0.67 (0.81)0.70 (0.84)
Appendix
Table A.2: Span-level NER performance of multilabel models on the CardioCCC held-out test set, averaged across the 10 fold models. Predictions are decoded using per-label sigmoid confidence scores for thresholding, matching the training objective, and then aggregated into entity spans. Because argmax label selection is invariant to sigmoid versus softmax scaling, each token receives a single label. Micro- and macro-averages are computed over the four entity types, with strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 reported. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
  F1 strict (relaxed)Recall strict (relaxed)Precision strict (relaxed)
LanguageModelDisMedProcSymDisMedProcSymDisMedProcSym
Czechrobeczech-base0.68 (0.86)0.85 (0.89)0.72 (0.84)0.62 (0.79)0.66 (0.83)0.80 (0.85)0.68 (0.79)0.60 (0.77)0.69 (0.88)0.90 ( 0.95 )0.77 (0.89)0.65 (0.82)
CardioBERTa.cs0.70 ( 0.87 )0.84 (0.89)0.73 (0.85)0.64 (0.80)0.69 (0.85)0.81 (0.86)0.68 (0.79)0.61 (0.76)0.72 ( 0.89 )0.87 (0.93)0.79 ( 0.91 )0.67 ( 0.83 )
xlm-roberta-large0.71 ( 0.87 )0.87 ( 0.91 )0.74 ( 0.87 )0.64 ( 0.82 )0.70 ( 0.87 )0.86 ( 0.90 )0.71 ( 0.83 )0.64 ( 0.82 )0.72 (0.88)0.88 (0.92)0.77 (0.90)0.65 (0.82)
DutchRobBERT2023-base0.63 (0.82)0.86 (0.89)0.66 (0.81)0.58 (0.78)0.61 (0.80)0.84 (0.86)0.63 (0.77)0.59 (0.80)0.64 (0.84)0.89 ( 0.92 )0.70 (0.86)0.57 (0.76)
CardioBERTa.nl0.64 (0.83)0.81 (0.85)0.66 (0.80)0.61 (0.78)0.61 (0.79)0.76 (0.80)0.58 (0.71)0.56 (0.72)0.67 (0.87)0.86 (0.90)0.76 ( 0.92 )0.66 ( 0.84 )
Appendix
Table A.3: Span-level NER performance of multilabel models on the CardioCCC held-out test set, broken down by entity type and averaged across the 10 fold models. Predictions are decoded using per-label sigmoid confidence scores for thresholding, matching the training objective, and then aggregated into entity spans. Because argmax label selection is invariant to sigmoid versus softmax scaling, each token receives a single label. Strict (exact boundary) and relaxed (overlapping boundary) precision, recall, and F1 are reported for each of the four entity types. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
LanguageModelF1RecallPrecision
strict (relaxed)strict (relaxed)strict (relaxed)
Czechrobeczech-base0.67 (0.85)0.68 (0.87)0.65 (0.84)
CardioBERTa.cs0.69 (0.86)0.72 ( 0.89 )0.67 (0.84)
xlm-roberta-large0.72 ( 0.88 )0.73 ( 0.89 )0.72 ( 0.87 )
DutchRobBERT2023-base0.63 (0.85)0.61 (0.82)0.64 (0.85)
CardioBERTa.nl0.66 (0.86)0.64 (0.83)0.68 ( 0.89 )
Appendix
Table A.4: Span-level NER performance of multiclass models on the CardioCCC held-out test set for disease recognition, averaged across the 10 fold models. Each model is trained only on the disease entity type. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Precision, recall, and F1 are reported under strict (exact boundary) and relaxed (overlapping boundary) matching. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
LanguageModelF1RecallPrecision
strict (relaxed)strict (relaxed)strict (relaxed)
Czechrobeczech-base0.86 (0.91)0.85 (0.90)0.87 (0.92)
CardioBERTa.cs0.85 ( 0.92 )0.85 ( 0.92 )0.84 (0.91)
xlm-roberta-large0.87 ( 0.92 )0.87 ( 0.92 )0.88 ( 0.93 )
DutchRobBERT2023-base0.88 (0.90)0.87 (0.88)0.90 ( 0.92 )
CardioBERTa.nl0.86 (0.90)0.85 (0.89)0.87 (0.91)
Appendix
Table A.5: Span-level NER performance of multiclass models on the CardioCCC held-out test set for medication recognition, averaged across the 10 fold models. Each model is trained only on the medication entity type. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Precision, recall, and F1 are reported under strict (exact boundary) and relaxed (overlapping boundary) matching. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
LanguageModelF1RecallPrecision
strict (relaxed)strict (relaxed)strict (relaxed)
Czechrobeczech-base0.74 (0.86)0.72 (0.83)0.76 (0.88)
CardioBERTa.cs0.75 (0.87)0.73 (0.85)0.77 (0.89)
xlm-roberta-large0.77 ( 0.88 )0.75 ( 0.86 )0.78 ( 0.90 )
DutchRobBERT2023-base0.67 (0.82)0.62 (0.76)0.72 (0.88)
CardioBERTa.nl0.68 (0.82)0.62 (0.75)0.74 ( 0.90 )
Appendix
Table A.6: Span-level NER performance of multiclass models on the CardioCCC held-out test set for procedure recognition, averaged across the 10 fold models. Each model is trained only on the procedure entity type. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Precision, recall, and F1 are reported under strict (exact boundary) and relaxed (overlapping boundary) matching. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
LanguageModelF1RecallPrecision
strict (relaxed)strict (relaxed)strict (relaxed)
Czechrobeczech-base0.61 (0.79)0.61 (0.79)0.61 (0.80)
CardioBERTa.cs0.62 (0.80)0.65 (0.83)0.61 (0.77)
xlm-roberta-large0.68 ( 0.83 )0.69 ( 0.85 )0.66 ( 0.81 )
DutchRobBERT2023-base0.62 (0.77)0.63 (0.81)0.60 (0.77)
CardioBERTa.nl0.65 (0.80)0.64 (0.79)0.65 (0.81)
Appendix
Table A.7: Span-level NER performance of multiclass models on the CardioCCC held-out test set for symptom recognition, averaged across the 10 fold models. Each model is trained only on the symptom entity type. Predictions are decoded using argmax over softmax-normalised logits and aggregated into entity spans. Precision, recall, and F1 are reported under strict (exact boundary) and relaxed (overlapping boundary) matching. XLM-RoBERTa-large is trained jointly on all seven languages and evaluated on the full multilingual test set and separately by language.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
ufal/robeczech-baseCzech93.08 (135.79)90.76 (154.51)
Dutch69.16 (34.04)69.77 (44.00)
English43.09 (20.78)40.85 (26.07)
Italian36.07 (24.75)35.60 (31.30)
Romanian56.92 (58.68)58.11 (67.13)
Appendix
Table A.8: Full-text (medical-span) pseudo-perplexity scores for Czech models evaluated on Czech ParaCliTe texts, broken down by document source language (cz = originally authored in Czech; all others = translated into Czech). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
ufal/robeczech-baseCzech43.70 (19.27)42.96 (33.03)
Dutch30.20 (4.79)31.77 (9.65)
English18.39 (2.56)18.13 (5.96)
Italian15.01 (3.28)14.58 (7.08)
Romanian19.71 (6.71)19.31 (12.18)
Appendix
Table A.9: Full-text (medical-span) pseudo-perplexity scores for Czech models evaluated on Czech ParaCliTe texts, broken down by document source language (cz = originally authored in Czech; all others = translated into Czech). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
CLTL/MedRoBERTa.nlCzech34.47 (74.7)34.65 (101)
Dutch27.71 (8.11)24.79 (16.2)
English33.23 (29.3)31.62 (41.6)
Italian57.70 (207.7)56.27 (242.4)
Romanian65.18 (239.8)65.22 (259.8)
Appendix
Table A.10: Full-text (medical-span) pseudo-perplexity scores for Dutch models evaluated on Dutch ParaCliTe texts, broken down by document source language (nl = originally authored in Dutch; all others = translated into Dutch). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
CLTL/MedRoBERTa.nlCzech25.93 (30.2)24.88 (51.7)
Dutch20.55 (4.9)19.79 (11)
English21.28 (14.8)22.51 (25.7)
Italian37.76 (62.5)37.12 (107.7)
Romanian40.96 (78.4)39.56 (117.7)
Appendix
Table A.11: Full-text (medical-span) pseudo-perplexity scores for Dutch models evaluated on Dutch ParaCliTe texts, broken down by document source language (nl = originally authored in Dutch; all others = translated into Dutch). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
allenai/biomed_roberta_baseCzech25.83 (18.09)25.42 (34.45)
Dutch33.15 (17.21)36.89 (27.05)
English20.36 (6.28)20.58 (10.34)
Italian16.67 (7.47)16.52 (15.57)
Romanian27.29 (20.38)27.23 (38.58)
Appendix
Table A.12: Full-text (medical-span) pseudo-perplexity scores for English models evaluated on English ParaCliTe texts, broken down by document source language (en = originally authored in English; all others = translated into English). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
allenai/biomed_roberta_baseCzech16.02 (4.14)15.34 (12.21)
Dutch18.10 (2.97)21.34 (7.33)
English13.31 (1.50)12.77 (3.41)
Italian9.22 (1.92)9.31 (5.33)
Romanian14.05 (4.11)13.57 (11.81)
Appendix
Table A.13: Full-text (medical-span) pseudo-perplexity scores for English models evaluated on English ParaCliTe texts, broken down by document source language (en = originally authored in English; all others = translated into English). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
IVN-RIN/bioBITCzech6.30 (6.71)6.01 (11.74)
Dutch9.49 (9.21)9.18 (14.50)
English3.86 (2.83)3.89 (4.84)
Italian11.14 (15.52)11.08 (25.40)
Romanian8.88 (14.69)8.99 (21.03)
Appendix
Table A.14: Full-text (medical-span) pseudo-perplexity scores for Italian models evaluated on Italian ParaCliTe texts, broken down by document source language (it = originally authored in Italian; all others = translated into Italian). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the median and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
IVN-RIN/bioBITCzech3.34 (1.55)3.29 (3.63)
Dutch4.60 (1.61)4.62 (3.80)
English2.35 (1.11)2.28 (1.85)
Italian4.87 (2.51)4.92 (6.60)
Romanian4.15 (2.65)4.10 (5.91)
Appendix
Table A.15: Full-text (medical-span) pseudo-perplexity scores for Italian models evaluated on Italian ParaCliTe texts, broken down by document source language (it = originally authored in Italian; all others = translated into Italian). Scores were computed following Salazar et al. (2020) ( our implementation ), with the median and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
FacebookAI/xlm-roberta-baseCzech16.97 (37.82)16.14 (52.58)
Dutch14.01 (32.38)14.30 (41.19)
English8.22 (16.19)7.89 (20.19)
Italian17.61 (62.95)17.91 (82.48)
Romanian30.17 (118.19)30.17 (131.46)
Appendix
Table A.16: Full-text (medical-span) pseudo-perplexity scores for Romanian models evaluated on Romanian ParaCliTe texts, broken down by document source language (ro = originally authored in Romanian; all others = translated into Romanian). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
FacebookAI/xlm-roberta-baseCzech6.07 (4.42)5.83 (11.60)
Dutch5.19 (3.68)5.37 (9.00)
English3.47 (2.52)3.33 (4.96)
Italian6.25 (6.27)6.22 (15.48)
Romanian9.84 (14.80)9.84 (26.21)
Appendix
Table A.17: Full-text (medical-span) pseudo-perplexity scores for Romanian models evaluated on Romanian ParaCliTe texts, broken down by document source language (ro = originally authored in Romanian; all others = translated into Romanian). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL
MedianGeometric mean
PlanTL-GOB-ES/Czech34.07 (32.66)32.87 (146.24)
roberta-base-biomedical-clinical-esDutch42.32 (25.87)51.74 (122.79)
English28.38 (5.69)25.36 (23.39)
Italian23.17 (15.19)25.49 (68.51)
Romanian25.78 (21.68)24.49 (100.40)
Appendix
Table A.18: Full-text (medical-span) pseudo-perplexity scores for Spanish models evaluated on Spanish ParaCliTe texts, broken down by document source language (es = originally authored in Spanish; all others = translated into Spanish). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the median and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL
MedianGeometric mean
PlanTL-GOB-ES/Czech26.81 (18.09)25.90 (86.57)
roberta-base-biomedical-clinical-esDutch32.90 (12.03)42.22 (67.18)
English23.79 (3.51)21.55 (15.73)
Italian16.39 (5.71)18.34 (37.18)
Romanian17.97 (9.53)17.79 (58.55)
Appendix
Table A.19: Full-text (medical-span) pseudo-perplexity scores for Spanish models evaluated on Spanish ParaCliTe texts, broken down by document source language (es = originally authored in Spanish; all others = translated into Spanish). Scores were computed following Salazar et al. (2020) ( our implementation ), with the median and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
KB/bert-base-swedish-casedCzech56.25 (256.15)53.23 (241.11)
Dutch43.59 (156.39)39.40 (143.75)
English36.39 (114.18)32.48 (102.8)
Italian35.29 (178.09)35.05 (157.31)
Romanian50.79 (229.7)54.10 (239.2)
Appendix
Table A.20: Full-text (medical-span) pseudo-perplexity scores for Swedish models evaluated on Swedish ParaCliTe texts, broken down by document source language (sv = originally authored in Swedish; all others = translated into Swedish). Scores were computed following Kauf and Ivanova (2023) ( Minicons ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
ModelSource languageFull-text PPL (Medical-span PPL)
MedianGeometric mean
KB/bert-base-swedish-casedCzech39.74 (144)38.21 (126)
Dutch27.94 (62.29)25.18 (67.46)
English21.80 (43.91)20.31 (47.91)
Italian25.42 (92.24)24.96 (87.25)
Romanian34.10 (134.04)35.74 (135.88)
Appendix
Table A.21: Full-text (medical-span) pseudo-perplexity scores for Swedish models evaluated on Swedish ParaCliTe texts, broken down by document source language (sv = originally authored in Swedish; all others = translated into Swedish). Scores were computed following Salazar et al. (2020) ( our implementation ), with the mean, median, and geometric mean aggregated across documents within each source-language group. Lower values indicate a better model fit to the text.
 DISEASEMEDICATIONPROCEDURESYMPTOM
LanguageModelMedianGeo. meanMedianGeo. meanMedianGeo. meanMedianGeo. mean
Czechrobeczech-base22.3227.97187.03191.1932.0541.3645.4558.37
CardioBERTa.cs5.238.4039.6951.407.5711.9011.3917.12
DutchRobBERT2023-base38.6444.6180.78165.648.551.559.166.6
MedRoBERTa.nl66.0187.4232.9277.7058.885.133.7553.2
CardioBERTa.nl_clinical18.0429.771.9111.913.421.512.522.6
Appendix
Table A.22: Medical-span pseudo-perplexity on the ParaCliTe dataset, reported per entity type for each language and model. Scores are computed following Kauf and Ivanova (2023) using Minicons .
 DISEASEMEDICATIONPROCEDURESYMPTOM
LanguageModelMedianGeo. meanMedianGeo. meanMedianGeo. meanMedianGeo. mean
Czechrobeczech-base2.635.6846.0348.533.537.546.4613.31
CardioBERTa.cs1.272.683.8911.211.393.182.625.75
DutchRobBERT2023-base5.37910.47148.81152.5737.92413.00414.57023.195
MedRoBERTa.nl26.85545.38357.57370.01522.29938.93221.04236.950
CardioBERTa.nl_clinical7.96216.79822.28641.2825.85112.1157.58015.788
Appendix
Table A.23: Medical-span pseudo-perplexity on the ParaCliTe dataset, reported per entity type for each language and model. Scores are computed following Salazar et al. (2020) using our implementation .
LanguageCorpus nameCorpus typeWords (M)
CzechEMEA CSBiomedical12.21
CzechWikipedia Life Science CsBiomedical112.01
CzechWikipedia Cardiology CsCardiology0.10
CzechABSTRACTS_CSGeneral0.21
CzechCorEtVasaCardiology10.09
DutchBioLORD definitionsBiomedical300.00
Appendix
Table B.1: Overview of collected corpora. Word counts are expressed in millions.
More in this topic

Clinical NLP

Clinical natural language processing (NLP) develops methods to extract, interpret, and generate information from healthcare text and conversations.

See all 145 papers

Explore similar work

CardsList
Nov 6, 2026

Multilingual Artificial Datasets for AI-Driven Cardiology: the CliTe and ParaCliTe corpora

Background and Aims. Comparable, multilingual, validated cardiology corpora are critical to advance trustworthy natural language processing (NLP) models for cardiology but such resources are scarce due to privacy concerns. This study aimed to cover this gap by presenting suitable artificial corpus creation guidelines and by releasing 3 multilanguage corpora. Methods. Following purpose-specific publicly-available corpus construction guidelines, cardiologists from 7 European hospitals generated artificial clinical records in Czech, Dutch, English, Italian, Romanian, Spanish and Swedish, ensuring clinical diversity and sex balance, resulting in the CliTe corpus. A CliTe subset underwent a multi-stage manual translation into the other 6 languages, resulting in the parallel ParaCliTe corpus, while the remaining reports were automatically translated, constituting ParaCliTe Silver. Results. CliTe is a collection of 699 artificial cardiology reports, mostly outpatient notes (340 documents), discharge reports (263), and inpatient notes (89), describing fictional patients with ischaemic heart disease (199 patients), heart failure (188), and valvular disease (79), among others. ParaCliTe contains reports from 94 fictitious patients, each available in seven languages with sentence-level alignment, resulting in 658 documents. The corpus mainly comprises discharge reports (322 documents), and outpatient (196) and inpatient (140) notes, describing patients with heart failure (399 documents), ischaemic heart disease (287), and valvular disease (238). Empirical results illustrate the utility of ParaCliTe and highlight the limitations of state-of-the-art NLP models adapted to the cardiology domain. Conclusions. The creation of multilingual, equitable and realistic corpora of artificial patients capable of strengthening NLP performance for cardiology is feasible when supported by detailed guidelines.

Lorenzo Pratesi, Eulalia Farre-Maduell, Machteld Boonstra +21

Translated SRL, Rome, Italy · Barcelona Supercomputing Center, Barcelona, Spain · Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands +14

Synthetic Clinical Data GenerationClinical NLP
AskPDF
Jul 28, 2026

CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios

Background: Most medical large language model (LLM) benchmarks focus on examination knowledge or isolated tasks and may not reflect the longitudinal, multimodal, and safety-critical workflow of cardiovascular care. Objective: To develop CardioBench, a real-world benchmark spanning the cardiovascular care continuum, and assess LLM performance across clinical dimensions and specialist tasks. Methods: CardioBench includes 2,263 items from 13 task-specific datasets derived from de-identified cardiovascular records and examination data. Sixteen cardiology physicians conducted annotation and reference construction, followed by cross-review from two senior cardiologists. Seven LLMs generated 15,841 outputs under standardized zero-shot settings. Open-ended tasks were evaluated using key-point coverage and holistic clinical quality, while CardioEthics was scored by accuracy. Results: GPT-5.4 achieved the highest macro-average (62.55) and item-weighted mean (62.19), followed by Gemini 3.1 Pro (59.95) and Qwen 3.6 27B (59.72). GPT-5.4 ranked first in all three dimensions. CardioAuxReport performed best (86.38), whereas CardioECGRead (17.25) and CardioEthics (17.34) were lowest. The largest gaps between holistic clinical quality and key-point coverage occurred in CardioComm (52.71), CardioEmergRescue (52.05), and CardioTreatPlan (48.80). Conclusions: To our knowledge, CardioBench is the largest real-world, multi-task benchmark for LLM evaluation across the cardiovascular care continuum and offers the broadest coverage of clinically authentic cardiology scenarios reported to date. It provides a rigorous framework for identifying model strengths, clinically important omissions, and priorities for future development.

Xiao Li, Mouxiao Bian, Zhaodi Wu +8

Department of Cardiology, Zhongshan Hospital Fudan University , Shanhai, China · Shanghai Institute of Cardiovascular Diseases, National Clinical , Shanghai, China · Research Center for Interventional Medicine , Shanghai, China +2

HealthcareClinical Language Model EvaluationElectrocardiogram InterpretationMedical LLMs
AskPDF
May 27, 2026

ClinicalEncoder26AM: A Multlilingual Diagnosable ColBERT Model; Evidences from the MultiClinNER Shared Task

ClinicalEncoder26AM is a multilingual Diagnosable ColBERT for clinical and biomedical texts, which aligns at multiple levels its token-level semantic with ClinicalMap25, a clinical latent space inspired by BioLORD-2023 and enriched with synthetic and annotated supervision. The post-training recipe builds upon BGE-M3, and combines synthetic clinical notes, patient--doctor conversations, and annotated resources such as MedMentions, while considering both named-entity-level and sentence-level representations in a multi-adapter distillation, along with a ColBERT-style retrieval objective. In this system demonstration paper, we evaluate the model in the MultiClinNER shared task by finetuning it as a BIO tagger for patient symptoms, disorders, and procedure spans, using a lightweight two-layer CNN head to improve local boundary detection. The resulting system remains simple, processes most documents in a single 8192-token window, and achieves state-of-the-art multilingual entity recall, while achieving Top 5 overall across all entity types and languages in Character-weighted F1 scores. Training curves further show that ClinicalEncoder26AM is markedly more data-efficient than the base M3 model, supporting the usefulness of its clinical post-training for downstream information extraction. The model can be downloaded on https://huggingface.co/Parallia/ClinicalEncoder26AM-Diagnosable-Colbert-L2-for-multilingual-medical-texts

François Remy

Parallia AI

Named Entity RecognitionHealthcareClinical Information ExtractionMulti-Vector RetrievalText Embedding ModelsClinical NLP
AskPDF