We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats: MCQA, NLI and open-ended QA. The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions. Each translation was evaluated and revised by two experts fluent in English and the corresponding target language. On HealMed, performance declined most in low-resource languages, although the size of the gap varied markedly across languages and models. The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps. Medical specialization alone did not ensure multilingual robustness. Furthermore, expert revision could either raise or lower measured performance, indicating that translation quality materially affects cross-language evaluation results.
Figures & tables
Figure 1: Overview of HealMed. a . Benchmark scope, comprising MCQA, NLI and open-ended QA across nine languages and nine source datasets. b . Construction pipeline, including English source-example selection, machine translation into eight target languages, two-stage review and revision by bilingual medical experts, and final consistency checking and curation.
Figure 2: Multilingual performance on the MCQA and NLI tasks in HealMed . a. Macro-average accuracy across four MCQA and two NLI datasets for 14 models. Blue bars show the lower-resource mean, and red extensions show the gap to the higher-resource mean. Circles, squares and diamonds denote proprietary, open-source and medically specialized models, respectively. b. Language-level accuracy across models. c. Within-model accuracy shifts relative to English. Grey points represent individual models; colored points and horizontal lines show the mean and interquartile range. Black, blue and red denote English, other higher-resource languages and lower-resource languages, respectively. Higher-resource languages comprise English, German, Spanish, Portuguese, Japanese and Chinese; lower-resource languages comprise Thai, Swahili and Zulu.
Figure 3: Open-ended QA performance on HealMed. a . Mean LLM-as-judge scores for ten models, macro-averaged across the three QA datasets. Blue and red points show higher- and lower-resource means; diamonds show means across all languages. b . Score shifts relative to each model’s English score. Positive values indicate higher scores than the English baseline. Grey lines show individual models and the dark line shows their mean. c . Proportions of responses with an overall score below 3, a safety score of 2 or lower, or specific language and applicability issues. Other higher-resource languages comprise English, German, Spanish, Portuguese, Japanese and Chinese. Bubble size and color indicate the proportion; values are shown when at least 10%.
Figure 4: Evaluation shifts between expert-reviewed HealMed and machine-translated (MT) data. a . English-adjusted mean accuracy shifts (HealMed minus MT) across 14 models and six MCQA and NLI datasets. Cells are labelled when ∣Δ∣≥3 percentage points (pp). b . English-adjusted mean LLM-as-judge score shifts across five models and three open-ended QA datasets. Symbols denote datasets, and horizontal lines span their mean shifts. In both panels, Overall reports the mean absolute model-level shift across the corresponding models and datasets. Positive values indicate higher performance on expert-reviewed data; English is shown unadjusted as a same-source control.
Language
GPT-4o
GPT-5.4
o4-mini
Gemini-3
English
0.00
0.00
0.00
0.00
German
0.00
0.00
0.00
0.00
Spanish
0.00
0.00
0.00
0.00
Portuguese
0.00
0.25
0.00
0.00
Japanese
0.00
0.25
0.00
0.00
Chinese
0.25
0.00
0.00
0.00
Table 1: Language-specific refusal rates in open-ended QA. Values are percentages of 400 zero-shot responses per language. A refusal was counted only when the model explicitly declined to answer and provided no substantive response. Overall rates were calculated across 3,600 responses per model. Gemini-3 denotes Gemini-3-Flash.
Language
Acc.
Flu.
Comp.
Revision (%)
German
4.87
4.75
4.83
12.1
Spanish
4.92
4.92
4.85
1.0
Portuguese
4.74
4.84
4.93
2.7
Japanese
4.70
4.62
4.96
5.7
Chinese
4.73
4.51
4.98
5.2
Thai
4.71
4.79
4.93
9.1
Table 2: Expert assessment and revision of machine-translated data by language. Accuracy (Acc.), fluency (Flu.) and completeness (Comp.) are mean expert ratings on five-point scales. Revision is the mean normalized word-level edit distance between the original machine translations and reviewer-submitted revisions. Each language contains 1,000 instances.
Criterion
Expert
LLM
Δ
Chinese
Completeness
4.22
3.80
−0.42
Reference alignment
4.24
3.62
−0.62
Clinical consensus
4.33
4.04
−0.29
Clinical appropriateness
4.31
3.93
−0.38
Safety
4.69
4.20
−0.49
Table 3: Criterion-level comparison of expert and LLM-based evaluations. Scores range from 1 to 5. Δ denotes the LLM score minus the expert score and was calculated before rounding.
Language
Traces
Target
English
Japanese
9/15
7 (77.8%)
2 (22.2%)
Chinese
15/15
15 (100%)
0 (0%)
Thai
7/15
0 (0%)
7 (100%)
Table 4: Language use in observable reasoning traces generated by HuatuoGPT-o1-72B. The “Traces” column denotes the number of responses containing an explicit reasoning trace among 15 responses examined per language. ”Target” and ”English” report the number and percentage of traces written predominantly in the target language or English, respectively.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Component
Samples per language, n
Expected model output
MCQA
HeadQA
75
Correct option
MedQA
75
Correct option
MedExpQA
75
Correct option
MMLU-Pro
75
Correct option
NLI
BioNLI
150
Relation label
MedNLI
150
Relation label
Appendix
Table 5: Benchmark components and sample allocation in HealMed. Sample counts denote the number of aligned examples included in each language.
Benchmark scope
Data construction
Human review
Translation audit
Benchmark
Reported scale
Lang.
Tasks
Language design
Review coverage (%)
Reviewers, n
Quality metrics
Score effects
MMedBench
53,566 QA pairs
6
MCQA; rationales
Aggregated; non-aligned
14.1
3
No
No
XMedBench
21,326 records
6
MCQA
Native + MT; non-aligned
10.2
NR
No
No
MedExpQA
2,488 records
4
MCQA
MT + manual revision; aligned
100
NR
No
No
WorldMedQA-V
568 evaluation items
4 + EN
Multimodal MCQA
Local–English pairs
100
4 country 7 EN
No
No
MultiMed-X
2,450 translations
7 + EN
NLI; open QA
MT + expert revision; aligned
100
∼ 12
No
No
Appendix
Table 6: Comparison of selected multilingual medical benchmarks. Reported scales use source-specific units and are not directly comparable. Quality metrics indicates whether aggregate translation-quality scores or revision statistics were reported; score effects indicates whether model performance was compared between machine-translated and expert-reviewed versions. WorldMedQA-V identified four country-level validators and seven contributors to English-translation validation, but did not report the number of unique reviewers because these roles may overlap. EN, English; MT, machine translation; N/A, not applicable; NR, not reported. For BRIDGE, reference standards were inherited from the source datasets, benchmark-wide human-review coverage and reviewer numbers were not reported.
Model
Parameters
Release date
Model type
Proprietary models
GPT-5.4
Not disclosed
5 Mar 2026
Proprietary
o4-mini
Not disclosed
16 Apr 2025
Proprietary
Gemini-3-Flash
Not disclosed
17 Dec 2025
Proprietary
Claude-Sonnet-5
Not disclosed
30 Jun 2026
Proprietary
Claude-Opus-4.8
Not disclosed
28 May 2026
Proprietary
Appendix
Table 7: Models and evaluation configurations used in HealMed.
Score
Accuracy
Fluency
Completeness
5
All concepts and medical terms are translated correctly and precisely. Terminology is professional, contextually appropriate and consistent with established usage in the target language.
The translation is natural and easy to read. Its grammar, wording and sentence structure conform to professional conventions in the target language.
The meaning and all relevant details of the source are retained. There are no omissions or unsupported additions.
4
Most concepts and terms are translated correctly. Minor errors, imprecise wording or simplified terminology may occur but do not affect overall understanding.
The translation is generally natural and clear. Minor stiffness, awkward wording or grammatical errors do not affect comprehension.
The main meaning is retained. Only minor or non-essential details are omitted or expressed unclearly.
3
The main concepts are conveyed, but some errors or imprecise terms may cause partial misunderstanding. The reader may need to infer the intended meaning of some terms.
The translation is understandable but noticeably unnatural in places. Rigid sentence structures, unsuitable word choices or grammatical errors require some effort from the reader.
Most of the source meaning is conveyed, but some information is missing, added or unclear. Important details may require inference.
2
Several important concepts or terms are mistranslated, substantially affecting comprehension. Terminology may be incorrect or inconsistent.
The translation is difficult to read smoothly. It contains awkward transitions, unclear connections or frequent grammatical and structural errors.
Core information is not fully preserved. Noticeable omissions or unnecessary additions reduce correspondence with the source and affect comprehension.
1
Frequent and severe mistranslations prevent the source meaning from being conveyed. Much of the content or terminology does not correspond to the source.
The translation is highly unnatural or difficult to understand. Literal phrasing, disorganized sentence structure and severe grammatical errors may make it unreadable.
Substantial omissions or incorrect additions prevent the translation from reflecting the source. Important passages are missing, and the intended meaning is difficult to recover.
Appendix
Table 8: Five-point rubric used by medical experts to assess machine translations. Each dimension was scored separately.
Setting
MCQA
BioNLI
MedNLI
Open-ended QA
Prompting strategy
zero-shot
zero-shot
zero-shot
zero-shot
Temperature
0.7 where supported; otherwise omitted
Maximum output tokens
32
32
32
2,048
Top- p
Not set; model- or provider-default
Top- k
Not set; model- or provider-default where supported
Random seed
Not set
Appendix
Table 9: Inference settings used for model evaluation.
Dataset
Model
EN
DE
ES
PT
JA
ZH
TH
SW
ZU
Avg.
HeadQA
Proprietary Models
Gemini-3-Flash
94.67
94.67
96.00
96.00
92.00
92.00
90.67
92.00
88.00
92.89
GPT-5.4
90.67
93.33
92.00
93.33
89.33
89.33
86.67
90.67
84.00
89.93
o4-mini
96.00
94.67
97.33
96.00
92.00
90.67
94.67
85.33
85.33
92.44
Claude-Opus-4.8
96.00
96.00
96.00
94.67
90.67
90.67
89.33
93.33
82.67
92.15
Claude-Sonnet-5
90.67
96.00
94.67
94.67
92.00
90.67
93.33
90.67
73.33
90.67
Appendix
Table 10: Complete zero-shot accuracy (%) on the four MCQA datasets. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined .
Dataset
Model
EN
DE
ES
PT
JA
ZH
TH
SW
ZU
Avg.
BioNLI
Proprietary Models
Gemini-3-Flash
79.33
71.33
74.00
72.67
71.33
71.33
74.67
73.33
70.00
73.11
GPT-5.4
73.33
68.00
70.00
68.67
68.00
68.67
68.00
67.33
66.67
68.74
o4-mini
74.00
71.33
71.33
71.33
71.33
70.67
70.00
66.67
63.33
70.00
Claude-Opus-4.8
78.00
70.67
74.67
72.00
72.00
76.00
74.00
74.67
70.67
73.63
Claude-Sonnet-5
72.67
68.00
71.33
68.00
68.00
69.33
68.67
67.33
62.00
68.37
Appendix
Table 11: Complete zero-shot accuracy (%) on the two NLI datasets. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined .
Dataset
Model
EN
DE
ES
PT
JA
ZH
TH
SW
ZU
Avg.
ExpertQA-Bio
Proprietary Models
GPT-5.4
4.08
4.29
4.22
4.28
4.19
4.53
4.52
4.12
4.03
4.25
Gemini-3-Flash
3.81
3.90
3.91
3.98
3.99
4.19
4.01
3.89
4.02
3.97
open-source Models
DeepSeek-V3
4.38
3.94
3.96
4.08
3.79
4.08
3.99
3.55
3.30
3.89
Gemma-3-27B-it
3.87
3.73
3.90
3.78
3.59
4.02
3.72
3.38
3.00
3.66
Appendix
Table 12: Complete LLM-as-judge scores on the three open-ended QA datasets. Scores range from 1 to 5 and are calculated as the mean of completeness, reference alignment, clinical consensus, clinical appropriateness and safety. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined .
This paper investigates Multilingual Medical Question Answering across high-resource (English, Spanish, French, Italian) and low-resource (Basque, Kazakh) languages. We evaluate three types of external evidence sources across models of varying size: curated repositories of specialized medical knowledge, web-retrieved content, and explanations from LLM's parametric knowledge. Moreover, we conduct experiments with multilingual, monolingual and cross-lingual retrieval. Our results demonstrate that larger models consistently achieve superior performance in English across baseline evaluations. When incorporating external knowledge, web-retrieved data in English proves most beneficial for high-resource languages. Conversely, for low-resource languages, the most effective strategy combines retrieval in both English and the target language, achieving comparable accuracy to high-resource language results. These findings challenge the assumption that external knowledge systematically improves performance and reveal that effective strategies depend on both the source of language resources and on model scale. Furthermore, specialized medical knowledge sources such as PubMed are limited: while they provide authoritative expert knowledge, they lack adequate multilingual coverage
Anar Yeginbergen, Maite Oronoz, Rodrigo Agerri
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces based on medical knowledge extracted from Wikipedia. We produce 500k traces in English, Italian, and Spanish, using a retrieval-augmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and out-of-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.
Pietro Ferrazzi, Aitor Soroa, Rodrigo Agerri
Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain
Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases. To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test reasoning. We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence. To facilitate further research, we make our benchmark publicly available.
Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4