HealMed: Multilingual Evaluation of Large Language Models in Medicine

Yingjian Chen; Fan Gao; Sherry T. Tong; Haoyu Zhang; Aosong Feng; Kevin W. Jin; Xing Wu; Jinghui Lu

cs.CLAug 20, 2026

HealMed: Multilingual Evaluation of Large Language Models in Medicine

Authors: Yingjian Chen, Fan Gao, Sherry T. Tong, Haoyu Zhang, Aosong Feng, Kevin W. Jin, Xing Wu, Jinghui Lu, +37 more

Organizations: Drew

Abstract

We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats: MCQA, NLI and open-ended QA. The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions. Each translation was evaluated and revised by two experts fluent in English and the corresponding target language. On HealMed, performance declined most in low-resource languages, although the size of the gap varied markedly across languages and models. The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps. Medical specialization alone did not ensure multilingual robustness. Furthermore, expert revision could either raise or lower measured performance, indicating that translation quality materially affects cross-language evaluation results.

Figures & tables

Figure 1: Overview of HealMed. a . Benchmark scope, comprising MCQA, NLI and open-ended QA across nine languages and nine source datasets. b . Construction pipeline, including English source-example selection, machine translation into eight target languages, two-stage review and revision by bilingual medical experts, and final consistency checking and curation.

Figure 2: Multilingual performance on the MCQA and NLI tasks in HealMed . a. Macro-average accuracy across four MCQA and two NLI datasets for 14 models. Blue bars show the lower-resource mean, and red extensions show the gap to the higher-resource mean. Circles, squares and diamonds denote proprietary, open-source and medically specialized models, respectively. b. Language-level accuracy across models. c. Within-model accuracy shifts relative to English. Grey points represent individual models; colored points and horizontal lines show the mean and interquartile range. Black, blue and red denote English, other higher-resource languages and lower-resource languages, respectively. Higher-resource languages comprise English, German, Spanish, Portuguese, Japanese and Chinese; lower-resource languages comprise Thai, Swahili and Zulu.

Figure 3: Open-ended QA performance on HealMed. a . Mean LLM-as-judge scores for ten models, macro-averaged across the three QA datasets. Blue and red points show higher- and lower-resource means; diamonds show means across all languages. b . Score shifts relative to each model’s English score. Positive values indicate higher scores than the English baseline. Grey lines show individual models and the dark line shows their mean. c . Proportions of responses with an overall score below 3, a safety score of 2 or lower, or specific language and applicability issues. Other higher-resource languages comprise English, German, Spanish, Portuguese, Japanese and Chinese. Bubble size and color indicate the proportion; values are shown when at least 10%.

Figure 4: Evaluation shifts between expert-reviewed HealMed and machine-translated (MT) data. a . English-adjusted mean accuracy shifts (HealMed minus MT) across 14 models and six MCQA and NLI datasets. Cells are labelled when $|\Delta|\geq 3$ percentage points (pp). b . English-adjusted mean LLM-as-judge score shifts across five models and three open-ended QA datasets. Symbols denote datasets, and horizontal lines span their mean shifts. In both panels, Overall reports the mean absolute model-level shift across the corresponding models and datasets. Positive values indicate higher performance on expert-reviewed data; English is shown unadjusted as a same-source control.

Language	GPT-4o	GPT-5.4
English	0.00	0.00
German	0.00	0.00
Spanish	0.00	0.00
Portuguese	0.00	0.25
Japanese	0.00	0.25
Chinese	0.25	0.00

Table 1: Language-specific refusal rates in open-ended QA. Values are percentages of 400 zero-shot responses per language. A refusal was counted only when the model explicitly declined to answer and provided no substantive response. Overall rates were calculated across 3,600 responses per model. Gemini-3 denotes Gemini-3-Flash.

Language	Acc.	Flu.	Comp.	Revision (%)
German	4.87	4.75	4.83	12.1
Spanish	4.92	4.92	4.85	1.0
Portuguese	4.74	4.84	4.93	2.7
Japanese	4.70	4.62	4.96	5.7
Chinese	4.73	4.51	4.98	5.2
Thai	4.71	4.79	4.93	9.1

Table 2: Expert assessment and revision of machine-translated data by language. Accuracy (Acc.), fluency (Flu.) and completeness (Comp.) are mean expert ratings on five-point scales. Revision is the mean normalized word-level edit distance between the original machine translations and reviewer-submitted revisions. Each language contains 1,000 instances.

Criterion	Expert	LLM	$\Delta$
Chinese
Completeness	4.22	3.80	$-0.42$
Reference alignment	4.24	3.62	$-0.62$
Clinical consensus	4.33	4.04	$-0.29$
Clinical appropriateness	4.31	3.93	$-0.38$
Safety	4.69	4.20	$-0.49$

Table 3: Criterion-level comparison of expert and LLM-based evaluations. Scores range from 1 to 5.

\Delta

denotes the LLM score minus the expert score and was calculated before rounding.

Language	Traces	Target	English
Japanese	9/15	7 (77.8%)	2 (22.2%)
Chinese	15/15	15 (100%)	0 (0%)
Thai	7/15	0 (0%)	7 (100%)

Table 4: Language use in observable reasoning traces generated by HuatuoGPT-o1-72B. The “Traces” column denotes the number of responses containing an explicit reasoning trace among 15 responses examined per language. ”Target” and ”English” report the number and percentage of traces written predominantly in the target language or English, respectively.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Component	Samples per language, $n$	Expected model output
MCQA	HeadQA	75	Correct option
	MedQA	75	Correct option
	MedExpQA	75	Correct option
	MMLU-Pro	75	Correct option
NLI	BioNLI	150	Relation label
	MedNLI	150	Relation label

Appendix

Table 5: Benchmark components and sample allocation in HealMed. Sample counts denote the number of aligned examples included in each language.

	Benchmark scope			Data construction	Human review		Translation audit
Benchmark	Reported scale	Lang.	Tasks	Language design	Review coverage (%)	Reviewers, $n$	Quality metrics	Score effects
MMedBench	53,566 QA pairs	6	MCQA; rationales	Aggregated; non-aligned	14.1	3	No	No
XMedBench	21,326 records	6	MCQA	Native + MT; non-aligned	10.2	NR	No	No
MedExpQA	2,488 records	4	MCQA	MT + manual revision; aligned	100	NR	No	No
WorldMedQA-V	568 evaluation items	4 + EN	Multimodal MCQA	Local–English pairs	100	4 country 7 EN	No	No
MultiMed-X	2,450 translations	7 + EN	NLI; open QA	MT + expert revision; aligned	100	$\sim$ 12	No	No

Appendix

Table 6: Comparison of selected multilingual medical benchmarks. Reported scales use source-specific units and are not directly comparable. Quality metrics indicates whether aggregate translation-quality scores or revision statistics were reported; score effects indicates whether model performance was compared between machine-translated and expert-reviewed versions. WorldMedQA-V identified four country-level validators and seven contributors to English-translation validation, but did not report the number of unique reviewers because these roles may overlap. EN, English; MT, machine translation; N/A, not applicable; NR, not reported. For BRIDGE, reference standards were inherited from the source datasets, benchmark-wide human-review coverage and reviewer numbers were not reported.

Proprietary models
Model	Parameters	Release date	Model type
GPT-5.4	Not disclosed	5 Mar 2026	Proprietary
o4-mini	Not disclosed	16 Apr 2025	Proprietary
Gemini-3-Flash	Not disclosed	17 Dec 2025	Proprietary
Claude-Sonnet-5	Not disclosed	30 Jun 2026	Proprietary
Claude-Opus-4.8	Not disclosed	28 May 2026	Proprietary

Appendix

Table 7: Models and evaluation configurations used in HealMed.

Score	Accuracy	Fluency	Completeness
5	All concepts and medical terms are translated correctly and precisely. Terminology is professional, contextually appropriate and consistent with established usage in the target language.	The translation is natural and easy to read. Its grammar, wording and sentence structure conform to professional conventions in the target language.	The meaning and all relevant details of the source are retained. There are no omissions or unsupported additions.
4	Most concepts and terms are translated correctly. Minor errors, imprecise wording or simplified terminology may occur but do not affect overall understanding.	The translation is generally natural and clear. Minor stiffness, awkward wording or grammatical errors do not affect comprehension.	The main meaning is retained. Only minor or non-essential details are omitted or expressed unclearly.
3	The main concepts are conveyed, but some errors or imprecise terms may cause partial misunderstanding. The reader may need to infer the intended meaning of some terms.	The translation is understandable but noticeably unnatural in places. Rigid sentence structures, unsuitable word choices or grammatical errors require some effort from the reader.	Most of the source meaning is conveyed, but some information is missing, added or unclear. Important details may require inference.
2	Several important concepts or terms are mistranslated, substantially affecting comprehension. Terminology may be incorrect or inconsistent.	The translation is difficult to read smoothly. It contains awkward transitions, unclear connections or frequent grammatical and structural errors.	Core information is not fully preserved. Noticeable omissions or unnecessary additions reduce correspondence with the source and affect comprehension.
1	Frequent and severe mistranslations prevent the source meaning from being conveyed. Much of the content or terminology does not correspond to the source.	The translation is highly unnatural or difficult to understand. Literal phrasing, disorganized sentence structure and severe grammatical errors may make it unreadable.	Substantial omissions or incorrect additions prevent the translation from reflecting the source. Important passages are missing, and the intended meaning is difficult to recover.

Appendix

Table 8: Five-point rubric used by medical experts to assess machine translations. Each dimension was scored separately.

Setting	MCQA	BioNLI	MedNLI	Open-ended QA
Prompting strategy	zero-shot	zero-shot	zero-shot	zero-shot
Temperature	0.7 where supported; otherwise omitted
Maximum output tokens	32	32	32	2,048
Top- $p$	Not set; model- or provider-default
Top- $k$	Not set; model- or provider-default where supported
Random seed	Not set

Appendix

Table 9: Inference settings used for model evaluation.

HeadQA	Proprietary Models
Dataset	Model	EN	DE	ES	PT	JA	ZH	TH	SW	ZU	Avg.
	Gemini-3-Flash	94.67	94.67	96.00	96.00	92.00	92.00	90.67	92.00	88.00	92.89
	GPT-5.4	90.67	93.33	92.00	93.33	89.33	89.33	86.67	90.67	84.00	89.93
	o4-mini	96.00	94.67	97.33	96.00	92.00	90.67	94.67	85.33	85.33	92.44
	Claude-Opus-4.8	96.00	96.00	96.00	94.67	90.67	90.67	89.33	93.33	82.67	92.15
	Claude-Sonnet-5	90.67	96.00	94.67	94.67	92.00	90.67	93.33	90.67	73.33	90.67

Appendix

Table 10: Complete zero-shot accuracy (%) on the four MCQA datasets. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined .

BioNLI	Proprietary Models
Dataset	Model	EN	DE	ES	PT	JA	ZH	TH	SW	ZU	Avg.
	Gemini-3-Flash	79.33	71.33	74.00	72.67	71.33	71.33	74.67	73.33	70.00	73.11
	GPT-5.4	73.33	68.00	70.00	68.67	68.00	68.67	68.00	67.33	66.67	68.74
	o4-mini	74.00	71.33	71.33	71.33	71.33	70.67	70.00	66.67	63.33	70.00
	Claude-Opus-4.8	78.00	70.67	74.67	72.00	72.00	76.00	74.00	74.67	70.67	73.63
	Claude-Sonnet-5	72.67	68.00	71.33	68.00	68.00	69.33	68.67	67.33	62.00	68.37

Appendix

Table 11: Complete zero-shot accuracy (%) on the two NLI datasets. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined .

ExpertQA-Bio	Proprietary Models
Dataset	Model	EN	DE	ES	PT	JA	ZH	TH	SW	ZU	Avg.
	GPT-5.4	4.08	4.29	4.22	4.28	4.19	4.53	4.52	4.12	4.03	4.25
	Gemini-3-Flash	3.81	3.90	3.91	3.98	3.99	4.19	4.01	3.89	4.02	3.97
	open-source Models
	DeepSeek-V3	4.38	3.94	3.96	4.08	3.79	4.08	3.99	3.55	3.30	3.89
	Gemma-3-27B-it	3.87	3.73	3.90	3.78	3.59	4.02	3.72	3.38	3.00	3.66

Appendix

Table 12: Complete LLM-as-judge scores on the three open-ended QA datasets. Scores range from 1 to 5 and are calculated as the mean of completeness, reference alignment, clinical consensus, clinical appropriateness and safety. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined .

Explore similar work

Effects of Cross-lingual Evidence in Multilingual Medical Question Answering
Apr 22, 2026Anar Yeginbergen, Maite Oronoz, Rodrigo AgerriKnowledge Augmentation for Language Models Healthcare
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Multilingual Medical Reasoning for Question Answering with Large Language Models
Dec 5, 2025Pietro Ferrazzi, Aitor Soroa, Rodrigo AgerriKnowledge Augmentation for Language Models Healthcare
Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
Jun 10, 2026Antoni Lasik, Jakub Pokrywka, Łukasz Grzybowski +7LLM Evaluation LLM Reliability
NASK National Research Institute · Adam Mickiewicz University · ARAAI Poland +4

Apr 22, 2026J/K move · Enter open · S save

Effects of Cross-lingual Evidence in Multilingual Medical Question Answering

Anar Yeginbergen, Maite Oronoz, Rodrigo Agerri

HiTZ Center - Ixa, University of the Basque Country UPV/EHU

Knowledge Augmentation for Language Models Healthcare Medical QA Multilingual QA Low-Resource Language Processing

Open paper AskPDF arXiv

This paper investigates Multilingual Medical Question Answering across high-resource (English, Spanish, French, Italian) and low-resource (Basque, Kazakh) languages. We evaluate three types of external evidence sources across models of varying size: curated repositories of specialized medical knowledge, web-retrieved content, and explanations from LLM's parametric knowledge. Moreover, we conduct experiments with multilingual, monolingual and cross-lingual retrieval. Our results demonstrate that larger models consistently achieve superior performance in English across baseline evaluations. When incorporating external knowledge, web-retrieved data in English proves most beneficial for high-resource languages. Conversely, for low-resource languages, the most effective strategy combines retrieval in both English and the target language, achieving comparable accuracy to high-resource language results. These findings challenge the assumption that external knowledge systematically improves performance and reveal that effective strategies depend on both the source of language resources and on model scale. Furthermore, specialized medical knowledge sources such as PubMed are limited: while they provide authoritative expert knowledge, they lack adequate multilingual coverage

Abstract

Explore similar work

Effects of Cross-lingual Evidence in Multilingual Medical Question Answering

Multilingual Medical Reasoning for Question Answering with Large Language Models

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?