Advances in large language models (LLMs) have created opportunities for cross-lingual transfer of medical knowledge from high-resource to low-resource languages to support access to healthcare information. However, medical question-answering (QA) datasets for fine-tuning and evaluating LLMs remain scarce for Tetun. To address this gap, we translated MedAlpaca's Medical Flashcards and WikiDoc Patient Information collections into Tetun using Gemini 2.5 Flash, producing 39,077 medical QA pairs. Translation quality was assessed on 50 randomly sampled QA pairs (100 documents) using automatic metrics against human reference translations and evaluations by two native Tetun-speaking healthcare professionals, focusing on medical accuracy and Tetun fluency. Human translations achieved higher mean ratings than LLM-generated translations for both criteria (4.43/5 vs. 4.02/5), with statistically significant differences (Wilcoxon signed-rank test, p < 0.001). To assess downstream utility, we fine-tuned Gemma and Llama variants on the translated MedAlpaca dataset using parameter-efficient fine-tuning (PEFT) and evaluated them on 325 Tetun-translated United States Medical Licensing Examination multiple-choice questions. Gemma 4 12B fine-tuned using QLoRA achieved relative improvements in mean accuracy of 10.79% with 4-bit NormalFloat (NF4) inference and 10.42% after merging the LoRA adapters into the BF16 base model, compared with the corresponding base models. These results highlight the viability of automatic dataset translation for Tetun medical QA and the effectiveness of PEFT in enhancing model performance. The datasets, models, and evaluation resources are accessible at https://github.com/labadain/labadain-medqa to support reproducible research on Tetun medical QA and facilitate machine translation, cross lingual, and multilingual research.
Figures & tables
Figure 1: Overview of the Labadain-MedQA workflow for constructing Tetun medical datasets for LLM fine-tuning and evaluation.
Figure 2: Iterative translation and orthography revision pipeline for Labadain-MedQA.
Figure 3: Examples of the Labadain-MedQA dataset structure, showing English source texts and their corresponding Tetun translations.
Figure 4: Word-count distributions of English source texts and their Tetun translations. Question lengths are displayed on a linear scale, while answer lengths are displayed on a logarithmic scale.
Figure 5: Human evaluation of medical accuracy and Tetun fluency, showing mean ratings with 95% confidence intervals and per-document comparisons ( n=100 ).
Criterion
Human
LLM
Kendall’s τb
Wilcoxon p
Paired t -test p
Medical accuracy
4.53±0.63
4.18±0.74
0.231
1.0×10−4
1.4×10−4
Tetun fluency
4.33±0.63
3.86±0.84
0.254
7.4×10−5
3.9×10−5
Table 1: Statistical comparison of human and LLM-generated translations for medical accuracy and Tetun fluency. Values are reported as mean ± STD of per-document annotator means ( n=100 ).
Method
BLEU-4
chrF++
ROUGE-L
Without normalization
23.45
55.17
58.78
With normalization
26.68
57.53
61.30
Improvement
+3.23
+2.36
+2.52
Table 2: Automatic evaluation of English-to-Tetun medical QA translations before and after orthographic normalization. Improvements indicate absolute score differences.
Model
Step 1
Step 2
Step 3
Average
Gemma 4 E2B + QLoRA (NF4)
0.2766
0.2569
0.2541
0.2625
Gemma 4 E4B + QLoRA (NF4)
0.4362
0.4587
0.4836
0.4595
LLaMA 3.1 8B + QLoRA (NF4)
0.2447
0.2110
0.1967
0.2175
LLaMA 3.2 1B + QLoRA (NF4)
0.1596
0.2294
0.1393
0.1761
LLaMA 3.2 3B + QLoRA (NF4)
0.1596
0.2477
0.2049
0.2041
Gemma 4 12B Base (NF4)
0.5106
0.4404
0.5082
0.4864
Table 3: Accuracy of QLoRA-fine-tuned Gemma and LLaMA model variants on Tetun-translated USMLE multiple-choice questions across Steps 1–3. Underlined values indicate the best performance under NF4 inference, while bold values indicate the best overall performance.
Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces based on medical knowledge extracted from Wikipedia. We produce 500k traces in English, Italian, and Spanish, using a retrieval-augmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and out-of-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.
Pietro Ferrazzi, Aitor Soroa, Rodrigo Agerri
Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain
We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats: MCQA, NLI and open-ended QA. The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions. Each translation was evaluated and revised by two experts fluent in English and the corresponding target language. On HealMed, performance declined most in low-resource languages, although the size of the gap varied markedly across languages and models. The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps. Medical specialization alone did not ensure multilingual robustness. Furthermore, expert revision could either raise or lower measured performance, indicating that translation quality materially affects cross-language evaluation results.
Open medical language models have converged on a single scale: every widely used system runs at 7B parameters or more, leaving the sub-billion regime uncharacterized. We present MedLLM, an open 0.1B-parameter medical language model trained through a fully open three-phase pipeline: general pretraining with curriculum sequence-length scheduling, domain fine-tuning on MedFineWeb, a reference-guided medical corpus we release that is selected from general web data by embedding similarity to medical question-answering (QA) data, and preference-aligned fine-tuning combining SFT with direct preference optimization (DPO). Across medical benchmarks, MedLLM shows a pattern visible only at sub-billion scale: medical competence does not degrade uniformly under compression but splits by task type. On context-grounded QA it comes within 2.9pp of a medically adapted 7B model and surpasses the instruction-tuned and general-purpose 7B baselines; on knowledge-recall QA it stays near the task floor on clinical-vignette MedQA yet significantly exceeds every 7B and sub-7B baseline on MedMCQA, indicating that where recall fails the constraint is model capacity rather than adaptation. This dissociation is masked at 7B, where both capabilities are present, and surfaces only when capacity is scarce.
Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh +1
German Research Center for Artificial Intelligence (DFKI), Germany · Saarland University, Germany