EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

Francesco Andrea Causio; Vittorio De Vita; Olivia Riccomi; Michele Ferramola; Federico Felizzi; Alessandro Tosi; Antonio Cristiano; Lorenzo De Mori

cs.CLApr 15, 2026

EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

Authors: Francesco Andrea Causio, Vittorio De Vita, Olivia Riccomi, Michele Ferramola, Federico Felizzi, Alessandro Tosi, Antonio Cristiano, Lorenzo De Mori, +12 more

Organizations: Section of Hygiene, University Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy · Società Italiana Intelligenza Artificiale in Medicina (SIIAM), Rome, Italy · NSBproject, Mantova, Italy · Department of Mental Health, Addiction Treatment Unit, ASL RM 4, Bracciano, Italy · Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy · Department of Psychiatry, University of Campania “Luigi Vanvitelli”, Naples, Italy · Université Paris Cité, Paris, France · Università Cattolica del Sacro Cuore, Rome, Italy · Department of Experimental Medicine, Section of Medical Pathophysiology, Food Science and Endocrinology, Sapienza University, Rome, Italy · Department of Diagnostic Imaging, Oncological Radiotherapy and Hematology, Università Cattolica del Sacro Cuore, Rome, Italy · Department of Medicine and Surgery, University of Perugia

Abstract

While Large Language Models (LLMs) have demonstrated high proficiency on English-centric medical examinations, their performance often declines when faced with non-English languages and multimodal diagnostic tasks. This study protocol describes the development of EuropeMedQA, the first comprehensive, multilingual, and multimodal medical examination dataset sourced from official regulatory exams in Italy, France, Spain, and Portugal. Following FAIR data principles and SPIRIT-AI guidelines, we describe a rigorous curation process and an automated translation pipeline for comparative analysis. We evaluate contemporary multimodal LLMs using a zero-shot, strictly constrained prompting strategy to assess cross-lingual transfer and visual reasoning. EuropeMedQA aims to provide a contamination-resistant benchmark that reflects the complexity of European clinical practices and fosters the development of more generalizable medical AI.

Figures & tables

Panel of Figure 1 — Figure 1: Study flow diagram. After dataset processing, three parallel tracks are produced: an English-translated version, the original-language dataset, and any additional subsets. LLM evaluation is performed on both the English and original-language tracks, and results are compared.

Variable	Description
questionID	A unique identifier for each question number in the dataset: country ID (IT/ES/FR/PT), as a progressive number.
testyear	The year in which the test was administered to the public.
numberintest	The position or number of the question within the test.
picture_provided	A flag (yes/no) indicating whether a visual element (e.g., image) is based on the presence of an image (e.g., EKG).
question_text	The text of the question itself including a scenario description, when appropriate.
picture_link	A URL or reference to the shared Drive folder where the question picture is archived, if available.

Table 2

Explore similar work

Multilingual Medical Reasoning for Question Answering with Large Language Models
Dec 5, 2025Pietro Ferrazzi, Aitor Soroa, Rodrigo AgerriKnowledge Augmentation for Language Models Healthcare
Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain
HealMed: Multilingual Evaluation of Large Language Models in Medicine
Aug 20, 2026Yingjian Chen, Fan Gao, Sherry T. Tong +42Multilingual Language Model Evaluation Medical LLMs
Drew
Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA
Aug 23, 2026Jingbo Wang, Sendong Zhao, Haochun Wang +2Medical VQA Medical VLMs
Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China

Dec 5, 2025J/K move · Enter open · S save

Multilingual Medical Reasoning for Question Answering with Large Language Models

Pietro Ferrazzi, Aitor Soroa, Rodrigo Agerri

Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain

Knowledge Augmentation for Language Models Healthcare Medical QA Multilingual QA Medical Reasoning Medical LLMs

Open paper AskPDF arXiv

Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces based on medical knowledge extracted from Wikipedia. We produce 500k traces in English, Italian, and Spanish, using a retrieval-augmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and out-of-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.

Abstract

Explore similar work

Multilingual Medical Reasoning for Question Answering with Large Language Models

HealMed: Multilingual Evaluation of Large Language Models in Medicine

Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA