MedInjection-FR: Exploring the Role of Native, Synthetic, and Translated Data in Biomedical Instruction Tuning

Ikram Belmadani; Oumaima El Khettari; Pacôme Constant dit Beaufils; Benoit Favre; Richard Dufour

cs.CLMar 6, 2026

MedInjection-FR: Exploring the Role of Native, Synthetic, and Translated Data in Biomedical Instruction Tuning

Authors: Ikram Belmadani, Oumaima El Khettari, Pacôme Constant dit Beaufils, Benoit Favre, Richard Dufour

Organizations: Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N, UMR 6004, 44000 Nantes, France · Nantes Univ., CHU Nantes, PHU 11: Santé Publique, Clinique des données, INSERM, CIC 1413, 44000 Nantes, France · Nantes Univ., CNRS, INSERM, L’institut du thorax, 44000 Nantes, France ·

Abstract

Instruction tuning has become essential for adapting large language models (LLMs) to follow domain-specific prompts. Yet, in specialized fields such as medicine, the scarcity of high-quality French instruction data limits effective supervision. To address this gap, we introduce MedInjection-FR, a large-scale French biomedical instruction dataset comprising 571K instruction-response pairs drawn from three complementary sources: native, synthetic, and translated data. We design a controlled experimental framework to systematically assess how data provenance affects instruction tuning, using Qwen-4B-Instruct fine-tuned across seven configurations combining these sources. Results show that native data yield the strongest performance, while mixed setups, particularly native and translated, provide complementary benefits. Synthetic data alone remains less effective but contributes positively when balanced with native supervision. Evaluation on open-ended QA combines automatic metrics, LLM-as-a-judge assessment, and human expert review; although LLM-based judgments correlate best with human ratings, they show sensitivity to verbosity. These findings highlight that data authenticity and diversity jointly shape downstream adaptation and that heterogeneous supervision can mitigate the scarcity of native French medical instructions.

Figures & tables

Figure 1: Sample instances from the three MedInjection-FR components. Each includes the task type (MCQ, MCQU, or OEQ) and, when applicable, its supporting context. The figure illustrates the diversity of medical reasoning tasks and prompt styles across data sources.

Figure 2: Distribution of medical specialties in MedInjection-FR (native + synthetic components). Proportions are normalized across fourteen major medical categories derived from specialty metadata.

Component	Train	Validation	Test	Total
Native	$57\,563$	$5055$	$14\,629$	$77\,247$
Generated	$76\,506$	–	–	$76\,506$
Translated	$366\,370$	$38\,011$	$13\,293$	$417\,674$
Total	$500\,439$	$43\,066$	$27\,931$	$571\,436$

Table 1: Number of instruction–response pairs in each component (native, synthetic, and translated) and their respective splits into training, validation, and test sets.

QA Type	OEQ	MCQ	MCQU
Count	$57\,509$	$59\,592$	$454\,335$

Table 2: Counts of OEQ, MCQ and MCQU question types within MedInjection-FR.

Model	BLEU	COMET
GPT-4o-mini	$51.01$	$0.8751$
Gemini 2.0 Flash	$53.72$	$0.8783$
WMT 2024 best system	$53.54$	$0.8760$

Table 3: BLEU and COMET scores obtained by Gemini 2.0 Flash-001 and GPT-4o-mini on biomedical translation benchmarks, compared to the best WMT 2024 system.

Figure 3: Quality evaluation of synthetic data by four LLM judges. The left panel reports OEQ scores on a 1-5 scale, and the right panel reports MCQ scores on a 1-3 scale.

Figure 4: t-SNE visualization of instruction embeddings across the three data sources (native, translated, synthetic).

		pearson_r	p_value
LLMs	MedGemma-27B	0.61	$1.70\text{\times}{10}^{-11}$
	GEMINI-FLASH-2.0	$0.57$	$7.33\text{\times}{10}^{-10}$
	GPT-4.1-mini	$0.49$	$1.80\text{\times}{10}^{-7}$
	HuatuoGPT-o1-72B	$0.47$	$7.78\text{\times}{10}^{-7}$
	GPT-5	$0.38$	$9.56\text{\times}{10}^{-5}$
Metrics	ROUGE-2	$0.36$	$2.22\text{\times}{10}^{-4}$

Table 4: Pearson correlation coefficients (

r

) and associated significance levels (

p

) between human expert judgments and automated evaluation metrics on 100 sampled OEQ responses.

Model	Greedy		Constrained		Greedy	Constrained	Greedy	Constrained
	MCQ				MCQU		Aggregation
Model	EM	Hamming	EM	Hamming	EM		EM
QWEN-4B (base)	$15.92$	$43.78$	$18.44$	$53.42$	$53.14$	$56.17$ *	$34.53$	$37.31$
QWEN-4B-NAT	27.5	59.73	27.5	59.82	$53.49$	$53.67$	$40.5$	$40.59$
QWEN-4B-TRAD	$18.56$	$53.08$	$18.0$	$52.8$	$54.65$	$54.88$	$36.6$	$36.44$
QWEN-4B-SYN	$13.3$	$49.79$	$12.73$	$49.94$	$46.46$	$46.73$	$29.88$	$29.73$

Table 5: Performance of all fine-tuning configurations on MCQ and MCQU under greedy and constrained decoding. Results are reported as Exact Match (EM) and Hamming scores. The aggregated columns report the average of EM scores across MCQ and MCQU formats. Since both settings correspond to MCQA and differ only in the number of correct options (single-answer vs. multiple-answer), we treat them as complementary variants of the same task. The aggregation therefore provides a compact summary of overall MCQA performance. Bold values indicate the best-performing models, while asterisks (*) denote the second-best results within each column.

Figure 5: Relationship between mean output length and LLM-judge accuracy across fine-tuned models. Each point represents one configuration, with the dashed line indicating the best-fit linear trend.

Model	OEQ
	BLEU	ROUGE-2	METEOR	BERTScore_F1	LLM-as-a-Judge
QWEN-4B (base)	$12.08$	$20.82$	30.78	$27.22$	0.36
QWEN-4B-NAT	$31.85$ *	$22.67$	$25.58$	$33.44$	$0.24$
QWEN-4B-TRAD	$29.48$	$21.42$	$26.49$	$27.39$	$0.22$
QWEN-4B-SYN	$26.73$	$20.81$	$24.79$	$27.54$	$0.31$ *
QWEN-4B-NAT-TRAD	32.09	$23.05$ *	$25.90$	$35.60$ *	$0.23$

Table 6: Results on OEQ evaluated with automatic text similarity metrics and the LLM-as-a-judge (MedGemma-27B) framework. Bold numbers mark the best results, and asterisks (*) indicate the second-best within each column.

Figure 6: Comparison between average output length and reference length for each model. The vertical dashed line marks the mean reference length (

\approx 90\ tokens

), and the horizontal line represents a perfect length ratio of 1.

Panel of Figure 7 — Figure 7: System prompt used to generate OEQA synthetic instruction–response pairs from the MORFITT abstract corpus.

Panel of Figure 8 — Figure 8: System prompt used to generate OEQA synthetic instruction–response pairs from DEFT clinical cases.

Panel of Figure 9 — Figure 9: System prompt used to generate OEQA synthetic instruction–response pairs from DiaMed clinical cases.

Panel of Figure 10 — Figure 10: System prompt used to generate MCQA items from the MORFITT abstract corpus.

Panel of Figure 11 — Figure 11: System prompt used to generate MCQA items from DEFT and DiaMed clinical cases.

Panel of Figure 12 — Figure 12: System prompt used to automatically evaluate OEQA synthetic question–answer pairs.

Panel of Figure 13 — Figure 13: System prompt used to automatically evaluate MCQA synthetic question–answer items.

Panel of Figure 14 — Figure 14: System prompt provided to LLM judges to score answer equivalence against the ground truth.

Model	Greedy		Constrained		Greedy	Constrained	Greedy	Constrained
	QCM				QCMU		Aggregation
Model	EM	Hamming	EM	Hamming	EM		EM
QWEN-4B (base)	$10.76$ 4.43	$37.09$ 5.84	$12.72$ 5.10	$45.10$ 7.25	$30.98$ 19.18	$33.68$ * 19.64	$20.87$ 13.92	$23.20$ 14.35
QWEN-4B-NAT	18.27 7.95	49.20 9.27	18.27 7.95	49.23 9.32	$32.80$ 18.03	$32.86$ 18.14	25.53 13.93	25.56 14.00
QWEN-4B-TRAD	$12.48$ 5.63	$45.41$ 6.83	$11.91$ 5.56	$44.96$ 6.86	$33.45$ 18.54	$33.54$ 18.70	$22.97$ 13.70	$22.73$ 13.79
QWEN-4B-SYN	$9.90$ 2.88	$44.47$ 4.55	$9.29$ 2.91	$44.62$ 4.63	$31.18$ 13.28	$31.27$ 13.43	$20.54$ 9.61	$20.28$ 9.72

Table 7: Performance of all configurations on QCM and QCMU under three answer-order permutations to assess positional bias. Results are macro-averaged across datasets and reported as Exact Match (EM) and Hamming scores. Standard deviations (superscript) correspond to the sample standard deviation across the three answer-order runs, reflecting sensitivity to option reordering. Aggregated columns report the average EM across QCM and QCMU. Bold values indicate the best-performing models, and asterisks (*) indicate the second-best within each column.

Instruction Tuning

Instruction tuning adapts pretrained language models to follow natural-language requests using curated or synthesized instruction-response examples.

See all 82 papers

Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
Mar 19, 2026Myeongkyun Kang, Soopil Kim, Xiaoxiao Li +1Medical VLMs Large Vision-Language Models
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea · Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea +2
Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA
Jun 17, 2026Ikram Belmadani, Oumaima El Khettari, Carlos Ramisch +3LLM Fine-Tuning Domain-Adaptive Pretraining
Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N UMR 6004, 44000 Nantes, France · Grenoble Alpes Univ., CNRS, INRIA, Grenoble INP, LIG UMR 5217, 38000 Grenoble, France
LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data
Oct 28, 2025Julian Valline, Cedric Lothritz, Siwen Guo +1LLM Fine-Tuning Synthetic Data Generation
Luxembourg Institute of Science and Technology 5 Av. des Hauts-Fourneaux L-4362 Esch-sur-Alzette

Mar 19, 2026J/K move · Enter open · S save

Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following

Myeongkyun Kang, Soopil Kim, Xiaoxiao Li, Sang Hyun Park

Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea · Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea+2

Medical VLMs Large Vision-Language Models Instruction Following Supervised Fine-Tuning Medical VQA

Open paper AskPDF arXiv

Large vision language models (LVLMs) have demonstrated impressive performance across a wide range of tasks. These capabilities largely stem from visual instruction tuning, which fine-tunes models on datasets consisting of curated image-instruction-output triplets. However, in the medical domain, constructing large-scale, high-quality instruction datasets is particularly challenging due to the need for specialized expert knowledge. To address this issue, we propose an instruction-free tuning approach that reduces reliance on handcrafted instructions, leveraging only image-description pairs for fine-tuning. Specifically, we introduce a momentum proxy instruction as a replacement for curated text instructions, which preserves the instruction-following capability of the pre-trained LVLM while promoting updates to parameters that remain valid during inference. Consequently, the fine-tuned LVLM can flexibly respond to domain-specific instructions, even though explicit instructions are absent during fine-tuning. Additionally, we incorporate a response shuffling strategy to mitigate the model's over-reliance on previous words, facilitating more effective fine-tuning. Our approach achieves state-of-the-art accuracy on multiple-choice visual question answering tasks across SKINCON, WBCAtt, CBIS, and MIMIC-CXR datasets, significantly enhancing the fine-tuning efficiency of LVLMs in medical domains.

Abstract

Instruction Tuning

Explore similar work

Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following

Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data