Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA

Jingbo Wang; Sendong Zhao; Haochun Wang; Bing Qin; Ting Liu

cs.AIAug 23, 2026

Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA

Authors: Jingbo Wang, Sendong Zhao, Haochun Wang, Bing Qin, Ting Liu

Organizations: Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China

Abstract

Medical visual question answering (VQA) is a crucial task in clinical AI, yet its evaluation has so far centered almost exclusively on English, limiting its relevance to linguistically diverse patients and clinicians. Recent multilingual medical VQA benchmarks show that large vision-language models (LVLMs) degrade in non-English languages, but lack a fine-grained analysis of how cross-lingual variation affects the distinct capabilities that medical VQA requires. To this end, we construct a multilingual medical VQA benchmark over eight languages, organized into four representative scenarios that isolate the core capabilities medical VQA requires. Evaluating five open- and closed-source LVLMs, we find that cross-lingual degradation is not uniform but highly scenario-dependent. We therefore propose MedVL-XLRepE, a training-free scenario-aware representation engineering method, leveraging LVLMs' superior English medical VQA capability to steer non-English representations toward their English counterparts at inference time. Across three LVLMs and eight languages, MedVL-XLRepE consistently mitigates cross-lingual degradation, with gains of up to 6.33%.

Figures & tables

Figure 1: Medical VQA scenarios studied in this work.

Figure 2: Construction pipeline of the multilingual medical VQA benchmark: data sourcing and scenario-wise reorganization, multilingual translation and terminology alignment, and benchmark validation and finalization.

Model	Scenario	EN	ZH	ES	FR	JA	TH	AR	BG
Open-source LVLMs
Gemma3-12B-IT	PR	66.67	67.49	64.21	64.48	62.57	64.21	65.85	59.84
	AAR	52.54	50.85	44.07	39.83	46.61	47.46	46.61	38.98
	SIU	43.66	41.79	40.30	42.54	40.30	40.67	39.55	40.30
	VTI	54.23	46.48	48.24	47.18	44.01	45.42	40.14	46.83
Qwen3.5-9B	PR	75.96	72.95	75.96	74.86	71.31	75.41	73.50	72.68

Table 1: Performance (%) of open- and closed-source LVLMs across four medical VQA scenarios and eight languages in our multilingual VQA benchmark. Bold and underlined values indicate the best and worst performance.

Figure 3: Overview of MedVL-XLRepE. The hidden-state difference between paired English and target-language inputs defines a cross-lingual shift $\delta_{x,tgt}^{(l)}$ . Aggregating $\delta_{x,tgt}^{(l)}$ over the scenario set yields a language vector and a medical vector, whose sum is added to $h_{x,tgt}^{(l)}$ to obtain the intervened representation $\tilde{h}_{x,tgt}^{(l)}$ .

Model	Scenario	XLRepE	EN	ZH	ES	FR	JA	TH	AR	BG
Gemma3-12B-IT	PR	$\times$ $\checkmark$	66.67 —	67.49 68.58 $\uparrow$ 1.09	64.21 65.03 $\uparrow$ 0.82	64.48 64.48 $\uparrow$ 0.00	62.57 63.11 $\uparrow$ 0.54	64.21 66.94 $\uparrow$ 2.73	65.85 68.03 $\uparrow$ 2.18	59.84 63.93 $\uparrow$ 4.09
	AAR	$\times$ $\checkmark$	52.54 —	50.85 52.54 $\uparrow$ 1.69	44.07 44.92 $\uparrow$ 0.85	39.83 44.92 $\uparrow$ 5.09	46.61 46.61 $\uparrow$ 0.00	47.46 50.00 $\uparrow$ 2.54	46.61 47.46 $\uparrow$ 0.85	38.98 44.07 $\uparrow$ 5.09
	SIU	$\times$ $\checkmark$	43.66 —	41.79 43.66 $\uparrow$ 1.87	40.30 43.28 $\uparrow$ 2.98	42.54 44.03 $\uparrow$ 1.49	40.30 42.54 $\uparrow$ 2.24	40.67 44.03 $\uparrow$ 3.36	39.55 42.54 $\uparrow$ 2.99	40.30 43.28 $\uparrow$ 2.98
	VTI	$\times$ $\checkmark$	54.23 —	46.48 47.54 $\uparrow$ 1.06	48.24 51.76 $\uparrow$ 3.52	47.18 49.30 $\uparrow$ 2.12	44.01 47.54 $\uparrow$ 3.53	45.42 46.13 $\uparrow$ 0.71	40.14 41.55 $\uparrow$ 1.41	46.83 47.18 $\uparrow$ 0.35
Qwen3.5-9B	PR	$\times$ $\checkmark$	75.96 —	72.95 74.32 $\uparrow$ 1.37	75.96 77.60 $\uparrow$ 1.64	74.86 76.23 $\uparrow$ 1.37	71.31 75.41 $\uparrow$ 4.10	75.41 77.32 $\uparrow$ 1.91	73.50 74.59 $\uparrow$ 1.09	72.68 77.60 $\uparrow$ 4.92
	AAR	$\times$ $\checkmark$	73.73 —	66.10 69.49 $\uparrow$ 3.39	69.49 71.19 $\uparrow$ 1.70	70.34 76.27 $\uparrow$ 5.93	69.49 75.42 $\uparrow$ 5.93	65.25 71.19 $\uparrow$ 5.94	67.80 69.49 $\uparrow$ 1.69	68.64 72.03 $\uparrow$ 3.39

Table 2: Effectiveness of MedVL-XLRepE across models, scenarios, and target languages. Each cell reports accuracy (%) for the baseline (

\times

) and MedVL-XLRepE (

\checkmark

Figure 4: PCA of multilingual hidden states before and after applying MedVL-XLRepE.

Figure 5: Performance under varying intervention layer

l

(left) and strength

\lambda

(right) on Qwen3.5-9B/PR.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario	Original	Multilingual
Perceptual Recognition	731	5,848
Attribute-Aware Recognition	236	1,888
Sequential Images Understanding	536	4,288
Vision-Text Integrated Reasoning	568	4,544
Total	2,071	16,568

Appendix

Table 3: Scenario-wise benchmark statistics under the 8-language setting.

Scenario	Chinese	Spanish	French	Japanese	Thai	Arabic	Bulgarian
PR	53.76	35.84	55.54	72.64	81.40	69.49	50.48
AAR	65.25	37.29	57.63	83.90	82.20	64.41	58.47

Appendix

Table 4: Percentage (%) of items whose Google Translate draft was modified by GPT-5.4 in the VQA-RAD PR and AAR subsets.

Group	EN	ZH	ES	FR	JA	TH	AR	BG	Avg	Max–Min
Open-source	62.25	58.37	58.42	56.82	55.79	55.65	54.08	56.38	57.22	8.17
Closed-source	75.97	73.35	75.33	72.46	73.09	74.36	74.47	73.19	74.03	3.51

Appendix

Table 5: Aggregated language performance (%) by source group.

Setting	EN	ZH	ES	FR	JA	TH	AR	BG	Avg	Max–Min
Overall	67.73	64.36	65.19	63.08	62.71	63.13	62.23	63.10	63.94	5.50
PR	74.43	71.58	72.35	71.53	68.74	71.80	69.24	70.49	71.27	5.68
AAR	69.15	65.42	65.59	61.02	65.93	63.56	65.59	64.40	65.08	8.14
SIU	59.26	55.00	57.31	55.67	54.33	56.05	54.11	54.70	55.80	5.15
VTI	68.10	65.42	65.49	64.08	61.83	61.13	60.00	62.82	63.61	8.10

Appendix

Table 6: Scenario-wise aggregated language performance (%).

Scenario	Rank 1	Rank 2	Rank 3	Rank 4	Rank 5	Rank 6	Rank 7	Rank 8
Overall	EN (67.73)	ES (65.19)	ZH (64.36)	TH (63.13)	BG (63.10)	FR (63.08)	JA (62.71)	AR (62.23)
PR	EN (74.43)	ES (72.35)	TH (71.80)	ZH (71.58)	FR (71.53)	BG (70.49)	AR (69.24)	JA (68.74)
AAR	EN (69.15)	JA (65.93)	ES (65.59)	AR (65.59)	ZH (65.42)	BG (64.40)	TH (63.56)	FR (61.02)
SIU	EN (59.26)	ES (57.31)	TH (56.05)	FR (55.67)	ZH (55.00)	BG (54.70)	JA (54.33)	AR (54.11)
VTI	EN (68.10)	ES (65.49)	ZH (65.42)	FR (64.08)	BG (62.82)	JA (61.83)	TH (61.13)	AR (60.00)

Appendix

Table 7: Scenario-wise language rankings averaged over all evaluated models.

Scenario	REPE	ZH	ES	FR	JA	TH	AR	BG
PR	Baseline Language Medical MedVL-XLRepE	72.95 73.77 $\uparrow$ 0.82 73.50 $\uparrow$ 0.55 74.32 $\uparrow$ 1.37	75.96 76.23 $\uparrow$ 0.27 75.41 $\downarrow$ 0.55 77.60 $\uparrow$ 1.64	74.86 77.05 $\uparrow$ 2.19 78.69 $\uparrow$ 3.83 76.23 $\uparrow$ 1.37	71.31 73.22 $\uparrow$ 1.91 73.22 $\uparrow$ 1.91 75.41 $\uparrow$ 4.10	75.41 75.41 $\uparrow$ 0.00 74.59 $\downarrow$ 0.82 77.32 $\uparrow$ 1.91	73.50 73.22 $\downarrow$ 0.28 74.59 $\uparrow$ 1.09 74.59 $\uparrow$ 1.09	72.68 73.77 $\uparrow$ 1.09 75.41 $\uparrow$ 2.73 77.60 $\uparrow$ 4.92
AAR	Baseline Language Medical MedVL-XLRepE	66.10 67.80 $\uparrow$ 1.70 66.95 $\uparrow$ 0.85 69.49 $\uparrow$ 3.39	69.49 69.49 $\uparrow$ 0.00 68.64 $\downarrow$ 0.85 71.19 $\uparrow$ 1.70	70.34 72.03 $\uparrow$ 1.69 72.88 $\uparrow$ 2.54 76.27 $\uparrow$ 5.93	69.49 70.34 $\uparrow$ 0.85 67.80 $\downarrow$ 1.69 75.42 $\uparrow$ 5.93	65.25 66.10 $\uparrow$ 0.85 66.95 $\uparrow$ 1.70 71.19 $\uparrow$ 5.94	67.80 66.95 $\downarrow$ 0.85 68.64 $\uparrow$ 0.84 69.49 $\uparrow$ 1.69	68.64 68.64 $\uparrow$ 0.00 68.64 $\uparrow$ 0.00 72.03 $\uparrow$ 3.39

Appendix

Table 8: Component ablation of MedVL-XLRepE on Qwen3.5-9B. Each cell lists, from top to bottom: Baseline / Language only / Medical only / MedVL-XLRepE. Colored markers show change relative to Baseline.

Group	$n$	Mean (95% CI)	Paired $t$	Wilcoxon	Sign
Overall	84	$+2.38$ [2.05, 2.71]	$14.37^{\dagger}$	$7.8\!\times\!10^{-15}$	80 / 0
Gemma3-12B-IT	28	$+2.08$ [1.53, 2.63]	$7.73^{\dagger}$	$8.3\!\times\!10^{-6}$	26 / 0
Qwen3.5-9B	28	$+3.25$ [2.58, 3.93]	$9.89^{\dagger}$	$5.6\!\times\!10^{-6}$	27 / 0
InternVL3.5-14B	28	$+1.81$ [1.46, 2.17]	$10.61^{\dagger}$	$5.6\!\times\!10^{-6}$	27 / 0

Appendix

Table 9: Statistical significance of MedVL-XLRepE cross-lingual gains, computed over the paired per-configuration gains in Table 2 . Mean gain is in accuracy points (%) with a

95\%

confidence interval. Wilcoxon reports the two-sided signed-rank

p

-value; Sign reports the number of configurations with a positive / negative gain. † denotes

p<10^{-5}

for the paired

t

-test (

p<10^{-15}

for the Overall row).

Panel of Figure 6 — Figure 6: Examples of the four clinical scenarios.

Explore similar work

Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study
Jun 2, 2026Pieter Christy Yan Yudhistira, Dzaki Rafif Malik, Novanto YudistiraVLM Evaluation VLM Robustness
Intelligent System Laboratory, Faculty of Computer Science Brawijaya University, Malang, Indonesia
HealMed: Multilingual Evaluation of Large Language Models in Medicine
Aug 20, 2026Yingjian Chen, Fan Gao, Sherry T. Tong +42Multilingual Language Model Evaluation Medical LLMs
Drew
EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation
Apr 15, 2026Francesco Andrea Causio, Vittorio De Vita, Olivia Riccomi +17Multilingual Language Model Evaluation Multilingual VLM Evaluation
Section of Hygiene, University Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy · Società Italiana Intelligenza Artificiale in Medicina (SIIAM), Rome, Italy · NSBproject, Mantova, Italy +8

Jun 2, 2026J/K move · Enter open · S save

Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study

Pieter Christy Yan Yudhistira, Dzaki Rafif Malik, Novanto Yudistira

Intelligent System Laboratory, Faculty of Computer Science Brawijaya University, Malang, Indonesia

VLM Evaluation VLM Robustness Radiology VLMs Medical VQA Multilingual VLM Evaluation

Open paper AskPDF arXiv

Medical Vision-Language Models (VLMs) are typically evaluated on English radiology visual question answering benchmarks, leaving their robustness under non-English clinical language largely unexplored. We introduce IndoRad-VQA, an Indonesian adaptation of VQA-RAD, to assess whether medical VLMs retain radiology reasoning ability when questions are asked in Bahasa Indonesia. Radiology question-answer pairs are translated into Indonesian with self-evaluation-based quality control to preserve clinical meaning, terminology consistency, and answer equivalence. We evaluate general-purpose, Southeast Asian multilingual, and medical-specific VLMs under English and Indonesian prompting settings. Beyond accuracy, we quantify the language robustness gap between English and Indonesian inputs. We also conduct an error analysis to identify failure modes of question answering, such as yes/no flips, laterality errors, and output-language mismatches. Our findings show that strong performance on English medical VQA benchmarks does not necessarily translate to robust behavior in Indonesian clinical contexts. We observe a performance gap of 8 to 25 percent between the English and Indonesian settings, depending on the evaluation metric. These results highlight the need for more inclusive multilingual evaluation of medical multimodal foundation models. The dataset is available at https://huggingface.co/datasets/Lab-IS/IndoRad-VQA.

Abstract

Explore similar work

Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study

HealMed: Multilingual Evaluation of Large Language Models in Medicine

EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation