Organizations: University of Tübingen, Germany · Artificial Intelligence Institute, University of South Carolina, USA · Thapar Institute of Engineering & Technology, India · Indian Institute of Technology Kanpur, India · Indian AI Research Organization, India
Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multilingual deployment, from retrieval-augmented generation pipelines returning mixed-language passages to users pasting multilingual web content. We introduce Multilingual Distractor Interference (MDI), a lightweight and fully replicable evaluation protocol requiring no new data or annotation, in which factual questions are preceded by a semantically irrelevant foreign-language sentence, and evaluate five instruction-tuned LLMs across TruthfulQA and TriviaQA under eight distractor conditions (40,000 evaluations). Our central finding is a metric confound: for Llama-3.1-8B under a Hindi distractor, 58% of responses switch to Devanagari script, yielding a raw hallucination proxy of 0.710, but manual review reveals that 120 of 148 script-switched responses that were correct under clean conditions remain semantically correct despite being written in the wrong script, reducing the adjusted semantic hallucination rate to 0.470. All other models respond through abstention escalation with no hallucination increase. A paragraph-length English distractor triggers near-universal abstention (0.806-0.998) across all models, consistent with reading-comprehension confusion, a failure mode with direct consequences for multilingual RAG pipelines. TruthfulQA multiple-choice accuracy is unaffected under all single-sentence conditions. These results show that exact-match hallucination rates in mixed-language settings should be decomposed into script-switching and semantic error components before drawing conclusions about model reliability.
Figures & tables
Figure 1: Multilingual Distractor Interference (MDI) evaluation pipeline and key findings. (a) An English factual QA prompt is contaminated with a semantically irrelevant foreign-language sentence. (b) Eight distractor conditions: clean, Spanish/Hindi/Chinese and their ×3 variants, and a paragraph-length English Wikipedia excerpt (wiki_para). (c) Five instruction-tuned LLMs evaluated under identical deterministic decoding settings. (d) Metrics: TruthfulQA accuracy and calibration; TriviaQA exact-match accuracy, abstention rate, hallucination proxy rate, and script-switching rate. (e) Three failure modes: script-switching hallucination (Llama-3.1-8B, Hindi), abstention escalation (Phi-3, Mistral-7B, Qwen-2.5), and stability (all models, TruthfulQA MC). Raw hallucination proxy of 0.710 adjusts to 0.470 after accounting for script-switched but semantically correct responses identified through manual review.
Model
Family
Params
Supported languages
Llama-3.1-8B-Instruct
Meta Llama
8B
8 (incl. Hindi, Chinese † )
Llama-3.2-3B-Instruct
Meta Llama
3B
8 (incl. Hindi, Chinese † )
Phi-3-mini-4k-instruct
Microsoft Phi
3.8B
English-primary
Mistral-7B-Instruct-v0.2
Mistral AI
7B
English-primary
Qwen2.5-3B-Instruct
Alibaba Qwen
3B
29+ (incl. Chinese)
Table 1: Models evaluated. All loaded in float16 on a single GPU via HuggingFace Transformers; inference uses greedy decoding throughout. Language support as stated in official model cards.
Model
Clean
Worst
Δ
p
Llama-3.1-8B
0.510
0.478 ( wiki )
− 0.032
ns
Llama-3.2-3B
0.554
0.454 ( zh_l )
− 0.100
∗∗∗
Phi-3-mini
0.652
0.616 ( wiki )
− 0.036
ns
Mistral-7B
0.678
0.626 ( wiki )
− 0.052
∗∗
Qwen-2.5-3B
0.440
0.440 ( es_l )
+ 0.000
ns
Table 2: TruthfulQA accuracy: clean baseline and worst distractor condition. 95% CIs via bootstrap ( n=2000 ). ns : not significant; ∗∗p<0.01 ; ∗∗∗p<0.001 .
Accuracy
Hallu. Proxy
Model
Clean
Δ
Clean
Δ
Deva%
Llama-3.1-8B
0.518
−0.290∗∗∗
0.416
+0.294∗∗∗
58.2
Llama-3.2-3B
0.410
−0.052∗∗∗
0.518
+0.018ns
1.2
Phi-3-mini
0.404
−0.020ns
0.514
−0.146∗∗∗
1.0
Mistral-7B
0.158
+0.010ns
0.744
−0.162∗∗∗
0.6
Qwen-2.5-3B
0.266
−0.124∗∗∗
0.382
−0.254∗∗∗
0.0
Table 3: TriviaQA results under Hindi distractor. “Deva%” = Devanagari script-switching rate. Δ columns are distractor − clean. ∗∗∗p<0.001 ; ns : not significant.
Figure 2: (a) TriviaQA response composition (correct / abstain / hallucination proxy) for each model under clean (C) and Hindi (H) distractor conditions. For Llama-3.1-8B, the orange segment shows the adjusted semantic hallucination rate (0.470) after removing script-switched but semantically correct responses from the raw proxy (0.710). (b) Devanagari script-switching rate under the Hindi distractor. Only Llama-3.1-8B switches spontaneously at scale. (c) Abstention rate under clean vs. paragraph-length English distractor ( wiki_para ), showing near-universal abstention across all models.
Model
Clean
Hindi
Δ Hi
wiki
Llama-3.1-8B
0.066
0.062
− 0.004
0.806
Llama-3.2-3B
0.072
0.106
+ 0.034
0.424
Phi-3-mini
0.082
0.248
+ 0.166
0.538
Mistral-7B
0.098
0.250
+ 0.152
0.860
Qwen-2.5-3B
0.352
0.730
+ 0.378
0.998
Table 4: TriviaQA abstention rates under Hindi and wiki_para conditions.
Most hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource languages. We investigate faithfulness hallucinations, defined as model-generated content that is fluent and plausible but diverges from the provided input or is internally inconsistent. Leveraging the multilingual MultiWikiQA dataset, we utilize the LettuceDetect framework to create synthetic hallucination datasets for 306 languages, from which we train token-level hallucination classifiers for 30 European languages. In this work, we present evaluations of model hallucinations on a selection of languages: English, Danish, German, and Icelandic. Using these classifiers, we evaluate the hallucination rates for Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B, and cogito-v1-preview-llama-70B. Our classifiers reveal notably higher hallucination rates for Qwen3-0.6B (up to 60% of answers containing at least one hallucination, peaking in Icelandic) and generally lower rates for larger models, with cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B performing best on most languages. Hallucination rates are consistently higher for lower-resource languages, particularly Icelandic.
Freja Thoresen, Dan Saattrup Smart
Alexandra Institute · Rued Langgaards Vej 7, 2300 København
Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet (Linstr,Lcontent,Lresp). Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2{,}430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution.
Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating genuine factual understanding. We introduce Systematic Wikidata-based Object-Relation Distortion (SWORD), a benchmark that evaluates whether models consistently reject factual errors across languages. SWORD generates syntactically well-formed but factually incorrect statements in eight widely spoken languages through controlled perturbations of Wikidata triples, ranging from random entity substitutions to semantically plausible property-based selections. Our distortion-based evaluation surfaces two critical insights that remain entirely obscured by conventional benchmarks. First, models counterintuitively achieve higher accuracy on semantically plausible distortions than on nonsensical random substitutions, suggesting reliance on distributional familiarity rather than genuine factual verification. Second, models exhibiting comparable baseline accuracy across languages show substantial performance degradation specifically on (East) Asian languages when presented with distorted statements, with cross-lingual performance gaps reaching up to 28 percentage points (49% relative reduction) in some models. These findings demonstrate that multilingual factual reasoning involves asymmetric capabilities that aggregate accuracy metrics systematically obscure.
Sanghyeok Park, Minji Kang, Hosung Kwak +1
Soongsil University, Seoul, Republic of Korea · KAIST, Daejeon, Republic of Korea