Code translation converts source code from one programming language to another while preserving its functionality. Although Large Language Models (LLMs) have improved code translation, their outputs still contain hallucinations: generated code that conflicts with the source program, violates target-language facts, or introduces contextually inconsistent dependencies. These failures are especially harmful in translation because small semantic deviations can silently change program behavior. Existing hallucination studies mainly focus on natural-language generation or code generation, leaving the forms, distribution, and mitigation of hallucinations in cross-language code translation insufficiently characterized. This paper presents a systematic empirical study of LLM hallucinations in code translation. Across three benchmark datasets, six language pairs, and seven LLMs, we construct a taxonomy with three major categories: Semantic Consistency Conflicts, Factual Knowledge Conflicts, and Project Context Deviations. We manually annotate 13,919 hallucination cases and analyze how hallucination types vary across models, datasets, and language directions. Guided by the taxonomy, we further propose HaluCodeTran, a mitigation framework that combines BM25-based retrieval of relevant translation examples with taxonomy-aware reflective refinement. The study provides an execution-tested characterization of code-translation hallucinations and a practical baseline for building more reliable LLM-based code migration tools.
Figures & tables
Figure 1. Taxonomy of Hallucinations in LLM-based Code Translation
Figure 2. Examples of Different Hallucination Types in LLM-Powered Code Translation
Figure 3
model
dataset
FRM
DCV
SCC
BSC
LKC
AKC
FKC
RDC
IFC
DC
PCD
Qwen 2.5
Avatar
33.52%
17.85%
51.37%
8.01%
0.55%
10.20%
18.76%
2.55%
12.39%
14.94%
29.87%
CodeNet
21.41%
32.85%
54.26%
6.44%
0.21%
11.43%
18.09%
0.62%
12.27%
14.76%
27.65%
Transcoder
19.08%
62.61%
81.69%
2.43%
0.26%
3.46%
6.15%
0.38%
6.40%
5.38%
12.16%
DS-Coder
Avatar
36.81%
15.54%
52.35%
9.00%
1.84%
7.98%
18.81%
4.29%
12.88%
11.66%
28.83%
CodeNet
29.22%
24.94%
54.16%
11.84%
2.02%
5.79%
19.65%
1.76%
11.84%
12.59%
26.20%
Transcoder
25.72%
60.49%
86.21%
1.44%
0.72%
2.01%
4.17%
1.15%
3.45%
5.03%
9.63%
Table 2. Average Hallucination Statistics Across Different Large Language Models
Figure 5. The hallucination distribution of different LLMs under various mitigation strategies.
Code translation converts programs between languages, frameworks, and representations while aiming to preserve behavior and sometimes improve safety or performance.
Code translation is one of the core capabilities of LLMs. However, evaluating the correctness of translations remains difficult, as commonly used metrics such as BLEU measure only syntactic similarity, disregarding program semantics. We propose a novel evaluation methodology for code translation tasks, emphasizing semantic equivalence over surface-level string similarity. Our approach applies established compiler testing methodology to a new domain, allowing the assessment of an LLM fine-tuned for binary lifting tasks (i.e. decompiling binaries to higher-level representations). We introduce a semantic correctness score, defined as the proportion of translations that produce correct execution outcomes, and demonstrate its application by evaluating LLM-based and heuristic decompilers. Our findings show that LLM-based approaches significantly outperform heuristic ones, while BLEU scores show negligible correlation with semantic correctness (r = -0.127 to 0.354), demonstrating that syntactic metrics fail to predict functional accuracy.
Julius Näumann, Sven Keidel, Amir Molzam Sharifloo +1
∗TU Darmstadt · †Hessian Center for Artificial Intelligence, Darmstadt, Germany · ‡National Research Center for Applied Cybersecurity ATHENE
Large Language Models for code generation frequently produce hallucinations in Fill-in-the-Middle (FIM) tasks -- plausible but incorrect completions such as invented API methods, invalid parameters, undefined variables, or non-existent imports. These failures pass superficial review yet introduce runtime errors. We introduce Delulu, a verified multi-lingual benchmark of 1,951 FIM samples across 7 languages and 4 hallucination types. Samples are curated through an adversarial pipeline: a frontier LLM generates plausible hallucinations, four diverse judge models evaluate them, embedding-based clustering mines progressively harder examples, self-contained Docker containers verify that golden completions compile while hallucinated variants produce the expected runtime error, and a final human-expert review removes any remaining biased or trivially decidable samples. We evaluate 11 open-weight FIM models from five families spanning 0.5B-32B parameters: a six-point Qwen2.5-Coder scaling slate, plus a cross-family slate (CodeLlama, DeepSeek-Coder-V2, StarCoder2). The strongest model reaches only 84.5% pass@1, no family exceeds 0.77 Edit Similarity, and every family produces hallucination-aligned completions on a non-trivial share of samples, confirming that the difficulty exposed by Delulu is task-intrinsic rather than family-specific. We release the benchmark, containers, and evaluation framework at https://github.com/microsoft/delulu.
Mahdi Erfanian, Nelson Daniel Troncoso, Aashna Garg +4
Recent hallucination detection techniques in large language models (LLMs) focus on directly extracting features from a model's internal representations and training a classifier on these features to detect hallucinations, demonstrating promising results. Notwithstanding this advancement, most internal-state hallucination detection techniques have been explored predominantly in English, raising the question of whether such internal signals generalize across different languages and domains. To address this gap, we present CrossHallu, the first study to evaluate the cross-lingual and cross-domain generalization of hallucination detection using internal representations from six LLMs on the generative question-answering task. We conduct a systematic Arabic <-> English evaluation using TruthfulQA, an Arabic translated version of TruthfulQA, and HalluScore. This evaluation encompasses monolingual training and testing, cross-lingual transfer, cross-domain transfer, and combined cross-lingual and cross-domain transfer. The results reveal that internal-state hallucination signals in LLMs transfer across languages and domains for most models, with cross-lingual performance highly dependent on both class separability and language alignment in the feature space, whereas cross-domain transfer within Arabic varies depending on the training and testing datasets used for the hallucination detector. The code is publicly available at https://github.com/aishaalansari57/CrossHal.
Aisha Alansari, Malak Alkhorasani, Hamzah Luqman
King Fahd University of Petroleum and Minerals · Imam Abdulrahman bin Faisal University