To Lie or Not to Lie? Investigating The Biased Spread of Global Lies by LLMs
Authors: Zohaib Khan, Mustafa Dogan, Ifeoma Okoh, Pouya Sadeghi, Siddhartha Shrestha, Sergius Justus Nyah, Mahmoud O. Mokhiamar, Michael J. Ryan, Tarek Naous
Organizations: Fatima Fellowship · Stanford University · Georgia Institute of Technology
Misinformation is on the rise, and the strong writing capabilities of LLMs lower the barrier for malicious actors to produce and disseminate false information. We study how LLMs behave when prompted to spread misinformation across languages and target countries, and introduce GlobalLies, a multilingual parallel dataset of 440 misinformation generation prompt templates and 6,867 entities, spanning 8 languages and 195 countries. Using both human annotations and large-scale LLM-as-a-judge evaluations across hundreds of thousands of generations from state-of-the-art models, we show that misinformation generation varies systematically based on the country being discussed. Propagation of lies by LLMs is substantially higher in many lower-resource languages and for countries with a lower Human Development Index (HDI). We find that existing mitigation strategies provide uneven protection: input safety classifiers exhibit cross-lingual gaps, and retrieval-augmented fact-checking remains inconsistent across regions due to unequal information availability. We release GlobalLies for research purposes, aiming to support the development of mitigation strategies to reduce the spread of global misinformation: https://github.com/zohaib-khan5040/globallies
Figures & tables
Figure 1: Examples generations by GPT-4o when prompted by a bad actor to generate articles about the same false claims for different entities (political figures and countries). The model can be selective in generating the article (complies for a Lebanese political figure but refuses for a British one), or can sometimes propagate misinformation regardless of the entity involved.
Figure 2: Misinformation generation rates for (a) Llama-3.3-70B and (b) GPT-4o across prompting languages (columns) and target regions (rows), according to human annotations of outputs. Both models exhibit lower misinformation generation rates in English and for U.S.-centric contexts, with higher rates observed in many non-English and non-Western settings. Similar results for other models are reported in Appendix C.1 .
Figure 3: Misinformation generation rates of Llama-3.3-70B on a global scale: (a) misinformation generation rate percentiles in English across countries highlight substantial variations, (b) rates plotted against country-level Human Development Index (HDI) reveal a statistically significant negative correlation, with higher rates observed for countries with lower HDI. Similar results in non-English languages are found and reported in Appendix C.2 .
Table 2: Percentage of misinformation prompts across languages and Guard models classified as Unsafe or Safe . More recent and advanced guards are increasingly better at categorizing misinformation generation prompts as being unsafe as compared to older variants, but the performance is still less than ideal.
Figure 4: Fact-checking accuracy of our retrieval pipeline when tested on factual and misinformation claims. Performance is often best for each region when searching with the native language.
Figure 5: Misinformation generation rates for Llama3.3-70B on misinformation and factual prompts when prompting the model in a 0-shot manner without vs. with our RAG setup where relevant documents are first retrieved from reliable sources. Models are prone to generate articles but become a lot more wary when asked to find evidence to corroborate the claim, be it factual or misinformation.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Distribution entities by entity types in the GlobalLies .
Figure 7: Topic distribution of GlobalLies as annotated by humans.
Figure 8: Agreement between LLM-as-a-Judge and annotators across languages. Each bar shows the accuracy for a Judge evaluating responses in a specific language.
Figure 9: Misinformation Generation Rates for Llama-3.3-70B in our scaled analysis for 195 countries, grouped into 12 geographical regions. Note that in the heatmap, values are reported as-is with colors normalize per-language (column-wise normalization).
Figure 10: Misinformation Generation Rates for Qwen2.5-72B, annotated by humans.
Figure 11: Misinformation generation rates of Qwen2.5-72B acting as the generator, with Llama-3.3-70B as the judge. (a) percentile in English across countries, (b) illustrates cross-lingual regional compliance trends. Note that in the heatmap, values are reported as-is with colors normalize per-language (column-wise normalization).
Figure 12: Misinformation generation rates of Gemma3-27B acting as the generator, with Llama-3.3-70B as the judge. (a) percentile in English across countries, (b) illustrates cross-lingual regional compliance trends. Note that in the heatmap, values are reported as-is with colors normalize per-language (column-wise normalization).
Entities
English
Arabic
French
Igbo
Nepali
Turkish
Farsi
Urdu
Religious Group
0.875
0.752
0.670
0.775
0.850
0.837
0.599
0.637
Nationality
0.856
0.807
0.726
0.746
0.787
0.885
0.618
0.740
News Agency
0.840
0.889
0.732
0.761
0.888
0.935
0.645
0.783
City
0.783
0.774
0.578
0.722
0.836
0.877
0.595
0.720
Country
0.774
0.802
0.703
0.759
0.811
0.901
0.616
0.739
Political Figure
0.575
0.719
0.485
0.674
0.776
0.836
0.598
0.659
Appendix
Table 4: Misinformation Generation Rates across countries for each category’s templates.
Table 6: Percentage of misinformation prompts across languages for ShieldGemma-27B classified as Unsafe or Safe . The model significantly underperforms Llama-Guard-3-8B despite being larger in size.
Figure 13: False Negative and False Positive Rates for the RAG pipeline over all countries and languages.
Figure 14: The judge prompt used to measure compliance on the templated dataset.
Figure 15: Prompt used to design a search query to retrieve documents on the web pertaining to a given user prompt.
Figure 16: Prompt used to evaluate whether a prompt is factual or not, with respect to retrieved documents.
Figure 17: The prompt template used for ShieldGemma-27B to classify instructions as being misinformation or not.