cs.CLMar 28, 2026

Culturally Adaptive Explainable LLM Assessment for Multilingual Information Disorder: A Human-in-the-Loop Approach

Authors: Maziar Kianimoghadam Jouneghani

Organizations: University of Turin

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recognizing information disorder is difficult because judgments about manipulation depend on cultural and linguistic context. Yet current Large Language Models (LLMs) often behave as monocultural, English-centric "black boxes," producing fluent rationales that overlook localized framing. Preliminary evidence from the multilingual Information Disorder (InDor) corpus suggests that existing models struggle to explain manipulated news consistently across communities. To address this gap, this ongoing study proposes a Hybrid Intelligence Loop, a human-in-the-loop (HITL) framework that grounds model assessment in human-written rationales from native-speaking annotators. The approach moves beyond static target-language few-shot prompting by pairing English task instructions with dynamically retrieved target-language exemplars drawn from filtered InDor annotations through In-Context Learning (ICL). In the initial pilot, the Exemplar Bank is seeded from these filtered annotations and used to compare static and adaptive prompting on Farsi and Italian news. The study evaluates span and severity prediction, the quality and cultural appropriateness of generated rationales, and model alignment across evaluator groups, providing a testbed for culturally grounded explainable AI.

Figures & tables

TaskGender (M / F)Annotator L1 (Italian / Farsi) [English items]
Severity (F-Score)0.44 / 0.370.36 / 0.47
Rationale Agreement (1–4)3.28 / 2.682.44 / 3.53
Table 1: Alignment between LLaMA 4 outputs and human annotators, stratified by gender and annotator L1 evaluating English news (adapted from the InDor baseline Gemelli et al. (2026) ).
Figure 1: Structural transition from the original baseline to the proposed Culturally Adaptive prompt. The adaptive approach resolves the reasoning and cultural framing mismatch through retrieved target-culture exemplars.
Figure 2: The proposed Hybrid Intelligence Loop. The full architecture supports iterative expert feedback, while the initial pilot evaluates only the seeded Exemplar Bank phase before introducing continuous correction.
Figure 3: Visual breakdown of the four prompting conditions used in the pilot. M1 combines English task instructions with retrieved target-language cultural exemplars, while A1 keeps the same retrieval setup but switches the instruction language back to the target-language.
More in this topic

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

Explore similar work

CardsList
May 19, 2026

Lost in Interpretation: The Plausibility-Faithfulness Trade-off in Cross-Lingual Explanations

LLMs deployed multilingually are often audited via English explanations for non-English inputs. We evaluate extractive explanations ''where the model identifies input token spans as evidence alongside a generated rationale'' and uncover a systematic trade-off: English-pivot explanations can achieve higher span agreement with human rationales while their evidence becomes less causally grounded in the model's prediction, as measured by both comprehensiveness and sufficiency. Across 3 tasks, 5languages, and 2multilingual LLM families, we find that English explanations frequently produce fluent but loosely anchored rationales, with comprehensiveness degrading by up to 5.7x relative to native-language conditions - even as task accuracy remains stable across settings. For socially nuanced classification, English pivots also fail to preserve pragmatic cues, reducing both faithfulness and span agreement. We recommend auditing explanations in the input language, reporting multi-faceted faithfulness metrics beyond lexical overlap, and treating English rationales as communication summaries rather than faithful decision traces.

Somnath Banerjee, Pranav Jha, Rima Hazra +1

Indian Institute of Technology Kharagpur · TCG Crest · National University of Singapore

Language Model AuditingCross-Lingual Reasoning in Language ModelsLLM InterpretabilityExplainability EvaluationMultilingual Language Model EvaluationFaithfulness of Language Model Explanations+3
AskPDF
Oct 19, 2026

The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs

Large Language Models (LLMs) increasingly operate across culturally diverse settings, yet existing evaluations largely measure cultural behavior at the output level. We introduce CultureLens, a reference-grounded benchmark and mechanistic study of how LLMs use cultural information. CultureLens contains 9,261 scenarios across 75 countries and systematically varies country, value, true-norm, and contradictory-norm evidence. Across Llama-3.1-8B, Gemma-3-12B, and Qwen-3.5-27B, stating a contradictory norm lowers aligned answers by 32-47 percentage points, even when the model answers correctly from the country alone. Mechanistically, models default to Anglophone norms when no country is given. The failure under a contradictory norm is not one of missing knowledge, as blocking the norm inside the model restores the correct answer in most cases. Models accept a stated norm according to how it is presented and not whether it is true, which leaves them most exposed to errors in retrieved content. Supervised fine-tuning with contradictory-norm examples raises aligned answers under a contradictory norm to 85-93%, including on unseen countries, at a small cost under a true norm.

Samee Arif, Furkan Danisman, Angana Borah +6

Cultural Bias in Language ModelsCultural Knowledge in Language ModelsCross-Cultural Language Model EvaluationKnowledge Conflicts in Language ModelsMechanistic Interpretability
AskPDF
Apr 2, 2026

Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation

Multilingual large language models (MLLMs) often answer culturally ambiguous questions inconsistently across languages, even for a fixed user persona -- e.g., Shakespeare in English but Cervantes in Spanish for the same British user. To robustly quantify this Cross-lingual Cultural Inconsistency, we introduce Singleton Fleiss's κSκ_S, a metric mathematically resilient to hallucinations. For mitigation, we propose Cross-lingual Cultural Consistent Preference Optimisation (C-3PO), a consensus-driven alignment framework. C-3PO raises κSκ_S by 0.07--0.10 over unaligned models, above every prompting and representation steering baseline averaged across personas, whilst largely preserving explicit user identities, cultural neutrality and intrinsic cultural knowledge; where it does not, a soft-consensus variant recovers most of the cost. Although trained only on multiple-choice questions, C-3PO also improves consistency in open-ended generation. Empirical evaluations demonstrate this inconsistency is more pronounced for lower-resource languages like Indonesian and Persian. Finally, early decoding of intermediate layers reveals that MLLMs implicitly personalise outputs towards the prompt language's stereotypical culture as forward-pass representations stabilise.

Lucas Resck, Isabelle Augenstein, Anna Korhonen

Language Technology Lab, University of Cambridge · University of Copenhagen

Cross-Lingual Consistency in Language ModelsCross-Cultural Language Model EvaluationPreference OptimizationPersonalized Language Model Alignment
AskPDF