Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stability
Authors: Nusrat Jahan Lia, Shubhashis Roy Dipta
Organizations: Institute of Information Technology, University of Dhaka, Dhaka, Bangladesh · University of Maryland, Baltimore County, Baltimore, Maryland, USA
Recent advances in multilingual representation learning aim to bridge the performance gap between high- and low-resource languages, yet their ability to preserve affective meaning across languages remains underexplored, particularly for underrepresented languages like Bengali. This research addresses cross-lingual sentiment misalignment between Bengali and English by introducing a controlled benchmarking framework evaluating four multilingual transformer models on parallel Bengali-English sentence pairs, stratified by dialect, to assess their representational stability. We demonstrate that a compressed model architecture exhibits a 28.7% "Sentiment Inversion Rate," fundamentally misinterpreting positive semantics as negative (or vice versa). Consequently, we identify a cross-lingual sentiment skew that we call "Asymmetric Empathy," where models systematically dampen or artificially amplify the affective weight of Bengali text relative to its exact English counterpart. Finally, we expose a key vulnerability regarding dialectal representation: a "Modern Bias" in the regional model, which exhibits a 57% increase in alignment error when processing the formal Bengali register compared to modern colloquial text. As foundational encoders continue to serve as safety classifiers and reward models for LLM pipelines, cross-lingual reliability becomes a critical concern. We therefore advocate for the integration of "Affective Stability" metrics into future cross-lingual benchmarks to detect and penalize polarity inversions, particularly in low-resource settings.
Figures & tables
Figure 1: The plot maps the predicted English sentiment vector ( X -axis) against the Bengali vector ( Y -axis) for a test sentence. The green corridor represents ideal cross-lingual sentiment alignment ( Y≈X ). While the large-scale XLM-T model successfully preserves polarity, the compressed (mDistilBERT) and regionally specialized (IndicBERT) architectures are projecting a negative English statement into a positive Bengali latent space (Red Zone).
Metric
Tabularis
XLM-T
IndicBERT
mDistilBERT
Mean Div.
0.200
0.276
0.375
0.417
Std Dev.
0.214
0.298
0.607
0.429
Sadhu Div.
0.239
0.286
0.459
0.456
Cholito Div.
0.161
0.266
0.292
0.379
Dialect Gap
0.078
0.020
0.167
0.077
Sadhu Err. Inc. (%)
48.4
7.6
57.1
20.5
Table 1: Metric scores per model
Metric
Formulation
Interpretation
Mean Alignment Error (Divergence)
μD(M)=n1∑i=1nDi
Lower μD indicates better average cross-lingual consistency.
Std. Deviation (Divergence)
σD(M)=n−11∑i=1n(Di−μD)2
Quantifies the variability in alignment quality across the dataset.
Robustness Index
R(M)=n1∑i=1n\mathbbm1[Di<0.1]×100%
% of pairs with negligible divergence; measures the “safe operating zone.”
Inversion Rate
Irate(M)=n1∑i=1nIi×100%
% of sentence pairs exhibiting sentiment polarity flips.
Mean Directional Bias
μB(M)=n1∑i=1nBi
>0 : English favored; <0 : Bengali favored; ≈0 : No systematic language skew.
Formal Penalty (Dialect Gap)
Δdialect(M)=μD(M,Sadhu)−μD(M,Cholito)
>0 implies a “Modern Bias” where the model struggles with formal registers.
Table 2: Summary of population-level alignment and dialectal metrics.
Figure 2: Sentiment Inversion Rate Across Models
Figure 3: Distribution of Alignment Error Density
Model
Mean Div. ( μD )
Inv. Rate ( Irate )
Affective Stability ( AS )
XLM-T
0.276
0.036
0.831
Tabularis
0.200
0.086
0.823
IndicBERT
0.375
0.200
0.650
mDistilBERT
0.417
0.287
0.564
Table 3: Affective Stability ( AS ) of evaluated models.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model Name
Repository (HuggingFace)
XLM-T
cardiffnlp/XLM-Toberta-sentiment Barbieri et al. (2022)
IndicBERT
ai4bharat/IndicBERTv2-sentiment
Tabularis
tabularisai/multilingual-sentiment
mDistilBERT
lxyuan/distilbert-multilingual
Appendix
Table 4: Model Repository Mapping
Figure 4: Directional Bias in Sentiment Scores (English − Bengali)
Figure 5: Illustrative case study validating Asymmetric Empathy in cross-lingual sentiment alignment, demonstrating severe instance-level directional bias.
Large language models (LLMs) frequently exhibit performance biases against regional dialects of low-resource languages. However, frameworks to quantify these disparities remain scarce. We propose a two-phase framework to evaluate dialectal bias, operationalized as comprehension degradation relative to standard Bengali, in LLM question-answering across nine Bengali dialects. First, we translate and gold-label standard Bengali questions into dialectal variants adopting a retrieval-augmented generation (RAG) pipeline to prepare 4,000 question sets. Since traditional translation quality evaluation metrics fail on unstandardized dialects, we evaluate fidelity using an LLM-as-a-judge, which human correlation confirms outperforms legacy metrics. Second, we benchmark 19 LLMs across these gold-labeled sets, running 68,395 RLAIF evaluations validated through multi-judge agreement and human fallback. Our findings reveal severe performance drops linked to linguistic divergence. For instance, responses to the highly divergent Chittagong dialect score 5.44/10, compared to 7.68/10 for Tangail. Furthermore, increased model scale does not consistently mitigate this bias. We contribute a validated translation quality evaluation method, a rigorous benchmark dataset, and a Critical Bias Sensitivity (CBS) metric for safety-critical applications.
K. M. Jubair Sami, Dipto Sumit, Ariyan Hossain +1
Department of Computer Science and Engineering BRAC University, Dhaka, Bangladesh
Safety-aligned large language models often exhibit sycophancy, which is the tendency to affirm users' opinions regardless of factual accuracy. Although well-studied in English, its manifestation in other languages remains largely unexamined, leaving billions of non-English speakers potentially vulnerable to model-validated misinformation. We present the first large-scale, multi-model evaluation of cross-lingual sycophancy, benchmarking six instruction-tuned models across 1.1 million instances spanning 38 languages and 33 topic categories. We identify a consistent resource-tier effect: sycophancy rates spike sharply in low-resource and zero-shot language settings. Critically, this degradation is topic-agnostic, as models fail uniformly across both benign and safety-critical prompts, offering no additional protection where it is most needed. We further identify tokenizer fertility as a structural driver of this alignment collapse. Collectively, our results demonstrate that prevailing alignment methodologies generalize poorly beyond high-resource languages, underscoring the urgent need for equitable multilingual safety techniques.
Current evaluation of multilingual Large Language Models (LLMs) rests on an implicit Translation-Isomorphism Assumption (TIA): that semantic structures across languages are congruent and mutually mappable without loss of information. We argue that this assumption is not merely violated in practice, but ill-posed in principle for a typologically identifiable class of concepts, including pragmatic markers, honorifics, and diachronically stratified terms. We formalize this failure using a usage-cloud framework, representing concepts as point sets of contextualized embeddings. We define α-unalignability as the impossibility of any mapping that simultaneously preserves lexical faithfulness (centroid correspondence) and structural faithfulness (local neighborhood topology). We provide three layers of evidence. Behaviorally, we show that FLORES-200 translation failures are predicted by language family and resource class but not by script, and that LOBSTER reasoning scores vary by family. Mechanistically, we report a Representation-Intervention Gap (RIG) in a nine-model case study on Yami: the models' activations encode a regularity along which Yami groups with other low-resource and Austronesian languages, yet interventions on language-specific neurons show no demonstrated advantage over random masks: the regularity is visible but not usable by this intervention. Finally, we operationalize these findings into a multidimensional diagnostic profile: Cycle-Consistency, Pragmatic-Load Disagreement, Manifold-Curvature Mismatch, and RIG. We argue that collapsing cultural competence into a single scalar incentivizes "probabilistic flattening," and that recognizing the unalignable class is a precondition for AI that respects, rather than erases, cultural divergence. This suggests that multilingual alignment is not a single well-defined objective, but a set of mutually incompatible projections.