Bengali has no grammatical gender, but honorificity is obligatory: every third-person reference must choose ordinary or respectful register. English has no such signal, so the system must choose a level of deference that the source never specified. We use this property to measure how large language models (LLMs) allocate respect in English--Bengali translation. Our evaluation set contains 2{,}016 sentences across 72 occupations. Each appears in seven sentence frames under four pronoun conditions. Masculine and feminine versions differ by one word, forming counterfactual pairs. We report a single Honorific Bias Score (HBS) for each model on a 0--100 scale, covering both the gender and occupation axes. A lower score means that respect is allocated more evenly. For the occupation analysis, we compute each occupation's honorific rate across the seven no-pronoun sentence frames. The standard deviation of these 72 occupation-level rates ranges from 17.2 to 35.5 percentage points across models, showing substantial variation in how honorifics are allocated across occupations. Occupation is associated with 3.2% to 51.5% of the variance in register choice, compared with at most 1.2% for gender. Gender effects are much smaller overall: only two of the nine models show significant directional gender gaps, and they favor opposite genders. HBS ranges from 19.9 to 70.1 across the nine models, showing substantial variation in how evenly the models allocate respect.
Figures & tables
Figure 1: One matched pair from our set, as translated by Llama 3.3 70B. The two English sentences differ by a single pronoun, and both say the scientist is respected. The Bengali differs in register: [bn] ( tini ) and [bn] ( koren ) are respectful forms, [bn] ( shey ) and [bn] ( kore ) are not. Nothing in the English asked for that difference.
ID
English source
Neutral
T1
The {occ} said she would arrive at nine.
T2
The {occ} finished her work and went home.
T3
I met the {occ} yesterday and she was very helpful.
T4
The {occ} explained that she had already submitted the report.
T5
The {occ} is not available right now, but she will call back later.
Respect- [-1pt] primed
P1
Everyone respects the {occ} because she works very hard.
Table 1: The seven sentence frames, shown in the feminine condition. T1–T5 contain no explicit cue of deference, while P1–P2 explicitly express respect.
Condition
English source
m
The doctor said he would arrive at nine.
f
The doctor said she would arrive at nine.
they
The doctor said they would arrive at nine.
none
The doctor will arrive at nine.
Both are valid Bengali. The model must choose one.
honorific
[bn]
Table 2: The four conditions, shown for T1 with doctor . m and f form the matched pair; none removes the pronoun entirely and serves as the no-pronoun condition, rewritten so the occupation remains the subject and the finite verb remains clause-final. The English source does not specify a level of deference. Bengali requires a register choice, and the two renderings below differ only in register.
Model
Developer
Vertex AI model ID
Gemini 2.5 Flash
Google
gemini-2.5-flash
Gemma 4 26B
Google
gemma-4-26b-a4b-it-maas
Llama 3.3 70B
Meta
meta/llama-3.3-70b-instruct-maas
Llama 4 Scout
Meta
meta/llama-4-scout-17b-16e-instruct-maas
Llama 4 Maverick
Meta
meta/llama-4-maverick-17b-128e-instruct-maas
DeepSeek-V3.1
DeepSeek
deepseek-ai/deepseek-v3.1-maas
Table 3: Models and Vertex AI model identifiers used in the evaluation.
Gender ( he / she pairs)
Occupation
Summary
Model
HAI
GSR
p
η2
spread
η2
HBS
mix
Llama 4 Maverick
−7.19
10.4
<.001
0.50
35.5
51.5
70.1
0.5
Llama 3.3 70B
+7.77
11.6
<.001
0.88
35.2
30.4
68.8
8.0
Llama 4 Scout
+3.01
9.8
0.269
0.11
33.7
29.2
49.0
4.1
DeepSeek-V3.1
−0.84
10.9
1.000
0.02
33.2
40.4
37.0
5.1
Kimi K2 Thinking
−2.85
12.5
0.556
0.10
30.7
25.0
42.1
6.2
Table 4: Gender and occupation effects across the nine models, sorted by occupation spread. HAI and spread are reported in percentage points; GSR and mix are percentages. η2 gives the percentage of variance in register choice associated with each factor. p is the Holm-corrected p -value from McNemar’s exact test. HBS is the 0–100 Honorific Bias Score defined in Section 3.4 . Bold marks the minimum value for GSR, spread, η2 , and HBS; for HAI, it marks the value closest to zero.
Large Language Models have demonstrated strong multilingual fluency, yet fluency alone does not guarantee socially appropriate language use. In high-context languages, communicative competence requires sensitivity to social hierarchy, relational roles, and interactional norms that are encoded directly in everyday language. Bangla exemplifies this challenge through its three-tiered pronominal system, kinship-based addressing, and culturally embedded social customs. We introduce BanglaSocialBench, the first benchmark designed to evaluate sociopragmatic competence in Bangla through context-dependent language use rather than factual recall. The benchmark spans three domains: Bangla Address Terms, Kinship Reasoning, and Social Customs, comprising 1,719 culturally grounded instances written and verified by native Bangla speakers. We evaluate twelve contemporary LLMs in a zero-shot setting and observe systematic patterns of cultural misalignment. Models frequently default to overly formal address forms, fail to recognize multiple socially acceptable address pronouns, and conflate kinship terminology across religious contexts. Our findings show that sociopragmatic failures are often structured and non-random; for example, inappropriate addressing choices concentrate heavily in downward-hierarchy (Elder→Younger) and informal contexts. This reveals persistent limitations in how current LLMs infer and apply culturally appropriate language use in realistic Bangladeshi social interactions.
Tanvir Ahmed Sijan, S. M Golam Rifat, Pankaj Chowdhury Partha +2
Jahangirnagar University, Dhaka, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Bangladesh University of Engineering and Technology, Dhaka, Bangladesh
Large language models (LLMs) increasingly participate in emotionally sensitive social conversations, where responses may shift from balanced support toward excessive validation or escalatory alignment. Existing sycophancy research primarily focuses on factual agreement and instruction-following settings, leaving culturally grounded conversational sycophancy underexplored. We introduce BenSyc, the first benchmark for studying conversational sycophancy in Bengali social contexts. Starting from 11,840 Reddit posts and 170k comments collected from communities across Bangladesh and West Bengal, we construct a human-validated benchmark with binary labels and a fine-grained five-level taxonomy spanning Invalidation, Neutral, Support, Validation, and Escalation. We evaluate more than 15 open and proprietary LLMs on conversational alignment classification and response generation tasks. Results show that distinguishing empathetic support from reinforcement-oriented validation remains challenging even for frontier instruction-tuned models: the best system achieves only 61.8 Macro-F1 on binary detection and 61.7 Macro-F1 on five-class classification. In generation settings, several models frequently produce strongly validating or escalatory responses in emotionally charged situations. Our findings highlight substantial variation across model families and conversational behaviors, underscoring the importance of culturally grounded multilingual benchmarks for evaluating socially aligned conversational AI systems.
Kazi Noshin, Sajib Acharjee Dip, Ranat Das Prangon +4
University of Illinois Urbana-Champaign, USA · Virginia Tech, USA · Bangladesh University of Engineering and Technology, Bangladesh +2
Large language models work well on English and behave in poorly understood ways on languages typologically far from it. Japanese is a clean example, where evaluation still leans on translation quality and JGLUE-style benchmarks, which roll lexical, syntactic and pragmatic competence into a single score. The phenomena on which general-purpose models fail Japanese users are pragmatic: honorifics, in-group and out-group reference, context-sensitive politeness, zero anaphora. I introduce J-PragEval-v0, a minimal-pair benchmark isolating four such phenomena from surface fluency, and combine it with linear probes and teacher-forced log-probability evaluation to ask where inside TinySwallow-1.5B (28 layers, hidden size 1536) the corresponding contrasts live. The four features split three ways. Honorific register sits cleanly in the residual stream: 0.96 balanced accuracy at layer 15, and the model flips its preferred continuation with the scenario on 93 percent of items. Implicit subject and in-group reference are not linearly decodable at the final prompt token (0.48 and 0.38), yet flip rates are 0.77 and 0.79, so the contrast is worked out during generation rather than stored at the prompt. Indirect refusal is the negative case: 0.95 probe accuracy collapsing to a 0.43 flip rate under length-normalised teacher forcing, because the current minimal pairs conflate politeness with continuation length. I also specify Pragmatic Representation Steering, a parameter-free inference-time method that edits residual-stream activations along the class-mean-difference directions probing identifies. Feasibility is argued indirectly rather than demonstrated: the contrastive activation addition baseline, the same geometry the method would inject, recovers probe accuracy within one to two points of logistic regression wherever a linear signal exists. Scaling to Llama-3.1-Swallow-8B is the next step.