CC-Mediation: Evaluating Large Language Models for Cross-Cultural Conflict Mediation

Suhyun Lee; Wenxuan Zhang; W. Quin Yow; Yang Deng

cs.CLSep 4, 2026

CC-Mediation: Evaluating Large Language Models for Cross-Cultural Conflict Mediation

Authors: Suhyun Lee, Wenxuan Zhang, W. Quin Yow, Yang Deng

Organizations: Department of Artificial Intelligence, Hanyang University, Seoul, Republic of Korea · Singapore University of Technology and Design, Singapore · School of Computing and Information Systems, Singapore Management University, Singapore

Abstract

Cross-cultural mediation by large language models (LLMs) requires deciding both when to intervene and how to respond in culturally grounded conflicts. Progress on this problem has been limited by the lack of (1) mediation datasets with measurable downstream effects and (2) principled metrics for evaluating intercultural stance change. To address these gaps, we introduce CC-Mediation, a cross-cultural mediation benchmark of $1{,}661$ ten-turn dialogues grounded in the Developmental Model of Intercultural Sensitivity (DMIS), containing culturally grounded conflicts, mediation interventions, and post-intervention trajectories. We further propose two DMIS-based evaluation metrics: Trajectory AUC, which measures the persistence of intercultural improvement over time, and a signed Wasserstein-1 distance, which measures the magnitude and direction of shifts in intercultural stance. Both metrics show strong agreement with human judgment of DMIS-grounded stance shift. Using CC-Mediation, we find that current LLMs have limitations on both axes: intervention timing (when) failure stems from a positional prior that ignores dialogue content, while mediation strategy (how) failure arises from a late-layer elicitation collapse rather than a knowledge deficit.

Figures & tables

Figure 1: An example of CC-Mediation and DMIS stage transition. At turn $t_{6}$ , differing cultural perspectives on “ good neighbour ” between Ryan (Cyprus) and Mia (Singapore) trigger a DMIS Defense -stage conflict. The model intervenes with a “Shared-values Reframe” strategy to mitigate adversarial framing and focus on a shared purpose. This mediation effectively lowers cognitive defenses, leading to the Acceptance stage.

Figure 2: CC-Mediation Construction Pipeline. Our four-stage pipeline transforms a cultural-conflict scenario into a labeled mediation dialogue: (1) two LLM agents generate a culturally grounded conflict dialogue via role-play; (2) a judge LLM assigns DMIS stage labels per turn; (3) a mediator LLM inserts $u^{+}$ at $t_{\text{conf}}$ and the dialogue resumes; (4) filtering via a strict utility-gap gate ( $\Delta\text{Trajectory AUC}>0\land\Delta W_{1}>0$ ).

	Train	Eval	Total
# Dialogues	$1{,}503$	$158$	$\mathbf{1{,}661}$
# Speaker utterances	$16{,}448$	$1{,}633$	$\mathbf{18{,}081}$
# GT mediations	$1{,}503$	$158$	$\mathbf{1{,}661}$
# Filtered mediations	$1{,}503$	—	$\mathbf{1{,}503}$
timing-differs	$1{,}313$ ( $87.4\%$ )	—	$\mathbf{1{,}313}$
content-only	$190$ ( $12.6\%$ )	—	$\mathbf{190}$

Table 1: CC-Mediation dataset statistics

Model ( Base / CoT )	TurnAcc	Early%	$\mathbf{\|\text{err}\|}$	Cram. $V$	$\mathbf{r_{\text{rb}}}$
Llama-3.1-8B	32.7 / 30.8	12.6 / 63.5	1.15 / 1.84	.54 / .32	$+.14$ / $-.59$
Gemma-2-9B	22.0 / 0 6.3	66.0 / 93.7	1.94 / 3.13	.20 / .16	$-.52$ / $-.94$
Phi-3.5-mini	11.3 / 18.2	79.9 / 80.5	2.51 / 2.36	.17 / .24	$-.71$ / $-.79$
Claude-3.5-haiku	15.7 / 0 5.7	84.3 / 94.3	2.42 / 3.14	.27 / .19	$-.66$ / $-.94$
Gemini-2.0-flash	33.3 / 17.6	66.7 / 82.4	1.85 / 2.57	.34 / .23	$-.27$ / $-.82$
Llama-3.3-70B	35.8 / 18.9	64.2 / 60.4	1.65 / 2.09	.36 / .27	$-.23$ / $-.60$

Table 3: Conflict-turn detection on CC-Mediation -eval

Panel of Figure 3 — Figure 3: Analysis of positional prior (left) and surface discourse markers (right) across the evaluated LLMs.

Figure 4: Layer-wise DMIS stage decoding accuracy, averaged across three open-weight models.

	AUC		$W_{1}$		Judge
Model	D	ND	D	ND	D	ND
Gemma-2-9B	0.75	0.72	0.78	0.81	3.20	3.34
Llama-3.1-8B	0.85	0.68	0.93	0.75	3.57	4.51
Phi-3.5-mini	0.74	0.69	0.79	0.66	3.69	3.20
Claude-3.5-Haiku	1.48	1.28	0.99	0.82	5.00	3.75
Gemini-2.0-Flash	1.54	1.44	0.81	0.76	4.87	3.34

Table 4: Mediator effectiveness across six LLMs

Model	Variant	turn_acc %	AUC ${}_{\text{all}}$	AUC ${}_{\text{exact}}$ ( $n$ )	$W_{1}$	Judge
Llama-3.1-8B	Base	$32.7$	$+0.622$	$+0.517$ ( $52$ )	$+0.632$	$2.88$
Llama-3.1-8B	SFT	$\mathbf{98.1}$	$\mathbf{+0.892}$	$\mathbf{+0.893}$ ( $\mathbf{156}$ )	$\mathbf{+0.791}$	$\mathbf{4.22}$
Gemma-2-9B	Base	$22.0$	$+0.913$	$+0.659$ ( $35$ )	$+0.895$	$2.62$
Gemma-2-9B	SFT	$\mathbf{96.9}$	$+0.816$	$\mathbf{+0.807}$ ( $\mathbf{153}$ )	$+0.885$	$\mathbf{4.26}$
Phi-3.5-mini	Base	$11.3$	$+0.832$	$+0.412$ ( $18$ )	$+0.944$	$2.28$
Phi-3.5-mini	SFT	$\mathbf{91.8}$	$+0.699$	$\mathbf{+0.711}$ ( $\mathbf{146}$ )	$+0.703$	$\mathbf{2.87}$

Table 5: Comparing Base and CC-Mediation-SFT: Parenthetical

n

denotes the number of scenarios (of 158) in which the model intervened exactly at

t^{*}

; AUC

{}_{\text{all}}

W_{1}

for Base and SFT are computed over different timing distributions and are not directly comparable (see text).

Stage	Description ( <desc> )	Mediation move ( <med> )
Denial	Speakers fail to register cultural difference as a meaningful category and treat the disagreement as if no cultural dimension were in play.	Surface the existence of a cultural dimension the speakers have not yet noticed using low-stakes, curiosity-arousing framing, while providing high support and avoiding deep value contrasts that would push the speakers into Defense.
Defense	Speakers experience cultural difference in a polarized us-versus-them frame and rely on simplistic, evaluative stereotypes of the other party.	Depolarize the conversation by avoiding direct cultural contrasts, emphasizing shared concerns and goals, and framing the in-group/out-group dynamic as a universal human pattern rather than a personal failing of either party.
Minimization	Speakers subsume cultural difference under an assumed universal sameness and treat their own cultural patterns as the neutral baseline that all parties supposedly share.	Turn each speaker’s attention onto their own cultural patterns, making visible that what they treat as universal common sense is in fact specific to their group, and surface the privilege of the dominant party in the conversation.

Table 6: CONFLICT_TABLE : per-stage description and mediation move used to fill the <desc> and <med> placeholders in the construction-pipeline prompts. Note the structural asymmetry across rows—each stage calls for a categorically different move, and an intervention that depolarizes Defense (e.g., “what shared concerns do you both have?”) would reinforce Minimization (which is already operating in the universal-sameness frame). This is the structural reason CC-Mediation supervises one mediation move per stage.

Stage	Definition
Ethnocentric
1 Denial	The inability to perceive cultural difference as a meaningful category; one’s own worldview is treated as the only available frame. Manifests as topic avoidance, disengagement, or treating the other party’s perspective as a non-issue (“doesn’t matter to me”, “live and let live”).
2 Defense	Cultural difference is perceived but framed as a threat to the in-group worldview. The speaker polarizes us-versus-them through simplistic evaluative stereotypes, defending the in-group as superior or denigrating the out-group (“in our way…”, “actually that’s not how it works”, ALL-CAPS evaluatives).
3 Minimization	Cultural difference is acknowledged but trivialized under an assumed universal sameness. The speaker treats their own cultural patterns as the neutral baseline that all parties supposedly share (“deep down we’re all the same”, “some values are just universal”).
Ethnorelative
4 Acceptance	Cultural difference is recognized as real, legitimate, and equally valid; other cultural frames are understood as alternatives rather than deviations from a norm.

Table 7: The six DMIS stages ( bennett1986 ; bennett2017 ) as operationalized in CC-Mediation . CC-Mediation restricts the conflict speaker’s starting position to the three ethnocentric stages (1–3); the targeted success criterion is a transition into the ethnorelative side (4–6).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Criterion	$\%$	Wilson $95\%$ CI	$p$ vs. $50\%$
Naturalness	$86\%$	$[80,91]$	$<10^{-22}$
Transition Smooth.	$79\%$	$[72,85]$	$<10^{-15}$
Conflict Salience	$73\%$	$[65,79]$	$<10^{-9}$

Appendix

Table 8: Human Agree rates on three dialogue-quality criteria, with Wilson

95\%

CI on the binomial proportion and one-sided exact binomial

p

-value against a

50\%

chance baseline (

n=159

). All three criteria reject the chance null at

p<10^{-9}

	Den.	Def.	Min.
A	$64\%$	$77\%$	$92\%$
B	$33\%$	$86\%$	$92\%$
C	$75\%$	$69\%$	$62\%$
Pooled	$\mathbf{57.9\%}$	$\mathbf{77.5\%}$	$\mathbf{82.1\%}$
Wilson 95% CI	$[45.0,69.8]$	$[66.8,85.4]$	$[72.3,89.0]$
$p$ vs. $1/3$ chance	$<10^{-4}$	$<10^{-13}$	$<10^{-15}$

Appendix

Table 9: Human Agree rates on the auto-assigned DMIS stage by annotator and stage. Pooled human rate

72.6\%

over

117

items (Wilson

95\%

[63.9,79.9]

). Per-stage one-sided exact binomial tests against the

1/3

chance baseline (random selection among Denial/Defense/Minimization) reject the null at

p<10^{-4}

on every stage.

Annotator	Acceptance Agree
A	$80\%$ ( $16/20$ )
B	$65\%$ ( $13/20$ )
C	$75\%$ ( $15/20$ )
Pooled	$\mathbf{73.3\%}$ ( $44/60$ )
Wilson $95\%$ CI	$[61.0,\ 82.9]$
$p$ vs. $1/6$ chance	$<10^{-21}$

Appendix

Table 10: Human Agree rate on the auto-assigned Acceptance label of post-conflict turns, under the protocol of Table 9 . Items were distributed across annotators; validity is assessed at the pooled level.

Criterion	Yes/ $n$	Rate	95% CI	$p$
Controllability	$69/93$	$74.2\%$	$[64.5,81.9]$	$<10^{-5}$
Specificity	$80/93$	$86.0\%$	$[77.4,91.7]$	$<10^{-12}$
Form	$75/93$	$80.6\%$	$[71.4,87.4]$	$<10^{-8}$

Appendix

Table 11: Per-utterance quality assessment on ground-truth mediation utterances (

n=93

), with Yes-rate, Wilson

95\%

CI, and one-sided exact binomial

p

-value against a

50\%

chance baseline.

Rater	Agree/ $n$	Rate	Wilson $95\%$ CI	$p$ vs. $50\%$
A	$12/18$	$66.7\%$	$[43.7,83.7]$	$0.119$
B	$5/5$	$100.0\%$	$[56.6,100.0]$	$0.031$
C	$7/10$	$70.0\%$	$[39.7,89.2]$	$0.172$
D	$8/12$	$66.7\%$	$[39.1,86.2]$	$0.194$
E	$16/22$	$72.7\%$	$[51.8,86.8]$	$0.026$
F	$4/5$	$80.0\%$	$[37.6,96.4]$	$0.188$

Appendix

Table 12: Pairwise Agree rates on the joint AUC–

W_{1}

criterion. Each item pairs two mediations (pre- and post-intervention dialogue); Agree means the annotator picked the higher-ranked side.

p

-values are one-sided exact binomial tests against

50\%

. Per-rater rows are descriptive; validity is assessed on the pooled proportion.

	Trajectory AUC	Signed $W_{1}$
Spearman $\rho$	$\mathbf{+0.637}$	$+0.562$
$95\%$ CI	$[+0.43,+0.75]$	$[+0.28,+0.72]$
Pearson $r$	$+0.678$	$+0.602$
$p$	$1.6\!\times\!10^{-4}$	$1.2\!\times\!10^{-3}$

Appendix

Table 13: Sample-level correlation (

n=30

) between trajectory metrics and the human mean Likert rating, averaged across

6

raters per sample. Confidence intervals from bootstrap with

10{,}000

resamples.

	Acceptance-	Acceptance-
	anchored	independent
$\rho$ vs. AUC	$+0.637$	$+0.372$
$p$	$<0.001$	$0.043$
High-AUC mean	$3.38$	$3.62$
Low-AUC mean	$2.36$	$3.31$
$\Delta$	$+1.02$	$+0.31$

Appendix

Table 14: AUC–human correlation by Likert-item vocabulary (identical samples, raters, and scale). The anchored item is the original of Table 13 ; the independent item removes all DMIS Acceptance vocabulary.

Mediator $M$	Continuation $C$	Labeler $L$ (logit source)	AUC	$W_{1}$	Judge
GPT-4o-mini	GPT-4o-mini	GPT-4o-mini (API)	$+0.82$	$+0.83$	$4.55$
GPT-4o-mini	GPT-4o-mini	Qwen3-14B (local logits)	$+0.72$	$+0.96$	$4.60$
GPT-4o-mini	Claude-3.5-Haiku	GPT-4o-mini (API)	$+0.83$	$+1.00$	$4.59$
GPT-4o-mini	Claude-3.5-Haiku	Qwen3-14B (local logits)	$+0.71$	$+1.13$	$4.64$
Claude-3.5-Haiku	GPT-4o-mini	GPT-4o-mini (API)	$+0.90$	$+0.96$	$4.39$
Claude-3.5-Haiku	GPT-4o-mini	Qwen3-14B (local logits)	$+0.75$	$+1.05$	$4.31$

Appendix

Table 15: Cross-Model Robustness of the chosen-side admission. Seven of the eight

(M,C,L)

cells of a

2{\times}2{\times}2

matrix that independently varies the mediator

M

, the continuation simulator

C

, and the labeler

L

. The cross-family pairs (GPT-4o-mini

\leftrightarrow

Claude-3.5-Haiku, GPT-4o-mini logprobs

\leftrightarrow

Qwen3-14B local logits) test whether the chosen mediation’s downstream effect survives when each role is replaced with a model from a different family. Trajectory AUC and signed

W_{1}

are both strictly positive in every completed cell, and the Judge score—which uses a fixed GPT-4o-mini rubric judge independent of

L

—stays above

4.3

throughout. AUC and

W_{1}

use the uniformly spaced developmental scale (

[-5,+5]

); Judge is a

1

–

5

LLM-judge rating.

		Cosine similarity			Coefficient of determination $R^{2}$
Backbone	Position	D	ND	$\Delta^{\text{utt}}_{\text{elic}}$	D	ND	$\Delta^{\text{utt}}_{\text{elic}}$
Llama-3.1-8B-Instruct	Prompt-end	0.475	0.487	$-0.012$	$-0.431$	$-0.382$	$-0.049$
Llama-3.1-8B-Instruct	Utterance-end	0.565	0.566	$-0.001$	$-0.161$	$-0.144$	$-0.018$
Gemma-2-9B-it	Prompt-end	0.527	0.538	$-0.011$	$-0.268$	$-0.214$	$-0.054$
Gemma-2-9B-it	Utterance-end	0.602	0.622	$-0.020$	$-0.045$	$-0.000$	$-0.045$
Phi-3.5-mini-instruct	Prompt-end	0.467	0.483	$-0.016$	$-0.487$	$-0.416$	$-0.071$

Appendix

Table 16: Final-layer values for the utterance-content probe. Across three backbones the rubric vs. no-rubric gap is bounded by

|\Delta^{\text{utt}}_{\text{elic}}|\!\leq\!0.02

on cosine and

\leq\!0.07

R^{2}

, at both positions and of consistently negative sign—an order of magnitude below, and opposite in sign to, the stage-channel

\Delta_{\text{elic}}\!\approx\!+0.15

. The Utterance-end

>

Prompt-end ordering on cosine (gap

0.08

–

0.13

) confirms the probe is responsive to in-context utterance content.

Figure 5: Layerwise DMIS stage decoding accuracy on SFT-trained models (Llama-3.1-8B, Gemma-2-9B, Phi-3.5-mini)

Figure 6: Marginal distribution of $\hat{t}$ for ground truth (GT), base LLMs, and SFT-trained models

Backbone	Holdout	TurnAcc	AUC	$W_{1}$	Judge
Llama-3.1-8B	(b) WVS-dim	$\mathbf{97.9}$	$1.00$	$0.95$	$3.6$
	(a) Country	$84.2$	$0.70$	$0.76$	$2.0$
	(c) Template	$\mathbf{96.8}$	$0.74$	$0.83$	$3.8$
Gemma-2-9B	(b) WVS-dim	$\mathbf{95.2}$	$0.93$	$0.88$	$4.1$
	(a) Country	$79.4$	$0.70$	$0.86$	$2.1$
	(c) Template	$\mathbf{97.3}$	$0.79$	$0.80$	$4.1$

Appendix

Table 17: Held-out distribution-shift results. Held-out scenarios ((a) Country

n{=}905

, (b) WVS-dim

n{=}330

, (c) Template

n{=}222

, identical across backbones) are entirely removed from training and each backbone is retrained per axis (

9

configurations). Reference: in-distribution SFT turn-acc

=98.1\%

(Llama-3.1-8B).

		Llama-3.1-8B	Gemma-2-9B	Phi-3.5-mini
		( $n{=}52$ )	( $n{=}34$ )	( $n{=}17$ )
AUC	Base $\rightarrow$ SFT	$+0.52\rightarrow+0.67$	$+0.68\rightarrow+0.86$	$+0.43\rightarrow+0.49$
AUC	$\Delta$ ( $p$ )	$+0.16$ ( $.30$ )	$+0.18$ ( $.24$ )	$+0.06$ ( $.67$ )
$W_{1}$	Base $\rightarrow$ SFT	$+0.65\rightarrow+0.77$	$+0.71\rightarrow+0.75$	$+0.43\rightarrow+0.34$
$W_{1}$	$\Delta$ ( $p$ )	$+0.12$ ( $.15$ )	$+0.05$ ( $.44$ )	$-0.09$ ( $1.00$ )
Judge	Base $\rightarrow$ SFT	$2.90\rightarrow 4.69$	$3.06\rightarrow 4.21$	$2.47\rightarrow 2.76$

Appendix

Table 18: Base vs. SFT on the identical scenarios where both intervened at

t^{*}

(paired, two-sided Wilcoxon signed-rank). Bold: significant at

p<0.001

Subset	$n$	AUC	$W_{1}$	Judge
Filter-passed	$41$ ( $25.9\%$ )	$+2.38$	$+1.80$	$4.93$
Filter-failed	$117$ ( $74.1\%$ )	$+1.38$	$+1.45$	$4.95$
All (unfiltered)	$158$	$+1.64$	$+1.54$	$4.94$

Appendix

Table 19: Evaluation-GT performance split by the Step-4 gate condition, applied post hoc for analysis only.

Model	ND	D	$\Delta$
Claude-3.5-Haiku	$88.5\%$	$99.4\%$	$+10.9$ pp
Llama-3.3-70B	$92.4\%$	$100\%$	$+7.6$ pp
Gemini-2.0-Flash	$92.4\%$	$90.4\%$	$-2.0$ pp
Llama-3.1-8B	$93.1\%$	$87.4\%$	$-5.7$ pp
Phi-3.5-mini	$67.9\%$	$89.3\%$	$+21.4$ pp
Gemma-2-9B	$\mathbf{82.4\%}$	$\mathbf{61.6\%}$	$\mathbf{-20.8}$ pp

Appendix

Table 20: Change in stage-marker application rate when the rubric is provided (ND

\rightarrow

D), by model.

Figure 7: The CC-Mediation dual-head architecture (used by CC-Mediation-SFT). A scenario of $T$ speaker turns is serialised with five added tokens: <TURN_END> bounds every turn and provides the sole hidden state read by the decision head, [A] / [B] prefix each speaker, and [SPEAK] / [/SPEAK] delimit the mediation utterance — inserted at the gold turn $t^{*}$ during training and at the predicted turn $\hat{t}$ at inference. A single transformer backbone produces per-position hidden states. At every <TURN_END> the decision head maps $h_{t}$ to a logit $\ell_{t}$ , yielding $\hat{p}_{t}=\sigma(\ell_{t})$ ; the first turn with $\hat{p}_{t}\geq 0.5$ is chosen as $\hat{t}$ . At $\hat{t}$ the LM head reads the hidden states inside the SPEAK span and autoregressively decodes $\hat{u}$ until [/SPEAK] or EOS. CC-Mediation-SFT trains the backbone, the decision head, and the LM head jointly under the SFT loss (§ E.3 ).

Dataset	speakers	conflict	dialogue
	Cross-cult.	Cult.-value	Multi-turn
SocialCC ( wu-etal-2025-socialcc )	✓	$\triangle$	✓
KODIS ( hale2025kodis )	✓	✗	✓
CulturePark ( li2024culturepark )	✓	$\triangle$	✓
CV survey dial. ( cao2024cudialog )	✓	$\triangle$	✓
Hate-speech interv. ( qian2019hate )	✗	✗	✓

Appendix

Table 21: Coverage of the three requirements.

\triangle

(conflict): cultural values addressed without explicit inter-speaker conflict;

\triangle

(speakers): cultural background reflected but not via interaction between speakers of different cultures.

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

Sep 10, 2026

ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation

Dispute mediation is essential for maintaining social harmony and resilience, yet developing skilled mediators is costly and time-consuming. Existing LLM-based mediation research remains limited by unrealistic task formulations, low-fidelity datasets, and coarse evaluation metrics that obscure turn-by-turn dynamics. To address these gaps, we introduce ProMediConv, a novel benchmarking framework that models mediation as a proactive, multi-stage, and party-aware dialogue process incorporating 11 mediation strategies and four party behavior pattern (BP) states. Using 972 complete real-world cases, we construct a high-fidelity mediation dataset with utterance-level annotations of strategies and BP states. Furthermore, to better assess agent impact, we propose MAD (Mean Attribute Difference), a fine-grained metric that captures BP shifts throughout the dialogue. Leveraging this framework, we establish a comprehensive benchmark by evaluating diverse models alongside our tailored baseline ProMediAgent. Extensive empirical analyses reveal critical behavioral phenomena and underscore the persistent challenges current models face in dynamic, multi-party mediation. Ultimately, ProMediConv provides a rigorous foundation and a vital quantitative standard for advancing AI-assisted conflict resolution. Our dataset and codebase are accessible at https://github.com/ZsWei66/ProMediConv_repo.

Zesheng Wei, Mengfan Li, Wenhao Liu +3

University of Science and Technology of China · Singapore Management University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

Automated Negotiation LLM Agent Evaluation AI Agent Benchmarks Multi-Turn Dialogue Evaluation Conversational Agents

Ask PDF

Jun 4, 2026

SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations

Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting emotions, intentions, and context. Existing testbeds rely on a few expert-authored domains, vary mainly strategic posture, and score every turn against every topic, introducing off-topic noise. We introduce SoCRATES, a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It constructs scenarios from real conflicts through an agentic pipeline across eight domains, probes five socio-cognitive adaptation axes (strategic posture, party composition, history length, emotional reactivity, and cultural identity), and scores each topic only on the turns that advance it via a topic-localized evaluator. The evaluator reaches 0.82 alignment with human experts, more than doubling a per-turn baseline. Benchmarking eight frontier LLMs, we find that even the strongest mediator closes only about a third of the unmediated consensus gap under diverse and realistic testbeds, with performance varying sharply by socio-cognitive axis, highlighting that progress lies in social adaptation to diverse conditions.

Taewon Yun, Hyeonseong Park, Jeonghwan Choi +3

1KAIST · 2Chungnam National University

LLM Agent Evaluation AI Agent Benchmarks

Ask PDF

Apr 2, 2026

Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation

Multilingual large language models (MLLMs) often answer culturally ambiguous questions inconsistently across languages, even for a fixed user persona -- e.g., Shakespeare in English but Cervantes in Spanish for the same British user. To robustly quantify this Cross-lingual Cultural Inconsistency, we introduce Singleton Fleiss's

κ_S

, a metric mathematically resilient to hallucinations. For mitigation, we propose Cross-lingual Cultural Consistent Preference Optimisation (C-3PO), a consensus-driven alignment framework. C-3PO raises

κ_S

by 0.07--0.10 over unaligned models, above every prompting and representation steering baseline averaged across personas, whilst largely preserving explicit user identities, cultural neutrality and intrinsic cultural knowledge; where it does not, a soft-consensus variant recovers most of the cost. Although trained only on multiple-choice questions, C-3PO also improves consistency in open-ended generation. Empirical evaluations demonstrate this inconsistency is more pronounced for lower-resource languages like Indonesian and Persian. Finally, early decoding of intermediate layers reveals that MLLMs implicitly personalise outputs towards the prompt language's stereotypical culture as forward-pass representations stabilise.

Lucas Resck, Isabelle Augenstein, Anna Korhonen

Language Technology Lab, University of Cambridge · University of Copenhagen

Cross-Lingual Consistency in Language Models Cross-Cultural Language Model Evaluation Preference Optimization Personalized Language Model Alignment

Ask PDF

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation

SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations

Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation