CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
Authors: Bryan Chen Zhengyu Tan, Weihua Zheng, Thong T. Doan, Bich Ngoc Doan, Jia Wang Peh, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Zhengyuan Liu, JinYeong Bak, Wafi Shamdi, Soo Kai Chie, Liew Yu Siong, Aina Azyyati Binti Mohamad Rezal, Lew Yan Yan Vanessa, Huadan Wu, Dylan Raharja, Nadya Yuki Wangsajaya, Akane Fukushige, Kazushi Kato, Koji Inoue, Tatsuya Kawahara, Jaehyung Seo, Dongjun Kim, Seungyoon Lee, Zi Haur Pang, Rui Yang Tan, Charibeth Ko Cheng, Maria Regina Justina Estuar, Jann Railey Montalan, Pham Minh Duc, Roy Ka-Wei Lee
Organizations: Singapore University of Technology and Design (SUTD), Singapore · Agency for Science, Technology and Research (A*STAR), Singapore · École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · Microsoft Research Asia (MSRA), Beijing, China · Sungkyunkwan University (SKKU), Republic of Korea · Universiti Brunei Darussalam (UBD), Brunei Darussalam · China University of Petroleum (East China), Qingdao, China · Nanyang Technological University (NTU), Singapore · Kyoto University, Kyoto, Japan · Konkuk University, Seoul, Republic of Korea · Upstage AI, Republic of Korea · Korea University, Seoul, Republic of Korea · De La Salle University (DLSU), Manila, Philippines · Ateneo de Manila University (ADMU), Quezon City, Philippines · AI Singapore (AISG), Singapore · University of British Columbia (UBC), Vancouver, Canada
Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking practical help over multiple turns in culturally grounded scenarios. We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue that covers 10 East and Southeast Asian regions, 58 subgroup identities, and 7 domains. Each simulated and evaluated episode produces a scored interaction where the assistant assists the user and infers cultural constraints from partial information. The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues. In our benchmark evaluation of 18 models, GPT-5 mini achieves the highest assistance quality. Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment. Performance gains from fine-tuning on 27,860 high-quality CultureConverse-DS samples improve in-domain assistance and transfer out-of-domain to cultural MCQ and safety classification benchmarks. We release the harness, both splits, and judge prompts to support interactive evaluation of cultural competency.
Figures & tables
Figure 1: CultureConverse pipeline: a seed (persona × template) is expanded into a blueprint grounded in cultural and taboo KBs, then decomposed into turn-by-turn shards . These guide a multi-turn simulation between a culturally situated user and the evaluated assistant. Finally, an LLM judge with full oracle access computes the evaluation (3H, TDQ). The assistant observes only the chat history; all taboos and success criteria remain hidden.
Figure 2: Formal generative sequence of a CultureConverse episode. A scenario seed ( s ) is augmented with retrieved knowledge to form a blueprint ( bs ), which partitions public, private, and oracle context. This blueprint is decomposed into an ordered turn-level shard plan ( rs ) that provides context for the interactive simulation ( e ) to evaluate a target assistant ( A ).
Role
Public context
Private state ( Ppriv )
Shard state ( qi )
Oracle state ( Poracle )
Eval assistant A
chat history
–
–
–
Gold teacher A
chat history
user demog. & identity
target user goal
criteria & taboos
Simulator U
chat history + scenario
own identity & style
active goal & directives
–
Judge J
full transcript
user demog. & identity
turn-by-turn goals
rubrics & taboos
Table 1: Role-specific visibility during simulation and evaluation.
Figure 3: Two truncated and scored CultureConverse episodes. The assistant A sees only the chat history from U ; the judge J additionally receives oracle access to success/failure criteria, taboos, and tripwires.
Partition
Detail
N
Standard
CTX
NORM
By Split and Language
Test (eval)
English
7,305
3,659
1,827
1,819
Test (eval)
Native
7,305
3,659
1,827
1,819
Train (gold)
English
137,144
68,722
34,258
34,164
Train (gold)
Native
137,151
68,726
34,259
34,166
By Region
Table 2: Dataset composition for CultureConverse -DS. CTX and NORM indicate scenarios with contextual and normative friction traps, respectively.
Comparison
% ± 1
MAE
Bias
Human–human (mean pairwise)
87.5
0.727
–
GPT-5 mini vs human consensus
90.1
0.581
-0.126
Table 3: Human agreement results on the annotation set. Bias is the judge minus human scores; negative values indicate the judge is stricter than humans.
3H components
3H
TDQ components
TDQ
Model
Help
Hon
Harm
Avg
Nat
Plaus
Typ
Avg
Clean %
GPT-5 mini ( 2025e )
3.80
4.60
4.45
4.28
4.07
3.72
3.65
3.81
87.7
GPT-5.4 ( 2026d )
3.58
4.28
4.47
4.11
3.95
3.78
3.67
3.80
91.0
GPT-5.4 mini ( 2026e )
3.39
4.32
4.33
4.01
3.95
3.72
3.66
3.78
89.4
Qwen 3.6 35B A3B ( 2026f )
3.69
3.54
4.24
3.82
3.99
3.17
3.58
3.58
85.6
Gemini 2.5 Flash ( 2025 )
3.38
3.94
4.08
3.80
3.98
3.48
3.56
3.67
83.8
Table 4: Aggregate results across 14,610 episodes. Rows are grouped by the same model families used in Figure 4 and sorted within each family by 3H. Full CIs for all metrics and models appear in Appendix O , Table 17 .
Figure 4: Mean Assistance Quality (3H) versus Training Data Quality (TDQ) across 14,610 episodes per evaluated model.
Table 5: Pre/post-training transfer for two 8B models, fine-tuned on the 27,860 filtered CultureConverse -DS subset (per-dataset breakdown in Table 18 ).
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Focus
# Reg.
# Lang.
Scale
Form
P
MT
A
H
T
SeaEval ( 2024a )
SEA eval
–
8
13.3K samples; 29 datasets
static eval
∘
∘
∘
∘
∘
SEA-HELM ( 2025b )
SEA suite
5
5
five-pillar evaluation suite
task suite
∘
△
∘
∘
∘
LoraxBench ( 2025 )
ID langs
1
20
84.9K datapoints; 6 tasks
QA/NLI/MT
∘
∘
∘
∘
∘
IndoMMLU ( 2023 )
Indonesia
1
1+9
14,981 exam questions; 64 tasks
exams
△
∘
∘
∘
∘
VMLU ( 2025 )
Vietnam
1
1
four Vietnamese LLM benchmarks
multitask
∘
∘
∘
∘
∘
MultiNativQA ( 2025 )
native QA
9
7
∼ 64K manually annotated QA pairs; 18 topics
QA
△
∘
∘
∘
∙
Appendix
Table 6: Feature comparison for culturally grounded LLM benchmarks. P = persona/subgroup metadata; MT = multi-turn; A = assistant-user help-seeking; H = hidden/private/oracle state; T = released training or tuning data. G denotes global or broad multi-region coverage; M denotes multiple languages.
Region group
Ids ; personas
Native-languages
Identity labels mapped to that language
Global controls: every identity is crossed with age cohorts 18–34 / 35–54 / 55+ and gender labels Female / Male; therefore 6 personas per identity.
BN Brunei
3 ; 18
Malay
Brunei Malay Muslim
Mandarin
Brunei Chinese Buddhist; Brunei Chinese Christian
CN China
6 ; 36
Mandarin
Han secular urban; Han traditional rural; Hui Muslim; Han Christian Protestant; Tibetan/Mongolian Buddhist; Uyghur Muslim
ID Indonesia
9 ; 54
Indonesian
Javanese Muslim Abangan; Javanese Muslim Santri; Sundanese Muslim; Chinese Indonesian Christian; Chinese Indonesian Buddhist/Confucian; Batak Christian; Balinese Hindu; Eastern Indonesia Christian; Minangkabau Muslim
JP Japan
5 ; 30
Japanese
Yamato Shinto; Yamato Buddhist; Ryukyuan/Okinawan; Ainu Indigenous; Zainichi Korean
Appendix
Table 7: Persona identity universe for the released ten-region benchmark, grouped by region and native-language routing.
Figure 5: Exact scenario specifications for the culinary domain and its 6 corresponding subdomains.
Figure 6: Exact scenario specifications for interpersonal dynamics and its 6 corresponding subdomains.
Figure 7: Exact scenario specifications for ritual, belief, and temporal observance and its 6 corresponding subdomains.
Figure 8: Exact scenario specifications for economic transaction and professional conduct and its 6 corresponding subdomains.
Figure 9: Exact scenario specifications for domestic living and community cohesion and its 6 corresponding subdomains.
Figure 10: Exact scenario specifications for lifestyle, mobility, and aesthetics and its 6 corresponding subdomains.
Figure 11: Exact scenario specifications for somatic health and traditional wellness and its 6 corresponding subdomains.
Taboo KB : 962 curated structured entries, embedded as a separate FAISS index.
BN
Brunei
1,380
52
CN
China
43,128
126
ID
Indonesia
15,167
76
JP
Japan
56,898
136
Appendix
Table 8: Breakdown of retrieval sources used by Blueprint generation.
Pipeline component
Model / stack
Role in CultureConverse
Notes on visibility and split
Template generation and validation
GPT-5.2
Generates and validates abstract, culture-agnostic scenario templates prior to persona binding.
Yields reusable scenario structures; executes independently of any persona, oracle, or conversational context.
Blueprint generation and validation
GPT-5 mini
Expands seeds using retrieved KB evidence to generate the public scenario, hidden user context, and evaluator-oracle criteria.
Applied across both train and test splits; automated validation verifies answerability.
Shard generation and validation
GPT-5 mini
Decomposes validated blueprints into 3–5 turn-level conversational phases, governed by strict transition logic.
Produces private simulator directives; the evaluated assistant observes only the resulting surface-level user utterances.
KB retrieval
Qwen3-Embedding-0.6B + FAISS ( 2025 )
Executes dense retrieval against the cultural and taboo knowledge bases during blueprint construction.
Retrieval metadata is preserved for provenance; the raw retrieval process is strictly hidden from the target assistant.
User simulator
GPT-5 mini
Acts as the simulated user ( U ), managing the conversational state (advance, loop, or fail) based on shard criteria.
Held constant across both blind evaluation and gold-mode generation to support a controlled comparison of assistants.
Gold-mode training assistant
GPT-5 mini
Operates as the reference assistant ( A ) during the synthesis of training trajectories, guided by full oracle context.
Utilised exclusively to construct the CultureConverse -DS training corpus.
Appendix
Table 9: Model deployment configuration and provenance for the CultureConverse pipeline. The matrix distinguishes between artefact-generation components, retrieval infrastructure, and target assistants.
Figure 17: Exact prompt fragments injected by the two conversational diversity controls. Bracketed variables are populated via deterministic, seed-based selection; mechanism (a) is appended prior to every simulated user turn for t≥1 , whereas mechanism (b) explicitly constrains only the initial user utterance at t=0 .
Figure 18: The complete pool of 40 response-style directives. A single entry is deterministically selected and injected into the prompt context prior to every simulated user ( U ) turn for t≥1 .
Figure 19: The complete pool of 48 opener-style archetypes utilised during episode initialisation. Two distinct styles are deterministically sampled via the episode seed and blended to construct a highly varied initial user utterance.
Lang
Eval 3H
Gold 3H
Δ 3H
Eval TDQ
Gold TDQ
Δ Clean
English
4.26
4.35
+0.08
3.79
3.76
+5.3 pp
Native
4.31
4.48
+0.18
3.83
3.82
+5.6 pp
Combined
4.28
4.41
+0.13
3.81
3.79
+5.5 pp
Appendix
Table 10: Comparison of Gold-mode (oracle-guided) versus blind generation using GPT-5 mini on the matched test split. Positive Δ values indicate improvements under Gold mode, demonstrating enhanced assistance quality and safety with negligible impact on scenario realism (TDQ).
Judge / ensemble
% ± 1
MAE
Bias
GPT-5 mini
90.1
0.581
-0.126
Qwen 3.6 Flash
83.0
0.682
-0.025
Llama 4 Maverick
81.1
0.703
-0.224
Gemini 3 Flash Preview
75.4
0.813
-0.245
Gemini 2.5 Flash-Lite
73.3
0.837
-0.389
DeepSeek V3.1
72.6
0.838
-0.326
Appendix
Table 11: Judge-panel sweep on the matched human-annotated subset. We compare 11 individual judges and all 165 three-judge ensembles, ranked by MAE. GPT-5 mini is the strongest individual judge; the best ensemble reduces MAE slightly but lowers within-one agreement and provides no meaningful bias improvement.
Metric
HH % ± 1
HH MAE
GPT-5 mini % ± 1
GPT-5 mini MAE
Bias
Helpfulness
90.0
0.674
89.2
0.647
-0.325
Honesty
89.7
0.691
94.6
0.568
+0.074
Harmlessness
90.6
0.597
90.5
0.572
+0.168
Naturalness
90.1
0.689
97.6
0.417
-0.034
Plausibility
83.4
0.829
82.6
0.651
-0.347
Typicality
81.1
0.882
86.1
0.628
-0.289
Appendix
Table 12: Agreement by evaluation metric. HH denotes mean human–human agreement across annotator pairs; GPT-5 mini columns compare the judge directly against the human consensus.
Region
HH % ± 1
HH MAE
GPT-5 mini % ± 1
GPT-5 mini MAE
Bias
BN
91.3
0.686
90.2
0.624
-0.034
CN
81.5
0.856
88.8
0.618
-0.296
ID
93.0
0.720
93.2
0.530
-0.087
JP
89.7
0.630
91.3
0.542
+0.089
KR
91.2
0.607
90.1
0.557
-0.294
MY
86.9
0.775
89.4
0.562
+0.038
Appendix
Table 13: Agreement by region. Across regional slices, GPT-5 mini remains close to the human–human bandwidth, with signed bias reported as judge score minus human consensus.
Figure 26: Complete instructions provided to the human annotation panel.
Figure 27: CultureConverse examples focused on household reciprocity and food hosting.
Figure 28: CultureConverse examples centred on familiar wedding and partner-family introduction tasks.
Figure 30: Additional native-language examples showing household etiquette and food-allergy handling in everyday family/community settings.
Model
Std 3H
Trap 3H
Δ 3H
Claude Haiku 4.5
3.55
3.47
-0.08
DeepSeek V3.1
3.66
3.59
-0.07
Nemotron 3 Super 120B A12B
3.67
3.60
-0.07
DeepSeek V3.2
3.77
3.70
-0.07
Qwen 3.6 35B A3B
3.86
3.79
-0.07
Gemini 3 Flash Preview
3.60
3.53
-0.07
Appendix
Table 14: Performance degradation on trap episodes. Negative Δ values indicate that normative and contextual friction consistently increase task difficulty.
Metric
Mean
Std
Median
%1
%2
%3
%4
%5
Helpfulness
3.80
0.58
4.00
0.0
1.1
25.3
65.7
7.8
Honesty
4.60
0.53
5.00
0.0
0.3
1.3
36.5
61.9
Harmlessness
4.45
0.97
5.00
2.1
6.6
2.1
22.6
66.6
Naturalness
4.07
0.26
4.00
0.0
0.0
0.2
92.7
7.1
Scenario plausibility
3.72
0.54
4.00
0.0
4.5
19.1
76.4
0.0
Cultural typicality
3.65
0.59
4.00
0.0
5.9
23.6
70.3
0.2
Appendix
Table 15: Full representative GPT-5 mini judge-score distribution across the 14,610 evaluated episodes.
Model
LR
LoRA r
LoRA α
Llama-3.1-8B-IT ( 2024 )
5×10−6
32
64
SEA-LION-v4-8B-IT ( 2026 )
1×10−6
8
16
Appendix
Table 16: Training configuration for the RQ3 fine-tuning runs. Both models were trained on 27,860 perfect-3H CultureConverse -DS samples (effective batch size 256, bf16) using LLaMA-Factory and PEFT.
Assistance quality
Training-data quality
Model
3H
Help
Hon
Harm
TDQ
Nat
Plaus
Typ
Clean %
GPT-5 mini
4.28 [4.28,4.29]
3.80 [3.79,3.81]
4.60 [4.59,4.61]
4.45 [4.43,4.47]
3.81 [3.81,3.82]
4.07 [4.06,4.07]
3.72 [3.71,3.73]
3.65 [3.64,3.66]
87.7 [87.1,88.2]
GPT-5.4
4.11 [4.10,4.12]
3.58 [3.57,3.59]
4.28 [4.28,4.29]
4.47 [4.45,4.48]
3.80 [3.80,3.81]
3.95 [3.95,3.95]
3.78 [3.78,3.79]
3.67 [3.66,3.68]
91.0 [90.5,91.4]
GPT-5.4 mini
4.01 [4.01,4.02]
3.39 [3.38,3.40]
4.32 [4.31,4.33]
4.33 [4.32,4.35]
3.78 [3.77,3.78]
3.95 [3.95,3.96]
3.72 [3.71,3.73]
3.66 [3.65,3.67]
89.4 [89.0,90.0]
Qwen 3.6 35B A3B
3.82 [3.82,3.83]
3.69 [3.68,3.70]
3.54 [3.53,3.55]
4.24 [4.22,4.26]
3.58 [3.58,3.59]
3.99 [3.99,4.00]
3.17 [3.16,3.18]
3.58 [3.57,3.59]
85.6 [85.0,86.2]
Gemini 2.5 Flash
3.80 [3.79,3.81]
3.38 [3.37,3.39]
3.94 [3.93,3.95]
4.08 [4.06,4.09]
3.67 [3.67,3.68]
3.98 [3.98,3.99]
3.48 [3.47,3.49]
3.56 [3.55,3.57]
83.8 [83.2,84.4]
Appendix
Table 17: Full metric means with 95% non-parametric bootstrap confidence intervals (1,000 episode-level resamples) for the reported assistant models. Bracketed values are confidence intervals; yellow cells mark the highest mean per metric, and blue/teal shading marks the two aggregate metric families.
Figure 33: Model-level component and family-average score heatmap. Each cell represents an assistant-level mean across all evaluated episodes.
Figure 34: Model-level language deltas (native minus English). Positive values indicate superior performance in native-language dialogues.
Figure 35: Region-by-model 3H language deltas (native minus English).
Figure 36: Age and gender 3H slice heatmaps. Performance variance across these broad demographic slices remains negligible ( ≤0.05 points).
Figure 37: Regional 3H disparities for GPT-5 mini. Bars indicate deviation from the model’s global average.
Figure 38: Region–identity 3H disparities for GPT-5 mini.
Figure 39: Identity-level 3H means for GPT-5 mini. The dashed line represents the unweighted mean across all retained identities.
Figure 40: Cross-model disparity heatmap. Each cell reports the best-minus-worst subgroup 3H mean. Region–identity is consistently the widest axis, while age and gender variance approach zero.
Figure 41: Judge score distribution from GPT-5 mini, split by English and native-language slots.
Llama-3.1-8B-IT ( 2024 )
SEA-LION-v4-8B-IT ( 2026 )
Dataset
Region
Base
SFT
Δ
Base
SFT
Δ
MCQ — Accuracy (%)
IndoMMLU
ID
53.91
54.08
+0.17
47.85
47.97
+0.12
KorNAT_common
KR
75.65
76.45
+0.80
69.10
69.25
+0.15
LoraxBench
ID
56.34
57.39
+1.05
48.82
48.89
+0.07
SOBACO
JP
33.33
34.00
+0.67
27.83
28.08
+0.25
Appendix
Table 18: Per-dataset OOD transfer after LoRA SFT on the 27,860-sample perfect-3H CultureConverse -DS subset. Metrics are accuracy for MCQ tasks and macro-F1 for CLS tasks; Δ is measured in percentage points relative to each base model.
Region
Dataset
Language
Fmt.
Eval n
Source scale / label focus
What the dataset tests
Panel A: MCQ cultural knowledge, commonsense, social norms, and safety reasoning (accuracy; 7 datasets).
ID
IndoMMLU ( 2023 )
Indonesian
MCQ
4,146
14,906 cultural-understanding exam-style items
Indonesian school, language, and local-knowledge questions.
KR
KorNAT_common ( 2024 )
Korean
MCQ
2,000
4,000 social-value + 6,000 common-knowledge items
Korean common knowledge and social-value reasoning, evaluated through direct answer selection.
ID
LoraxBench ( 2025 )
20 Indonesian languages
MCQ
1,530
11,730 cultural-understanding items
Low-resource Indonesian-language cultural QA and regional knowledge transfer.
Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.
Culture is lived through conversation, yet existing Indonesian cultural commonsense benchmarks evaluate LLMs on short and isolated prompts, stripping away the dialogic context in which cultural nuances actually surface. We introduce CultureTalk-ID, the first dialogue-based benchmark for cultural commonsense in Indonesian and its local languages, comprising 4,496 culturally grounded dialogues across 11 languages and 13 culturally salient topics, curated through a multi-stage human pipeline with native speakers to ensure authenticity. CultureTalk-ID introduces three complementary tasks, namely dialogue-based multiple-choice cultural commonsense reasoning, culturally faithful machine translation, and language steering, which jointly probe whether LLMs can understand, transfer, and generate culturally grounded language.
Muhammad Dehan Al Kautsar, Salsabila Pranida, Bilal Elbouardi +1
Mohamed bin Zayed University of Artificial Intelligence
Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that language. We call this the Illusion of Cultural Alignment. To test this assumption directly, we introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers. Unlike translated benchmarks, MSQA targets locally grounded knowledge and reduces shortcuts from English-centric cross-lingual transfer. Evaluating 18 LLMs, we find substantial cultural degradation and a pronounced Locality Effect: cultural competence tracks pre-training exposure more closely than general reasoning ability. We further show that common inference-time remedies do not dissolve the illusion. Models remain overconfident on unfamiliar cultural questions, repeated sampling yields unstable rather than reliable correctness, and retrieval augmentation helps unevenly on long-tail facts. These findings indicate that cultural alignment cannot be inferred from multilingual ability alone and requires deeper intervention than calibration, sampling, or retrieval at inference time
Existing research largely reduces cultural intelligence in LLMs to a knowledge-level problem, overlooking whether models can effectively utilize their acquired knowledge in realistic scenarios. To bridge this gap, we introduce CultureForest, a benchmark for Cultural Norm Grounded Reasoning. Each question is grounded in a small set of atomic norms, enabling verifiable and attributable evaluation. CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions, and supports a progressive evaluation from multiple-choice to open-ended generation. Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings, accompanied by pronounced cross-region disparities. Through targeted analysis, we uncover several consistent patterns: (1) test-time reasoning yields limited gains and may exacerbate inequity; (2) models exhibit highly shared regional preference structures; (3) model responses are markedly conservative, especially under stricter cultural constraints; and (4) by disentangling cultural knowledge acquisition from cultural reasoning, we show that while LLMs possess substantial cultural knowledge, their performance is further bottlenecked by its effective use. These findings point to a necessary shift from knowledge-centric evaluation toward measuring knowledge-grounded reasoning.
Yangfan Ye, Xiaocheng Feng, Jialong Tang +5
Harbin Institute of Technology · Harvard University · The University of Hong Kong