CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia

Bryan Chen Zhengyu Tan; Weihua Zheng; Thong T. Doan; Bich Ngoc Doan; Jia Wang Peh; Xiaoyuan Yi; Jing Yao; Xing Xie

cs.CLAug 28, 2026

CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia

Authors: Bryan Chen Zhengyu Tan, Weihua Zheng, Thong T. Doan, Bich Ngoc Doan, Jia Wang Peh, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Zhengyuan Liu, JinYeong Bak, Wafi Shamdi, Soo Kai Chie, Liew Yu Siong, Aina Azyyati Binti Mohamad Rezal, Lew Yan Yan Vanessa, Huadan Wu, Dylan Raharja, Nadya Yuki Wangsajaya, Akane Fukushige, Kazushi Kato, Koji Inoue, Tatsuya Kawahara, Jaehyung Seo, Dongjun Kim, Seungyoon Lee, Zi Haur Pang, Rui Yang Tan, Charibeth Ko Cheng, Maria Regina Justina Estuar, Jann Railey Montalan, Pham Minh Duc, Roy Ka-Wei Lee

Organizations: Singapore University of Technology and Design (SUTD), Singapore · Agency for Science, Technology and Research (A*STAR), Singapore · École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · Microsoft Research Asia (MSRA), Beijing, China · Sungkyunkwan University (SKKU), Republic of Korea · Universiti Brunei Darussalam (UBD), Brunei Darussalam · China University of Petroleum (East China), Qingdao, China · Nanyang Technological University (NTU), Singapore · Kyoto University, Kyoto, Japan · Konkuk University, Seoul, Republic of Korea · Upstage AI, Republic of Korea · Korea University, Seoul, Republic of Korea · De La Salle University (DLSU), Manila, Philippines · Ateneo de Manila University (ADMU), Quezon City, Philippines · AI Singapore (AISG), Singapore · University of British Columbia (UBC), Vancouver, Canada

Abstract

Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking practical help over multiple turns in culturally grounded scenarios. We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue that covers 10 East and Southeast Asian regions, 58 subgroup identities, and 7 domains. Each simulated and evaluated episode produces a scored interaction where the assistant assists the user and infers cultural constraints from partial information. The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues. In our benchmark evaluation of 18 models, GPT-5 mini achieves the highest assistance quality. Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment. Performance gains from fine-tuning on 27,860 high-quality CultureConverse-DS samples improve in-domain assistance and transfer out-of-domain to cultural MCQ and safety classification benchmarks. We release the harness, both splits, and judge prompts to support interactive evaluation of cultural competency.

Figures & tables

Figure 1: CultureConverse pipeline: a seed (persona $\times$ template) is expanded into a blueprint grounded in cultural and taboo KBs, then decomposed into turn-by-turn shards . These guide a multi-turn simulation between a culturally situated user and the evaluated assistant. Finally, an LLM judge with full oracle access computes the evaluation (3H, TDQ). The assistant observes only the chat history; all taboos and success criteria remain hidden.

Panel of Figure 2 — Figure 2: Formal generative sequence of a CultureConverse episode. A scenario seed ( $s$ ) is augmented with retrieved knowledge to form a blueprint ( $b_{s}$ ), which partitions public, private, and oracle context. This blueprint is decomposed into an ordered turn-level shard plan ( $r_{s}$ ) that provides context for the interactive simulation ( $e$ ) to evaluate a target assistant ( $A$ ).

Role	Public context	Private state ( $P_{\mathrm{priv}}$ )	Shard state ( $q_{i}$ )	Oracle state ( $P_{\mathrm{oracle}}$ )
Eval assistant $\boldsymbol{\mathcal{A}}$	chat history	–	–	–
Gold teacher $\boldsymbol{\mathcal{A}}$	chat history	user demog. & identity	target user goal	criteria & taboos
Simulator $\boldsymbol{\mathcal{U}}$	chat history + scenario	own identity & style	active goal & directives	–
Judge $\boldsymbol{\mathcal{J}}$	full transcript	user demog. & identity	turn-by-turn goals	rubrics & taboos

Table 1: Role-specific visibility during simulation and evaluation.

Panel of Figure 3 — Figure 3: Two truncated and scored CultureConverse episodes. The assistant $\boldsymbol{\mathcal{A}}$ sees only the chat history from $\boldsymbol{\mathcal{U}}$ ; the judge $\boldsymbol{\mathcal{J}}$ additionally receives oracle access to success/failure criteria, taboos, and tripwires.

Partition	Detail	$N$	Standard	CTX	NORM
By Split and Language
Test (eval)	English	7,305	3,659	1,827	1,819
Test (eval)	Native	7,305	3,659	1,827	1,819
Train (gold)	English	137,144	68,722	34,258	34,164
Train (gold)	Native	137,151	68,726	34,259	34,166
By Region

Table 2: Dataset composition for CultureConverse -DS. CTX and NORM indicate scenarios with contextual and normative friction traps, respectively.

Comparison	% $\pm$ 1	MAE	Bias
Human–human (mean pairwise)	87.5	0.727	–
GPT-5 mini vs human consensus	90.1	0.581	-0.126

Table 3: Human agreement results on the annotation set. Bias is the judge minus human scores; negative values indicate the judge is stricter than humans.

	3H components			3H	TDQ components			TDQ
Model	Help	Hon	Harm	Avg	Nat	Plaus	Typ	Avg	Clean %
GPT-5 mini ( 2025e )	3.80	4.60	4.45	4.28	4.07	3.72	3.65	3.81	87.7
GPT-5.4 ( 2026d )	3.58	4.28	4.47	4.11	3.95	3.78	3.67	3.80	91.0
GPT-5.4 mini ( 2026e )	3.39	4.32	4.33	4.01	3.95	3.72	3.66	3.78	89.4
Qwen 3.6 35B A3B ( 2026f )	3.69	3.54	4.24	3.82	3.99	3.17	3.58	3.58	85.6
Gemini 2.5 Flash ( 2025 )	3.38	3.94	4.08	3.80	3.98	3.48	3.56	3.67	83.8

Table 4: Aggregate results across 14,610 episodes. Rows are grouped by the same model families used in Figure 4 and sorted within each family by 3H. Full CIs for all metrics and models appear in Appendix O , Table 17 .

Figure 4: Mean Assistance Quality (3H) versus Training Data Quality (TDQ) across 14,610 episodes per evaluated model.

	Llama-3.1-8B-IT ( 2024 )			SEA-LION-v4-8B-IT ( 2026 )
Task / Metric	Base	SFT	$\Delta$	Base	SFT	$\Delta$
In-domain assistance quality on CultureConverse (3H, 1–5 judge scale; 27,860 high-quality CultureConverse -DS samples)
English — 3H average	3.03	3.10	+0.07	2.83	2.88	+0.05
Helpfulness	2.70	3.06	+0.36	2.28	2.23	$-$ 0.05
Honesty	3.16	3.05	$-$ 0.11	3.01	3.10	+0.09
Harmlessness	3.23	3.19	$-$ 0.04	3.19	3.30	+0.11

Table 5: Pre/post-training transfer for two 8B models, fine-tuned on the 27,860 filtered CultureConverse -DS subset (per-dataset breakdown in Table 18 ).

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Focus	# Reg.	# Lang.	Scale	Form	P	MT	A	H	T
SeaEval ( 2024a )	SEA eval	–	8	13.3K samples; 29 datasets	static eval	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$
SEA-HELM ( 2025b )	SEA suite	5	5	five-pillar evaluation suite	task suite	$\circ$	$\triangle$	$\circ$	$\circ$	$\circ$
LoraxBench ( 2025 )	ID langs	1	20	84.9K datapoints; 6 tasks	QA/NLI/MT	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$
IndoMMLU ( 2023 )	Indonesia	1	1+9	14,981 exam questions; 64 tasks	exams	$\triangle$	$\circ$	$\circ$	$\circ$	$\circ$
VMLU ( 2025 )	Vietnam	1	1	four Vietnamese LLM benchmarks	multitask	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$
MultiNativQA ( 2025 )	native QA	9	7	$\sim$ 64K manually annotated QA pairs; 18 topics	QA	$\triangle$	$\circ$	$\circ$	$\circ$	$\bullet$

Appendix

Table 6: Feature comparison for culturally grounded LLM benchmarks. P = persona/subgroup metadata; MT = multi-turn; A = assistant-user help-seeking; H = hidden/private/oracle state; T = released training or tuning data. G denotes global or broad multi-region coverage; M denotes multiple languages.

Region group	Ids ; personas	Native-languages	Identity labels mapped to that language
Global controls: every identity is crossed with age cohorts 18–34 / 35–54 / 55+ and gender labels Female / Male; therefore 6 personas per identity.
BN Brunei	3 ; 18	Malay	Brunei Malay Muslim
		Mandarin	Brunei Chinese Buddhist; Brunei Chinese Christian
CN China	6 ; 36	Mandarin	Han secular urban; Han traditional rural; Hui Muslim; Han Christian Protestant; Tibetan/Mongolian Buddhist; Uyghur Muslim
ID Indonesia	9 ; 54	Indonesian	Javanese Muslim Abangan; Javanese Muslim Santri; Sundanese Muslim; Chinese Indonesian Christian; Chinese Indonesian Buddhist/Confucian; Batak Christian; Balinese Hindu; Eastern Indonesia Christian; Minangkabau Muslim
JP Japan	5 ; 30	Japanese	Yamato Shinto; Yamato Buddhist; Ryukyuan/Okinawan; Ainu Indigenous; Zainichi Korean

Appendix

Table 7: Persona identity universe for the released ten-region benchmark, grouped by region and native-language routing.

Panel of Figure 5 — Figure 5: Exact scenario specifications for the culinary domain and its 6 corresponding subdomains.

Panel of Figure 6 — Figure 6: Exact scenario specifications for interpersonal dynamics and its 6 corresponding subdomains.

Panel of Figure 7 — Figure 7: Exact scenario specifications for ritual, belief, and temporal observance and its 6 corresponding subdomains.

Panel of Figure 8 — Figure 8: Exact scenario specifications for economic transaction and professional conduct and its 6 corresponding subdomains.

Panel of Figure 9 — Figure 9: Exact scenario specifications for domestic living and community cohesion and its 6 corresponding subdomains.

Panel of Figure 10 — Figure 10: Exact scenario specifications for lifestyle, mobility, and aesthetics and its 6 corresponding subdomains.

Panel of Figure 11 — Figure 11: Exact scenario specifications for somatic health and traditional wellness and its 6 corresponding subdomains.

Set	Region	Wiki tag counts	Taboo entries
Cultural KB : 181,120 PetScan rows; 180,992 unique indexed entries; Qwen3-Embedding-0.6B + FAISS.
Taboo KB : 962 curated structured entries, embedded as a separate FAISS index.
BN	Brunei	1,380	52
CN	China	43,128	126
ID	Indonesia	15,167	76
JP	Japan	56,898	136

Appendix

Table 8: Breakdown of retrieval sources used by Blueprint generation.

Pipeline component	Model / stack	Role in CultureConverse	Notes on visibility and split
Template generation and validation	GPT-5.2	Generates and validates abstract, culture-agnostic scenario templates prior to persona binding.	Yields reusable scenario structures; executes independently of any persona, oracle, or conversational context.
Blueprint generation and validation	GPT-5 mini	Expands seeds using retrieved KB evidence to generate the public scenario, hidden user context, and evaluator-oracle criteria.	Applied across both train and test splits; automated validation verifies answerability.
Shard generation and validation	GPT-5 mini	Decomposes validated blueprints into 3–5 turn-level conversational phases, governed by strict transition logic.	Produces private simulator directives; the evaluated assistant observes only the resulting surface-level user utterances.
KB retrieval	Qwen3-Embedding-0.6B + FAISS ( 2025 )	Executes dense retrieval against the cultural and taboo knowledge bases during blueprint construction.	Retrieval metadata is preserved for provenance; the raw retrieval process is strictly hidden from the target assistant.
User simulator	GPT-5 mini	Acts as the simulated user ( $\boldsymbol{\mathcal{U}}$ ), managing the conversational state (advance, loop, or fail) based on shard criteria.	Held constant across both blind evaluation and gold-mode generation to support a controlled comparison of assistants.
Gold-mode training assistant	GPT-5 mini	Operates as the reference assistant ( $\boldsymbol{\mathcal{A}}$ ) during the synthesis of training trajectories, guided by full oracle context.	Utilised exclusively to construct the CultureConverse -DS training corpus.

Appendix

Table 9: Model deployment configuration and provenance for the CultureConverse pipeline. The matrix distinguishes between artefact-generation components, retrieval infrastructure, and target assistants.

Panel of Figure 17 — Figure 17: Exact prompt fragments injected by the two conversational diversity controls. Bracketed variables are populated via deterministic, seed-based selection; mechanism (a) is appended prior to every simulated user turn for $t\geq 1$ , whereas mechanism (b) explicitly constrains only the initial user utterance at $t=0$ .

Panel of Figure 18 — Figure 18: The complete pool of 40 response-style directives. A single entry is deterministically selected and injected into the prompt context prior to every simulated user ( $\boldsymbol{\mathcal{U}}$ ) turn for $t\geq 1$ .

Panel of Figure 19 — Figure 19: The complete pool of 48 opener-style archetypes utilised during episode initialisation. Two distinct styles are deterministically sampled via the episode seed and blended to construct a highly varied initial user utterance.

Lang	Eval 3H	Gold 3H	$\Delta$ 3H	Eval TDQ	Gold TDQ	$\Delta$ Clean
English	4.26	4.35	+0.08	3.79	3.76	+5.3 pp
Native	4.31	4.48	+0.18	3.83	3.82	+5.6 pp
Combined	4.28	4.41	+0.13	3.81	3.79	+5.5 pp

Appendix

Table 10: Comparison of Gold-mode (oracle-guided) versus blind generation using GPT-5 mini on the matched test split. Positive

\Delta

values indicate improvements under Gold mode, demonstrating enhanced assistance quality and safety with negligible impact on scenario realism (TDQ).

Judge / ensemble	% $\pm$ 1	MAE	Bias
GPT-5 mini	90.1	0.581	-0.126
Qwen 3.6 Flash	83.0	0.682	-0.025
Llama 4 Maverick	81.1	0.703	-0.224
Gemini 3 Flash Preview	75.4	0.813	-0.245
Gemini 2.5 Flash-Lite	73.3	0.837	-0.389
DeepSeek V3.1	72.6	0.838	-0.326

Appendix

Table 11: Judge-panel sweep on the matched human-annotated subset. We compare 11 individual judges and all 165 three-judge ensembles, ranked by MAE. GPT-5 mini is the strongest individual judge; the best ensemble reduces MAE slightly but lowers within-one agreement and provides no meaningful bias improvement.

Metric	HH % $\pm$ 1	HH MAE	GPT-5 mini % $\pm$ 1	GPT-5 mini MAE	Bias
Helpfulness	90.0	0.674	89.2	0.647	-0.325
Honesty	89.7	0.691	94.6	0.568	+0.074
Harmlessness	90.6	0.597	90.5	0.572	+0.168
Naturalness	90.1	0.689	97.6	0.417	-0.034
Plausibility	83.4	0.829	82.6	0.651	-0.347
Typicality	81.1	0.882	86.1	0.628	-0.289

Appendix

Table 12: Agreement by evaluation metric. HH denotes mean human–human agreement across annotator pairs; GPT-5 mini columns compare the judge directly against the human consensus.

Region	HH % $\pm$ 1	HH MAE	GPT-5 mini % $\pm$ 1	GPT-5 mini MAE	Bias
BN	91.3	0.686	90.2	0.624	-0.034
CN	81.5	0.856	88.8	0.618	-0.296
ID	93.0	0.720	93.2	0.530	-0.087
JP	89.7	0.630	91.3	0.542	+0.089
KR	91.2	0.607	90.1	0.557	-0.294
MY	86.9	0.775	89.4	0.562	+0.038

Appendix

Table 13: Agreement by region. Across regional slices, GPT-5 mini remains close to the human–human bandwidth, with signed bias reported as judge score minus human consensus.

Panel of Figure 26 — Figure 26: Complete instructions provided to the human annotation panel.

Panel of Figure 27 — Figure 27: CultureConverse examples focused on household reciprocity and food hosting.

Panel of Figure 28 — Figure 28: CultureConverse examples centred on familiar wedding and partner-family introduction tasks.

Panel of Figure 29 — Figure 29: Additional native-language examples involving everyday gift-giving and wedding reciprocity.

Panel of Figure 30 — Figure 30: Additional native-language examples showing household etiquette and food-allergy handling in everyday family/community settings.

Model	Std 3H	Trap 3H	$\Delta$ 3H
Claude Haiku 4.5	3.55	3.47	-0.08
DeepSeek V3.1	3.66	3.59	-0.07
Nemotron 3 Super 120B A12B	3.67	3.60	-0.07
DeepSeek V3.2	3.77	3.70	-0.07
Qwen 3.6 35B A3B	3.86	3.79	-0.07
Gemini 3 Flash Preview	3.60	3.53	-0.07

Appendix

Table 14: Performance degradation on trap episodes. Negative

\Delta

values indicate that normative and contextual friction consistently increase task difficulty.

Metric	Mean	Std	Median	%1	%2	%3	%4	%5
Helpfulness	3.80	0.58	4.00	0.0	1.1	25.3	65.7	7.8
Honesty	4.60	0.53	5.00	0.0	0.3	1.3	36.5	61.9
Harmlessness	4.45	0.97	5.00	2.1	6.6	2.1	22.6	66.6
Naturalness	4.07	0.26	4.00	0.0	0.0	0.2	92.7	7.1
Scenario plausibility	3.72	0.54	4.00	0.0	4.5	19.1	76.4	0.0
Cultural typicality	3.65	0.59	4.00	0.0	5.9	23.6	70.3	0.2

Appendix

Table 15: Full representative GPT-5 mini judge-score distribution across the 14,610 evaluated episodes.

Model	LR	LoRA $r$	LoRA $\alpha$
Llama-3.1-8B-IT ( 2024 )	$5\times 10^{-6}$	32	64
SEA-LION-v4-8B-IT ( 2026 )	$1\times 10^{-6}$	8	16

Appendix

Table 16: Training configuration for the RQ3 fine-tuning runs. Both models were trained on 27,860 perfect-3H CultureConverse -DS samples (effective batch size 256, bf16) using LLaMA-Factory and PEFT.

	Assistance quality				Training-data quality
Model	3H	Help	Hon	Harm	TDQ	Nat	Plaus	Typ	Clean %
GPT-5 mini	4.28 [4.28,4.29]	3.80 [3.79,3.81]	4.60 [4.59,4.61]	4.45 [4.43,4.47]	3.81 [3.81,3.82]	4.07 [4.06,4.07]	3.72 [3.71,3.73]	3.65 [3.64,3.66]	87.7 [87.1,88.2]
GPT-5.4	4.11 [4.10,4.12]	3.58 [3.57,3.59]	4.28 [4.28,4.29]	4.47 [4.45,4.48]	3.80 [3.80,3.81]	3.95 [3.95,3.95]	3.78 [3.78,3.79]	3.67 [3.66,3.68]	91.0 [90.5,91.4]
GPT-5.4 mini	4.01 [4.01,4.02]	3.39 [3.38,3.40]	4.32 [4.31,4.33]	4.33 [4.32,4.35]	3.78 [3.77,3.78]	3.95 [3.95,3.96]	3.72 [3.71,3.73]	3.66 [3.65,3.67]	89.4 [89.0,90.0]
Qwen 3.6 35B A3B	3.82 [3.82,3.83]	3.69 [3.68,3.70]	3.54 [3.53,3.55]	4.24 [4.22,4.26]	3.58 [3.58,3.59]	3.99 [3.99,4.00]	3.17 [3.16,3.18]	3.58 [3.57,3.59]	85.6 [85.0,86.2]
Gemini 2.5 Flash	3.80 [3.79,3.81]	3.38 [3.37,3.39]	3.94 [3.93,3.95]	4.08 [4.06,4.09]	3.67 [3.67,3.68]	3.98 [3.98,3.99]	3.48 [3.47,3.49]	3.56 [3.55,3.57]	83.8 [83.2,84.4]

Appendix

Table 17: Full metric means with 95% non-parametric bootstrap confidence intervals (1,000 episode-level resamples) for the reported assistant models. Bracketed values are confidence intervals; yellow cells mark the highest mean per metric, and blue/teal shading marks the two aggregate metric families.

Figure 33: Model-level component and family-average score heatmap. Each cell represents an assistant-level mean across all evaluated episodes.

Figure 34: Model-level language deltas (native minus English). Positive values indicate superior performance in native-language dialogues.

Figure 35: Region-by-model 3H language deltas (native minus English).

Panel of Figure 36 — Figure 36: Age and gender 3H slice heatmaps. Performance variance across these broad demographic slices remains negligible ( $\leq 0.05$ points).

Figure 37: Regional 3H disparities for GPT-5 mini. Bars indicate deviation from the model’s global average.

Figure 38: Region–identity 3H disparities for GPT-5 mini.

Figure 39: Identity-level 3H means for GPT-5 mini. The dashed line represents the unweighted mean across all retained identities.

Figure 40: Cross-model disparity heatmap. Each cell reports the best-minus-worst subgroup 3H mean. Region–identity is consistently the widest axis, while age and gender variance approach zero.

Figure 41: Judge score distribution from GPT-5 mini, split by English and native-language slots.

		Llama-3.1-8B-IT ( 2024 )			SEA-LION-v4-8B-IT ( 2026 )
Dataset	Region	Base	SFT	$\Delta$	Base	SFT	$\Delta$
MCQ — Accuracy (%)
IndoMMLU	ID	53.91	54.08	+0.17	47.85	47.97	+0.12
KorNAT_common	KR	75.65	76.45	+0.80	69.10	69.25	+0.15
LoraxBench	ID	56.34	57.39	+1.05	48.82	48.89	+0.07
SOBACO	JP	33.33	34.00	+0.67	27.83	28.08	+0.25

Appendix

Table 18: Per-dataset OOD transfer after LoRA SFT on the 27,860-sample perfect-3H CultureConverse -DS subset. Metrics are accuracy for MCQ tasks and macro-F1 for CLS tasks;

\Delta

is measured in percentage points relative to each base model.

Region	Dataset	Language	Fmt.	Eval $n$	Source scale / label focus	What the dataset tests
Panel A: MCQ cultural knowledge, commonsense, social norms, and safety reasoning (accuracy; 7 datasets).
ID	IndoMMLU ( 2023 )	Indonesian	MCQ	4,146	14,906 cultural-understanding exam-style items	Indonesian school, language, and local-knowledge questions.
KR	KorNAT_common ( 2024 )	Korean	MCQ	2,000	4,000 social-value + 6,000 common-knowledge items	Korean common knowledge and social-value reasoning, evaluated through direct answer selection.
ID	LoraxBench ( 2025 )	20 Indonesian languages	MCQ	1,530	11,730 cultural-understanding items	Low-resource Indonesian-language cultural QA and regional knowledge transfer.
JP	SOBACO ( 2025 )	Japanese	MCQ	1,200	5,976 social-bias + 5,976 cultural-commonsense items	Japanese social-bias and cultural-commonsense decisions with norm-sensitive answer choices.
CN	SafetyBench ( 2024 )	Chinese / English	MCQ	1,500	1,828 hate-speech + 3,815 bias/ethics safety items	Safety knowledge for hate speech, bias, and ethics, separated from generative safety behaviour.

Appendix

Table 19: OOD evaluation suite composition. ’Eval

n

’ denotes the sample count utilised for evaluation.

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages
Jul 23, 2026Muhammad Dehan Al Kautsar, Salsabila Pranida, Bilal Elbouardi +1Multilingual Language Model Evaluation Multi-Turn Dialogue Evaluation
Mohamed bin Zayed University of Artificial Intelligence
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual Language Model Evaluation LLM Evaluation
ByteDance Seed · Nanjing University · M-A-P
CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs
Jun 1, 2026Yangfan Ye, Xiaocheng Feng, Jialong Tang +5LLM Evaluation Cultural Bias in Language Models
Harbin Institute of Technology · Harvard University · The University of Hong Kong

Jul 23, 2026J/K move · Enter open · S save

CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages

Muhammad Dehan Al Kautsar, Salsabila Pranida, Bilal Elbouardi, Fajri Koto

Mohamed bin Zayed University of Artificial Intelligence

Multilingual Language Model Evaluation Multi-Turn Dialogue Evaluation Cultural Knowledge in Language Models Machine Translation

Open paper AskPDF arXiv

Culture is lived through conversation, yet existing Indonesian cultural commonsense benchmarks evaluate LLMs on short and isolated prompts, stripping away the dialogic context in which cultural nuances actually surface. We introduce CultureTalk-ID, the first dialogue-based benchmark for cultural commonsense in Indonesian and its local languages, comprising 4,496 culturally grounded dialogues across 11 languages and 13 culturally salient topics, curated through a multi-stage human pipeline with native speakers to ensure authenticity. CultureTalk-ID introduces three complementary tasks, namely dialogue-based multiple-choice cultural commonsense reasoning, culturally faithful machine translation, and language steering, which jointly probe whether LLMs can understand, transfer, and generate culturally grounded language.

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages

MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs