LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations

Viet-Thanh Pham; Lizhen Qu; Thuy-Trang Vu; Gholamreza Haffari; Dinh Phung

cs.AIMar 2, 2026

LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations

Authors: Viet-Thanh Pham, Lizhen Qu, Thuy-Trang Vu, Gholamreza Haffari, Dinh Phung

Organizations: Department of Data Science & AI, Monash University

Abstract

Large language models (LLMs) are increasingly deployed as autonomous agents, yet evaluations focus primarily on task success rather than cultural appropriateness or evaluator reliability. We introduce LiveCultureBench, a multi-cultural, dynamic benchmark that embeds LLMs as agents in a simulated town and evaluates them on both task completion and adherence to socio-cultural norms. The simulation models a small city as a location graph with synthetic residents having diverse demographic and cultural profiles. Each episode assigns one resident a daily goal while others provide social context. An LLM-based verifier generates structured judgments on norm violations and task progress, which we aggregate into metrics capturing task-norm trade-offs and verifier uncertainty. Using LiveCultureBench across models and cultural profiles, we study (i) cross-cultural robustness of LLM agents, (ii) how they balance effectiveness against norm sensitivity, and (iii) when LLM-as-a-judge evaluation is reliable for automated benchmarking versus when human oversight is needed.

Figures & tables

Figure 1: Illustration of our proposed social simulation framework, LiveCultureBench . LLM-based agents are spawned in a dynamic town environment, and a dedicated Verifier Agent living outside of the simulation is used to evaluate the Target Agent’s performance and behaviors on task completion and cultural norm adherence.

Figure 2: Target Agent performance from different LLM backbones.

Figure 3: Analysis of performance of different LLMs when (i) interacting in multicultural scenarios, and (ii) interacting in different locations.

Metric	F1	Precision	Recall
Goal Completion	92.41	93.78	91.07
Norm Violation	89.36	87.92	90.86
Profile Faithfulness	90.88	92.14	89.65
Contextual Awareness	95.27	96.11	94.45
Coherence	96.08	97.34	94.97

Table 1: Average performance of the Verifier Agent on each evaluation task on our held-out test sets.

Figure 4: Conformal sampling results for different LLMs as our Verifier Agent.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Age bin	Share (%)
15-19	4.2
20-24	16.8
25-29	19.8
30-34	15.6
35-39	9.7
40-44	5.8

Appendix

Table 2: Age distribution (15-84).

Gender	Share (%)
Male	49.7
Female	50.3

Appendix

Table 3: Gender distribution.

Nationality	Share (%)
English	24.8
Australian	22.5
Chinese	8.8
Irish	8.2
Scottish	6.9
Italian	6.7

Appendix

Table 4: Nationality distribution.

Occupation	Share (%)
Professionals	39.4
Managers	13.3
Community and Personal Service Workers	11.0
Clerical and Administrative Workers	11.0
Technicians and Trades Workers	7.7
Sales Workers	7.0

Appendix

Table 5: Occupation distribution.

Family composition	Share (%)
Couple without children	62.8
Couple with children	21.5
One parent family	10.0
Other family	5.7

Appendix

Table 6: Family composition distribution.

Household composition	Share (%)
Family household	43.1
Lone person household	43.1
Group household	13.7

Appendix

Table 7: Household composition distribution.

Figure 5: Target Agent performance from different LLM backbones.

Location	Job titles
Restaurant	Head Chef; Sous Chef; Line Cook; Pastry Chef; Restaurant Manager; Waiter/Waitress; Bartender; Barista; Host/Hostess; Dishwasher
School	Primary School Teacher; Secondary School Teacher; School Principal; School Administrator; School Counsellor; Teacher Aide; School Librarian; School Cleaner; IT Support Technician (School)
Hospital	General Practitioner; Surgeon; Registered Nurse; Enrolled Nurse; Physiotherapist; Radiographer; Pharmacist; Ward Clerk; Hospital Receptionist; Hospital Cleaner
Office	Software Engineer; Data Analyst; Accountant; Human Resources Officer; Project Manager; UX Designer; Marketing Specialist; Sales Representative; Office Administrator; Customer Support Officer
Gym	Personal Trainer; Group Fitness Instructor; Gym Receptionist; Gym Manager; Physiologist; Nutrition Coach; Cleaning Staff (Gym)
Mall	Retail Sales Assistant; Cashier; Store Manager; Security Guard; Customer Service Officer; Cleaner (Mall); Barista (Mall Cafe); Food Court Attendant

Appendix

Table 8: Catalogue of job titles by location type.

Category	Relationship types
Family	mother; father; parent; son; daughter; child; sibling; spouse; partner; relative
Household	housemate; flatmate
Work	colleague; manager; subordinate
Education	teacher; student; classmate
Other	friend; neighbour; stranger

Appendix

Table 9: Catalogue of relationship types.

Location	Quantity	# of Cultural Norms
School	10	3,920
Office	20	7,010
Apartment	22	10,274
Hospital	12	2,658
Park	10	4,08
Shopping Mall	12	4,398

Appendix

Table 10: Location and cultural norms distribution in the simulation.

Culture	Cultural Norms
English	4,781
Australian	2,939
Chinese	2,869
Irish	1,171
Scottish	346
Italian	1,319

Appendix

Table 11: Cultural norm distribution.

Model	temperature	top_p	top_k	min_p
Qwen 3	0.6	0.95	20	0
Ministral 3 Reasoning	0.7	0.95	–	–
Llama 3	0.6	0.9	–	–
Gemini 2.5	1.0	0.95	64	–

Appendix

Table 12: Decoding parameter values of different LLMs in our experiments.

Culture Group	Qwen 3			Ministral 3 Reasoning		Llama 3		Gemini
Culture Group	Qwen3-8B	Qwen3-14B	Qwen3-32B	Ministral-3-8B	Ministral-3-14B	Llama3-8B	Llama3-70B	Gemini-2.5-Flash	Gemini-2.5-Pro
# of Cultures: 1	0.48	0.52	0.57	0.39	0.40	0.33	0.55	0.69	0.76
# of Cultures: 2	0.48	0.48	0.53	0.35	0.40	0.30	0.54	0.64	0.74
# of Cultures: 3	0.45	0.45	0.52	0.31	0.37	0.29	0.51	0.67	0.72
# of Cultures: 4	0.40	0.49	0.46	0.36	0.36	0.26	0.40	0.61	0.64
# of Cultures: 5	0.30	0.50	0.36	0.25	0.16	0.09	0.34	0.63	0.61

Appendix

Table 13: Target Agent performance in multicultural scenarios Norm Adherence is used as the evaluation metric here.

Culture Group	Qwen 3			Ministral 3 Reasoning		Llama 3		Gemini
Culture Group	Qwen3-8B	Qwen3-14B	Qwen3-32B	Ministral-3-8B	Ministral-3-14B	Llama3-8B	Llama3-70B	Gemini-2.5-Flash	Gemini-2.5-Pro
# Cultures: 1	0.68	0.73	0.79	0.87	0.88	0.66	0.78	0.84	0.95
# Cultures: 2	0.63	0.73	0.70	0.85	0.78	0.63	0.75	0.83	0.92
# Cultures: 3	0.64	0.73	0.73	0.85	0.84	0.59	0.72	0.84	0.93
# Cultures: 4	0.59	0.64	0.78	0.77	0.84	0.56	0.77	0.81	0.86
# Cultures: 5	0.63	0.68	0.75	0.83	0.78	0.63	0.75	0.80	0.91

Appendix

Table 14: Target Agent performance in multicultural scenarios Goal Completion is used as the evaluation metric here.

Location	Qwen 3			Ministral 3 Reasoning		Llama 3		Gemini
Location	Qwen3-8B	Qwen3-14B	Qwen3-32B	Ministral-3-8B	Ministral-3-14B	Llama3-8B	Llama3-70B	Gemini-2.5-Flash	Gemini-2.5-Pro
School	0.36	0.44	0.49	0.38	0.29	0.27	0.40	0.63	0.68
Office	0.42	0.49	0.45	0.20	0.27	0.19	0.42	0.56	0.62
Apartment	0.53	0.60	0.59	0.41	0.45	0.37	0.63	0.75	0.78
Hospital	0.40	0.39	0.54	0.25	0.21	0.18	0.51	0.61	0.58
Park	0.53	0.56	0.58	0.42	0.41	0.36	0.56	0.72	0.78

Appendix

Table 15: Target Agent performance in different locations. Norm Adherence is used as the evaluation metric here.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Cross-Cultural Language Model Evaluation LLM Agent Evaluation
(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University
The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs
Oct 19, 2026Samee Arif, Furkan Danisman, Angana Borah +6Cultural Bias in Language Models Cultural Knowledge in Language Models
CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs
Jun 1, 2026Yangfan Ye, Xiaocheng Feng, Jialong Tang +5LLM Evaluation Cultural Bias in Language Models
Harbin Institute of Technology · Harvard University · The University of Hong Kong

Sep 1, 2026J/K move · Enter open · S save

WorldBench: Culturally Grounded Benchmark for Multilingual Agents

Leonardo Ranaldi, Sherrie Shen, Jushi Kai, Alexandra Birch

(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University

Cross-Cultural Language Model Evaluation LLM Agent Evaluation AI Agent Benchmarks Cultural Knowledge in Language Models Long-Horizon LLM Agents

Open paper AskPDF arXiv

Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, performance across languages, and application to realistic, grounded scenarios. To address these concerns, we present WorldBench: a comprehensive, multilingual benchmark of genuine, persona-grounded everyday workflows, where agents can act in a sandbox via structured actions. WorldBench comprises 1,600 tasks across seven languages and eight cultures, filtered and refined through feedback from human annotators with language- and culture-specific expertise. For evaluation, we extend metrics from previous works and introduce Constrained Task Success (CTS), which combines natural language instructions and testbeds to score task completion, minimal modification, and other complementary metrics through deterministic and LLM-as-a-Judge evaluations. Our experiments show that frontier models reach only 49.2% CTS, with all models demonstrating large gaps between correctness and environment preservation. We thereby show that current agents remain brittle in multilingual, agentic scenarios, especially for long-horizon tasks and under state-preservation constraints

Abstract

AI Agent Benchmarks

Explore similar work

WorldBench: Culturally Grounded Benchmark for Multilingual Agents

The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs