The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturation and leakage. In this paper, we propose a dynamic benchmarking framework for evaluating multilingual and multicultural capabilities through the social deduction game Spyfall. In our setup, models must engage in strategic dialogue to either identify a secret agent or avoid detection, utilizing culturally relevant locations or local foods. Our results show that our game-based rankings are consistent with Chatbot Arena rankings on the overlapping models. However, we find a significant performance gap in non-English contexts: models are generally less proficient when handling locally specific entities and often struggle with rule-following or strategic integrity in non-English languages. We demonstrate that this game-based approach provides a scalable, leakage-resistant, and culturally nuanced alternative to traditional NLP benchmarks. The dataset (game history) and source code can be accessed here: https://huggingface.co/datasets/haryoaw/cultural-spyfall and https://github.com/haryoa/cultural-spyfall.
Figures & tables
Figure 1: In this example, the Spy fails to identify the target entity, Jam Gadang. Due to insufficient context and careful questioning from the other players, the Spy guesses Monas (a landmark in a different part of Indonesia) and loses the game.
Figure 2: The flow of the game of Multicultural Spyfall
Model
Overall
Generic Location
Local Location
Local Food
G en
G id
G arz
G zh
L id
L arz
L zh
F id
F arz
F zh
1
Gemini-P
1136 ±11
1086 ±33
1116 ±35
1163 ±40
1174 ±39
1154 ±38
1171 ±41
1161 ±42
1075 ±36
1157 ±37
1136 ±38
2
Gemini-F
1107 ±11
1057 ±34
1071 ±34
1130 ±38
1103 ±39
1086 ±34
1161 ±36
1139 ±39
1098 ±35
1112 ±38
1150 ±38
3
Qwen30B-T
1016 ±12
1045 ±33
982 ±35
1032 ±37
1030 ±38
996 ±35
1071 ±36
985 ±38
1002 ±36
1032 ±37
1005 ±34
4
Gemma3-12B
1003 ±11
995 ±32
1031 ±34
1050 ±35
997 ±36
975 ±34
1000 ±38
994 ±36
1023 ±36
1001 ±36
981 ±36
5
Qwen3-8B
967 ±11
948 ±35
954 ±33
964 ±37
970 ±34
939 ±33
959 ±39
959 ±35
994 ±33
1034 ±37
962 ±35
Table 1: Bradley-Terry ratings across scenarios with 95% bootstrap confidence interval with 1000 resamples. Gold , Silver , Bronze , and 4th indicate top 4 per column.
Figure 3: Head-to-head win rate (%) of the row model against the column model, over 600 matches per pair. Blue: the row model wins more often; orange: it loses more often.
Model
en
id
zh
arz
Gemini-F
0.0
0.0
0.0
0.0
Gemini-P
0.0
0.0
0.0
0.0
Gemma3-12B
0.0
1.6
4.9
1.6
Llama3.1-8B
17.3
37.1
29.6
13.6
Qwen30B-T
0.7
0.4
0.0
0.0
Qwen3-8B
0.0
1.8
8.7
2.4
Table 2: Non-Spy Information Leakage Rates (%) by Language.
Model
id
zh
arz
Gemini-F
id 98.5 ms 1.0
zh 99.9 wuu 0.1
arz 70.89 apd 10.45
Gemini-P
id 98.7 ms 1.2
zh 99.9 ja 0.1
arz 80.47 apd 7.56
Gemma3-12B
id 98.8 ms 0.9
zh 99.5 en 0.2
msa 59.44 acx 19.89
Llama3.1-8B
id 96.6 ms 2.3
zh 89.2 en 8.0
msa 29.76 acx 28.99
Qwen30B-T
id 98.9 ms 0.7
zh 95.4 en 4.2
msa 49.51 acx 25.93
Qwen3-8B
id 98.6 ms 0.6
zh 99.9 en 0.1
msa 50.3 acx 34.70
Table 3: Language detection of outputs. Top-1 (first) / Top-2 (second).
Table 4: Detailed Guess Distribution for High-Entropy Locations and low accuracy Et
G
L
F
Acc
H
Acc
H
Acc
H
en
67.3
1.49
–
–
–
–
id
67.1
1.49
49.6
1.81
40.9
2.35
zh
64.2
1.55
54.5
1.50
51.6
2.14
arz
59.5
1.69
33.8
2.10
24.4
2.80
Table 5: Overall Accuracy (Acc) and Entropy (H) by Dataset
Figure 4: Spy win rate (%) by model and scenario. Colors are centered at 50% (blue: the spy wins more often than it loses; orange: less often). Bold : best model per column; All pools every scenario.
Category
Count
%
Spy Guess Wrong
2,917
32.41
Spy Guess Correct
3,257
36.19
Vote Majority to Spy
716
7.96
Vote Majority to Non-Spy
363
4.03
Spy Quit
458
5.09
Non-Spy Quit
1,289
14.32
Table 6: Overall Game Outcome Statistics
Model
Guess ×
Got Voted
Quit
Llama3.1-8B
82.4
5.4
12.2
Qwen3-8B
81.1
7.1
11.8
Qwen30B-T
80.8
6.6
12.6
Gemma3-12B
78.0
8.8
13.2
Gemini-P
63.6
25.9
10.5
Gemini-F
60.8
29.4
9.7
Table 7: Non-Spy Victory Distribution by Model (when model is non-spy). Column denotes spy last action. Percentages show the win rate win methods of Non-spies
Figure 5: Spy vote dispersion score by scenario (rows) and model playing the spy (columns); higher is better for the spy. Bold : best model per row; All is the unweighted mean over models (column) or scenarios (row).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Probability that the row model has a higher Bradley-Terry rating than the column model, from 1000 bootstrap resamples of all matches. Values ≥ 0.975 (or ≤ 0.025) indicate a significant ordering at the 5% level; dashed outlines mark the only pair that is not significant.
Figure 7: How games end when each model plays the spy, by language (150 games per model for en , 450 for each other language, pooled over the G, L and F scenarios). The black tick separates spy wins (below) from spy losses (above). No game reached the turn limit.
Figure 8: Spy guess accuracy (%) by scenario, excluding games with leakage.
Model
Generic Spy Win
Local Food Spy Win
Local Loc. Spy Win
Gemini-P
49.33% (n=74)
24.67% (n=37)
33.33% (n=50)
Gemini-F
47.33% (n=71)
28.67% (n=43)
38.67% (n=58)
Qwen30B-T
50.00% (n=75)
15.33% (n=23)
21.33% (n=32)
Gemma3-12B
33.33% (n=50)
10.67% (n=16)
10.00% (n=15)
Qwen3-8B
22.00% (n=33)
8.67% (n=13)
10.67% (n=16)
Llama3.1-8B
7.33% (n=11)
1.33% (n=2)
1.33% (n=2)
Appendix
Table 8: Spy win rates with guessed location across Generic, Local Food, and Local Location settings.
Model
Generic Non-Spy Win
Local Food Non-Spy Win
Local Loc. Non-Spy Win
Gemini-P
34.00% (n=51)
60.00% (n=90)
50.67% (n=76)
Gemini-F
23.33% (n=35)
54.00% (n=81)
44.00% (n=66)
Qwen30B-T
28.00% (n=42)
54.67% (n=82)
50.00% (n=75)
Gemma3-12B
30.00% (n=45)
43.33% (n=65)
35.33% (n=53)
Qwen3-8B
20.67% (n=31)
59.33% (n=89)
38.00% (n=57)
Llama3.1-8B
4.67% (n=7)
4.67% (n=7)
2.67% (n=4)
Appendix
Table 9: Non-Spy win rates (spy guessed wrong) across Generic, Local Food, and Local Location settings.
Model
id
zh
arz
Avg
Gemini-F
2.0
3.3
0.7
2.0
Gemini-P
3.8
6.0
9.8
6.5
Gemma3-12B
3.6
7.6
6.4
5.9
Llama3.1-8B
2.0
5.3
3.1
3.5
Qwen30B-T
0.2
0.4
1.6
0.7
Qwen3-8B
4.0
6.4
4.7
5.0
Appendix
Table 10: Vote wrong rate (%) by model across languages when these models play as a spy. This metric shows how often the opposing non-spy players incorrectly voted out a teammate. Avg is the unweighted mean across languages.
Figure 9: Non-spy vote entropy by model and scenario; lower means the non-spies agree more. Bold : lowest per column.
Model
en
id
zh
arz
Gemini-F
100.0 (31)
94.1(17)
92.9(14)
87.5(8)
Gemini-P
95.2 (21)
100.0(23)
100.0(16)
100.0(6)
Gemma3-12B
93.8 (16)
94.4(18)
85.7(14)
100.0(4)
Llama3.1-8B
66.7 (6)
100.0(5)
83.3(6)
–
Qwen30B-T
88.5 (26)
95.8(24)
86.7(15)
83.3(6)
Qwen3-8B
88.0 (25)
96.3(27)
80.0(10)
0.0(1)
Appendix
Table 11: Spy guess accuracy with non-spy leakage by language.
Figure 10: Non-spy voting accuracy (%): the share of non-skip votes that target the spy, with the number of votes below. Colors are centered at the 25% random baseline (blue: above, orange: below). Bold : best model per column.
Rank
Model
Rating
Win Rate (%)
1
Gemini-P
1135.8
70.13
2
Gemini-F
1107.4
66.00
3
Qwen30B-T
1015.7
51.93
4
Gemma3-12B
1003.0
49.93
5
Qwen3-8B
966.8
44.30
6
Llama3.1-8B
771.3
17.70
Appendix
Table 12: Bradley-Terry ratings and overall win rates for each model.
Spy win (%)
Non-spy win (%)
BT rating
Model
R1
R2
R3
R1
R2
R3
R1
R2
R3
Gemini-P
60.6
64.8
52.1
81.8
85.5
81.8
1133
1157
1091
Gemini-F
53.0
38.6
49.4
85.4
82.0
86.5
1137
1067
1121
Gemma3-12B
39.6
42.5
37.7
64.6
64.6
71.7
1010
1013
1025
Llama3.1-8B
11.3
8.5
9.4
4.9
14.6
15.4
638
701
715
Qwen30B-T
43.0
43.8
39.7
68.5
75.0
72.2
1029
1044
1025
Appendix
Table 13: Consistency on F arz : win rate as spy and as non-spy, and Bradley-Terry (BT) rating. R1, R2, R3 = three independent repeats. Results obtained on the 598 games. ICC(2,1) = intraclass correlation (two-way random, absolute agreement) across models; mean SD = average per-model standard deviation across runs (percentage points for win rates, rating points for ratings).
Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark (https://github.com/lunyiliu/GaoYao).
Large language models (LLMs) are increasingly deployed as autonomous agents, yet evaluations focus primarily on task success rather than cultural appropriateness or evaluator reliability. We introduce LiveCultureBench, a multi-cultural, dynamic benchmark that embeds LLMs as agents in a simulated town and evaluates them on both task completion and adherence to socio-cultural norms. The simulation models a small city as a location graph with synthetic residents having diverse demographic and cultural profiles. Each episode assigns one resident a daily goal while others provide social context. An LLM-based verifier generates structured judgments on norm violations and task progress, which we aggregate into metrics capturing task-norm trade-offs and verifier uncertainty. Using LiveCultureBench across models and cultural profiles, we study (i) cross-cultural robustness of LLM agents, (ii) how they balance effectiveness against norm sensitivity, and (iii) when LLM-as-a-judge evaluation is reliable for automated benchmarking versus when human oversight is needed.
Viet-Thanh Pham, Lizhen Qu, Thuy-Trang Vu +2
Department of Data Science & AI, Monash University
Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks -- where models must reason within real-world, context-rich scenarios -- largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs' multilingual and multicultural competence on grounded tasks. CulturALL is built via a human--AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages from 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. Experiments show that the best LLM achieves 44.48% accuracy on CulturALL, underscoring substantial room for improvement.
Peiqin Lin, Chenyang Lyu, Wenjiang Luo +22
Alibaba Group · Beijing Language and Culture University · LMU Munich +6