The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturation and leakage. In this paper, we propose a dynamic benchmarking framework for evaluating multilingual and multicultural capabilities through the social deduction game Spyfall. In our setup, models must engage in strategic dialogue to either identify a secret agent or avoid detection, utilizing culturally relevant locations or local foods. Our results show that our game-based rankings are consistent with Chatbot Arena rankings on the overlapping models. However, we find a significant performance gap in non-English contexts: models are generally less proficient when handling locally specific entities and often struggle with rule-following or strategic integrity in non-English languages. We demonstrate that this game-based approach provides a scalable, leakage-resistant, and culturally nuanced alternative to traditional NLP benchmarks. The dataset (game history) and source code can be accessed here: https://huggingface.co/datasets/haryoaw/cultural-spyfall and https://github.com/haryoa/cultural-spyfall.
Figures & tables
Figure 1: In this example, the Spy fails to identify the target entity, Jam Gadang. Due to insufficient context and careful questioning from the other players, the Spy guesses Monas (a landmark in a different part of Indonesia) and loses the game.
Figure 2: The flow of the game of Multicultural Spyfall
Model
Overall
Generic Location
Local Location
Local Food
G en
G id
G arz
G zh
L id
L arz
L zh
F id
F arz
F zh
1
Gemini-P
1136 ±11
1086 ±33
1116 ±35
1163 ±40
1174 ±39
1154 ±38
1171 ±41
1161 ±42
1075 ±36
1157 ±37
1136 ±38
2
Gemini-F
1107 ±11
1057 ±34
1071 ±34
1130 ±38
1103 ±39
1086 ±34
1161 ±36
1139 ±39
1098 ±35
1112 ±38
1150 ±38
3
Qwen30B-T
1016 ±12
1045 ±33
982 ±35
1032 ±37
1030 ±38
996 ±35
1071 ±36
985 ±38
1002 ±36
1032 ±37
1005 ±34
4
Gemma3-12B
1003 ±11
995 ±32
1031 ±34
1050 ±35
997 ±36
975 ±34
1000 ±38
994 ±36
1023 ±36
1001 ±36
981 ±36
5
Qwen3-8B
967 ±11
948 ±35
954 ±33
964 ±37
970 ±34
939 ±33
959 ±39
959 ±35
994 ±33
1034 ±37
962 ±35
Table 1: Bradley-Terry ratings across scenarios with 95% bootstrap confidence interval with 1000 resamples. Gold , Silver , Bronze , and 4th indicate top 4 per column.
Figure 3: Head-to-head win rate (%) of the row model against the column model, over 600 matches per pair. Blue: the row model wins more often; orange: it loses more often.
Model
en
id
zh
arz
Gemini-F
0.0
0.0
0.0
0.0
Gemini-P
0.0
0.0
0.0
0.0
Gemma3-12B
0.0
1.6
4.9
1.6
Llama3.1-8B
17.3
37.1
29.6
13.6
Qwen30B-T
0.7
0.4
0.0
0.0
Qwen3-8B
0.0
1.8
8.7
2.4
Table 2: Non-Spy Information Leakage Rates (%) by Language.
Model
id
zh
arz
Gemini-F
id 98.5 ms 1.0
zh 99.9 wuu 0.1
arz 70.89 apd 10.45
Gemini-P
id 98.7 ms 1.2
zh 99.9 ja 0.1
arz 80.47 apd 7.56
Gemma3-12B
id 98.8 ms 0.9
zh 99.5 en 0.2
msa 59.44 acx 19.89
Llama3.1-8B
id 96.6 ms 2.3
zh 89.2 en 8.0
msa 29.76 acx 28.99
Qwen30B-T
id 98.9 ms 0.7
zh 95.4 en 4.2
msa 49.51 acx 25.93
Qwen3-8B
id 98.6 ms 0.6
zh 99.9 en 0.1
msa 50.3 acx 34.70
Table 3: Language detection of outputs. Top-1 (first) / Top-2 (second).
Table 4: Detailed Guess Distribution for High-Entropy Locations and low accuracy Et
G
L
F
Acc
H
Acc
H
Acc
H
en
67.3
1.49
–
–
–
–
id
67.1
1.49
49.6
1.81
40.9
2.35
zh
64.2
1.55
54.5
1.50
51.6
2.14
arz
59.5
1.69
33.8
2.10
24.4
2.80
Table 5: Overall Accuracy (Acc) and Entropy (H) by Dataset
Figure 4: Spy win rate (%) by model and scenario. Colors are centered at 50% (blue: the spy wins more often than it loses; orange: less often). Bold : best model per column; All pools every scenario.
Category
Count
%
Spy Guess Wrong
2,917
32.41
Spy Guess Correct
3,257
36.19
Vote Majority to Spy
716
7.96
Vote Majority to Non-Spy
363
4.03
Spy Quit
458
5.09
Non-Spy Quit
1,289
14.32
Table 6: Overall Game Outcome Statistics
Model
Guess ×
Got Voted
Quit
Llama3.1-8B
82.4
5.4
12.2
Qwen3-8B
81.1
7.1
11.8
Qwen30B-T
80.8
6.6
12.6
Gemma3-12B
78.0
8.8
13.2
Gemini-P
63.6
25.9
10.5
Gemini-F
60.8
29.4
9.7
Table 7: Non-Spy Victory Distribution by Model (when model is non-spy). Column denotes spy last action. Percentages show the win rate win methods of Non-spies
Figure 5: Spy vote dispersion score by scenario (rows) and model playing the spy (columns); higher is better for the spy. Bold : best model per row; All is the unweighted mean over models (column) or scenarios (row).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Probability that the row model has a higher Bradley-Terry rating than the column model, from 1000 bootstrap resamples of all matches. Values ≥ 0.975 (or ≤ 0.025) indicate a significant ordering at the 5% level; dashed outlines mark the only pair that is not significant.
Figure 7: How games end when each model plays the spy, by language (150 games per model for en , 450 for each other language, pooled over the G, L and F scenarios). The black tick separates spy wins (below) from spy losses (above). No game reached the turn limit.
Figure 8: Spy guess accuracy (%) by scenario, excluding games with leakage.
Model
Generic Spy Win
Local Food Spy Win
Local Loc. Spy Win
Gemini-P
49.33% (n=74)
24.67% (n=37)
33.33% (n=50)
Gemini-F
47.33% (n=71)
28.67% (n=43)
38.67% (n=58)
Qwen30B-T
50.00% (n=75)
15.33% (n=23)
21.33% (n=32)
Gemma3-12B
33.33% (n=50)
10.67% (n=16)
10.00% (n=15)
Qwen3-8B
22.00% (n=33)
8.67% (n=13)
10.67% (n=16)
Llama3.1-8B
7.33% (n=11)
1.33% (n=2)
1.33% (n=2)
Appendix
Table 8: Spy win rates with guessed location across Generic, Local Food, and Local Location settings.
Model
Generic Non-Spy Win
Local Food Non-Spy Win
Local Loc. Non-Spy Win
Gemini-P
34.00% (n=51)
60.00% (n=90)
50.67% (n=76)
Gemini-F
23.33% (n=35)
54.00% (n=81)
44.00% (n=66)
Qwen30B-T
28.00% (n=42)
54.67% (n=82)
50.00% (n=75)
Gemma3-12B
30.00% (n=45)
43.33% (n=65)
35.33% (n=53)
Qwen3-8B
20.67% (n=31)
59.33% (n=89)
38.00% (n=57)
Llama3.1-8B
4.67% (n=7)
4.67% (n=7)
2.67% (n=4)
Appendix
Table 9: Non-Spy win rates (spy guessed wrong) across Generic, Local Food, and Local Location settings.
Model
id
zh
arz
Avg
Gemini-F
2.0
3.3
0.7
2.0
Gemini-P
3.8
6.0
9.8
6.5
Gemma3-12B
3.6
7.6
6.4
5.9
Llama3.1-8B
2.0
5.3
3.1
3.5
Qwen30B-T
0.2
0.4
1.6
0.7
Qwen3-8B
4.0
6.4
4.7
5.0
Appendix
Table 10: Vote wrong rate (%) by model across languages when these models play as a spy. This metric shows how often the opposing non-spy players incorrectly voted out a teammate. Avg is the unweighted mean across languages.
Figure 9: Non-spy vote entropy by model and scenario; lower means the non-spies agree more. Bold : lowest per column.
Model
en
id
zh
arz
Gemini-F
100.0 (31)
94.1(17)
92.9(14)
87.5(8)
Gemini-P
95.2 (21)
100.0(23)
100.0(16)
100.0(6)
Gemma3-12B
93.8 (16)
94.4(18)
85.7(14)
100.0(4)
Llama3.1-8B
66.7 (6)
100.0(5)
83.3(6)
–
Qwen30B-T
88.5 (26)
95.8(24)
86.7(15)
83.3(6)
Qwen3-8B
88.0 (25)
96.3(27)
80.0(10)
0.0(1)
Appendix
Table 11: Spy guess accuracy with non-spy leakage by language.
Figure 10: Non-spy voting accuracy (%): the share of non-skip votes that target the spy, with the number of votes below. Colors are centered at the 25% random baseline (blue: above, orange: below). Bold : best model per column.
Rank
Model
Rating
Win Rate (%)
1
Gemini-P
1135.8
70.13
2
Gemini-F
1107.4
66.00
3
Qwen30B-T
1015.7
51.93
4
Gemma3-12B
1003.0
49.93
5
Qwen3-8B
966.8
44.30
6
Llama3.1-8B
771.3
17.70
Appendix
Table 12: Bradley-Terry ratings and overall win rates for each model.
Spy win (%)
Non-spy win (%)
BT rating
Model
R1
R2
R3
R1
R2
R3
R1
R2
R3
Gemini-P
60.6
64.8
52.1
81.8
85.5
81.8
1133
1157
1091
Gemini-F
53.0
38.6
49.4
85.4
82.0
86.5
1137
1067
1121
Gemma3-12B
39.6
42.5
37.7
64.6
64.6
71.7
1010
1013
1025
Llama3.1-8B
11.3
8.5
9.4
4.9
14.6
15.4
638
701
715
Qwen30B-T
43.0
43.8
39.7
68.5
75.0
72.2
1029
1044
1025
Appendix
Table 13: Consistency on F arz : win rate as spy and as non-spy, and Bradley-Terry (BT) rating. R1, R2, R3 = three independent repeats. Results obtained on the 598 games. ICC(2,1) = intraclass correlation (two-way random, absolute agreement) across models; mean SD = average per-model standard deviation across runs (percentage points for win rates, rating points for ratings).