LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo

Ojas Jain; Dhruv Kumar

cs.AIApr 7, 2026

LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo

Authors: Ojas Jain, Dhruv Kumar

Organizations: Department of Computer Science and Information Systems, BITS Pilani

Abstract

We introduce LudoBench, a benchmark for evaluating LLM strategic reasoning in Ludo, a stochastic multi-agent board game whose dice mechanics, piece capture, safe-square navigation, and home-path progression introduce meaningful planning complexity. LudoBench comprises 480 handcrafted spot scenarios across 12 behaviorally distinct decision categories, each isolating a specific strategic choice. We additionally contribute a fully functional 4-player Ludo simulator supporting Random, Heuristic, Game-Theory, and LLM agents. The game-theory agent uses Expectiminimax search with depth-limited lookahead to provide a principled strategic ceiling beyond greedy heuristics. Evaluating six models spanning four model families, we find that all models agree with the game-theory baseline only 40-46% of the time. Models split into distinct behavioral archetypes: finishers that complete pieces but neglect development, and builders that develop but never finish. Each archetype captures only half of the game theory strategy. Models also display measurable behavioral shifts under history-conditioned grudge framing on identical board states, revealing prompt-sensitivity as a key vulnerability. LudoBench provides a lightweight and interpretable framework for benchmarking LLM strategic reasoning under uncertainty. All code, the spot dataset (480 entries) and model outputs are available at https://anonymous.4open.science/r/LudoBench-5CBF/

Figures & tables

Property	( Campbell et al., 2002 )	( Silver et al., 2016 )	( Zhuang et al., 2025 )	( FAIR, 2022 )	(ours)
	Chess	Go	Poker	Dipl.	Ludo
Stochastic	✗	✗	✓	✗	✓
Complete information	✓	✓	✗	✓	✓
Multi-agent (3+)	✗	✗	✓	✓	✓
Multi-piece coord.	✓	✓	✗	✗	✓
Identical pieces	✗	✓	✗	✗	✓

Table 1: Positioning of LudoBench against existing LLM game benchmarks. ✓ = present, ✗ = absent.

Table 3

Category	QP	Q7B	DS	Haiku	Scout	Gemma
blocked	0.03	0.28	0.03	0.03	0.13	0.20
capture_vs_home_finish	0.00	0.60	0.00	0.00	0.08	0.13
capture_vs_home	0.00	0.68	0.00	0.00	0.10	0.13
capture_vs_openexisting	0.00	0.00	0.00	0.00	0.08	0.03
capture_vs_safe	0.00	0.55	0.00	0.05	0.15	0.20
capture	0.00	0.50	0.00	0.00	0.10	0.15

Table 4: Illegal-move rate ( llm_invalid_rate ) by model and category under the neutral persona. Each value is the fraction of spots where the model’s output was not a legal move.

Table 8

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Zone	Indices	Notes
Main Board	0…51 (52 squares)	Circular, shared by all players
Base (off-board)	$-1$	Pieces start here; leave only on dice $=6$
Home Path P0	52…57 (home_end $=57$ )	Private; entered after one full lap
Home Path P1	58…63 (home_end $=63$ )	Private
Home Path P2	64…69 (home_end $=69$ )	Private
Home Path P3	70…75 (home_end $=75$ )	Private

Appendix

Table 7: Board zone specifications.

Rule	Behavior
Dice range	1 to 6
Leave base	Only on dice $=6$ ; lands on player start square
Extra turn	Only on dice $=6$ (NOT for capture, NOT for reaching home)
Home entry	Exact landing required; overshoot is illegal
Capture	Only on non-safe main-board squares; captured piece $\rightarrow$ base ( $-1$ )
Same-player stacking	Blocked on all squares (exception: home_end )

Appendix

Table 8: Movement and capture rules implemented in LudoBench .

Group	Metric	Definition
Reliability	llm_invalid_rate	Fraction of all spots where the LLM’s raw output was not a legal move. Computed before heuristic fallback.
Behavioral	capture_rate	Fraction choosing the capture move.
	safe_rate	Fraction choosing the safe-square move.
	home_entry_rate	Fraction choosing home path entry or progress.
	home_finish_rate	Fraction choosing to finish a piece at home_end.
	open_rate	Fraction choosing to bring a base piece into play (dice $=$ 6).

Appendix

Table 9: Metrics definitions. All behavioral rates computed on valid outputs only; llm_invalid_rate computed on all outputs pre-fallback.

Metric	QP	Q7B	DS	Haiku	Scout	Gemma	GT	Heur.	Rand.
Pure Preference (none persona)
capture_rate	0.50	0.55	0.15	0.53	0.22	0.29	0.88	1.00	0.45
safe_rate	0.28	0.47	0.12	0.75	0.05	0.15	0.88	0.75	0.53
home_entry_rate	0.42	0.79	0.17	0.82	0.14	0.16	0.13	1.00	0.50
bring_out_rate	0.30	0.60	0.88	0.05	0.70	0.82	1.00	1.00	0.78
move_existing_rate	0.70	0.40	0.12	0.95	0.30	0.18	0.00	0.00	0.22

Appendix

Table 10: Full behavioral profiles under the neutral persona. GT = Game-Theory Agent, Heur. = Heuristic Agent, Rand. = Random Agent. All baselines are deterministic and persona-invariant.

Category	Metric	QP	Q7B	DS	Haiku	Scout	Gemma	GT	Heur.	Rand.
cvs_safe	capture	0.42	0.61	0.10	0.66	0.29	0.34	0.98	1.00	0.63
cvs_safe	safe	0.57	0.39	0.90	0.34	0.71	0.66	0.03	0.00	0.38
cvs_home	capture	0.57	0.15	0.72	0.20	0.78	0.83	0.98	1.00	0.35
cvs_home	home_entry	0.42	0.85	0.28	0.80	0.22	0.17	0.03	0.00	0.65
cvs_finish	capture	0.20	0.00	1.00	0.12	0.95	1.00	0.00	1.00	0.48
cvs_finish	home_finish	0.80	1.00	0.00	0.88	0.05	0.00	1.00	0.00	0.53

Appendix

Table 11: Tradeoff category decision rates under the neutral persona. cvs = capture_vs. GT = Game-Theory Agent, Heur. = Heuristic Agent, Rand. = Random Agent.

Metric	QP	Q7B	DS	Haiku	Scout	Gemma
change_rate	0.33	0.00	0.20	0.05	0.06	0.03
ret. grudge	0.80	0.57	0.78	0.31	0.86	0.88
ret. noconflict	0.68	0.57	0.73	0.33	0.83	0.85
grudge effect ( $\Delta$ )	+0.12	0.00	+0.05	$-$ 0.03	+0.03	+0.03

Appendix

Table 12: Grudge sensitivity under the neutral persona. change_rate = fraction of spots where the model’s decision changed between neutral and grudge framing on identical boards. The GT agent scores 0% on all metrics.

		QP					Q7B					DS					Haiku					Scout					Gemma
Cat.	Met.	n	a	g	s	u	n	a	g	s	u	n	a	g	s	u	n	a	g	s	u	n	a	g	s	u	n	a	g	s	u
cvf	cap	.20	.93	.12	.62	.57	.00	.14	.06	.00	.16	1.0	1.0	1.0	1.0	1.0	.12	.05	.15	.03	.25	.95	.97	1.0	.92	1.0	1.0	1.0	1.0	.94	.95
cvf	fin	.80	.07	.88	.38	.42	1.0	.86	.94	1.0	.84	.00	.00	.00	.00	.00	.88	.95	.85	.97	.75	.05	.03	.00	.08	.00	.00	.00	.00	.06	.05
cvs	cap	.42	.15	.53	.40	.15	.61	.71	.80	.63	.39	.10	.20	.23	.15	.25	.66	.84	.75	.88	.72	.29	.19	.21	.09	.19	.34	.14	.17	.11	.17
cvs	safe	.57	.85	.47	.60	.85	.39	.29	.20	.37	.61	.90	.80	.77	.85	.75	.34	.16	.25	.12	.28	.71	.81	.79	.91	.81	.66	.86	.83	.89	.83
cap	cap	.50	.28	.57	.38	.33	.55	.67	.50	.68	.65	.15	.17	.30	.07	.17	.53	.90	.78	.85	.74	.22	.16	.29	.17	.19	.29	.16	.24	.21	.16

Appendix

Table 13: Persona effects on key metrics across all six models. n=none, a=aggressive, g=greedy, s=safe, u=unforgiving; cvf=capture_vs_home_finish, cvs=capture_vs_safe, cap=capture, hom=home_entry, ext=extra_turn.

piece State	Score Formula	Range
Base ( $\text{pos}=-1$ )	Fixed value	50
Main board, stays on main	$\text{rel\_pos}+\text{dice}$	0–51
Main board, enters home	$52+\text{home\_step}$	53+
Already in home path	$52+(\text{new\_pos}-\text{home\_start}+1)$	Increasing
Capture available	$+100$ bonus (non-safe square, opponent present)	Additive
Safe square destination	$+20$ bonus (dest. is safe)	Additive

Appendix

Table 14: HeuristicAgent scoring formula. The agent selects the piece with the highest composite score among all legal moves.

Category	Entries	Unique Boards	History
blocked	40	40	No
capture	40	40	No
capture_vs_home	40	40	No
capture_vs_home_finish	40	40	No
capture_vs_openexist.	40	40	No
capture_vs_safe	40	40	No

Appendix

Table 15: Dataset summary.

\dagger

Each category’s source file follows the pattern spots_<category>.json . grudge_paired contains 40 boards

\times

2 conditions (neutral _a and grudge _b ), yielding 80 entries from 40 unique boards.

Category	2P	3P	4P
	Player Count
blocked	14	13	13
capture	14	13	13
capture_vs_home	14	13	13
capture_vs_home_finish	14	13	13
capture_vs_openexist.	14	13	13

Appendix

Table 16: Distribution of spots across player configurations (2-player, 3-player, 4-player). † grudge_paired counts are doubled due to _a / _b pairing. All non-grudge categories are approximately balanced across configurations.

Field	Type	Description
id	string	Unique identifier (e.g., cvs_2p_001 , grudge_pair1001_a )
scenario	string	Category label matching one of the 12 evaluation types
players	list[int]	Active player IDs in this game (e.g., [0, 1] for 2-player)
llm_player_id	int	Which player index the LLM controls
current_player	int	Whose turn it is (always equals llm_player_id )
dice	int	Dice roll value for this decision (1–6)

Appendix

Table 17: JSON schema for each spot entry. The history_text field is present only in grudge_paired scenarios.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

Choosing LLMs for Strategic Agent Interactions
Oct 13, 2026Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz +1Model Selection LLM Decision-Making
University of Exeter United Kingdom · IBM Research Switzerland
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
May 22, 2026Vartan Shadarevian, Kia Ghods, Alex Kenich +1LLM Evaluation Imperfect-Information Games
Princeton University · Google
Not Yet: Humans Outperform LLMs in a Colonel Blotto Tournament
May 21, 2026Dmitry Dagaev, Egor Ivanov, Petr Parshakov +2Game-Playing Agents Game Theory
HSE University · New Economic School · Central Economic Mathematical Institute, Russian Academy of Sciences +3

Oct 13, 2026J/K move · Enter open · S save

Choosing LLMs for Strategic Agent Interactions

Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz, Shiqiang Wang

University of Exeter United Kingdom · IBM Research Switzerland

Model Selection LLM Decision-Making AI Agent Benchmarks Game-Playing Agents

Open paper AskPDF arXiv

Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capability dimensions. However, they only evaluate strategic performance across individual fixed models. In parallel, contemporary LLM routers pick models under quality-cost trade-offs, but with no notion of a strategic structure. They do not account for an opponent that might be unknown, and that can adapt over time and shift the feedback signal. To address this, we extend the game-theoretic evaluations and define strategic selection as the meta game layered on top of the base game. In each round, a strategic selector fields an LLM from a portfolio of models to navigate a game payoff matrix that is latent and learned online. We find that the evaluation signal is game-specific, and that extracting a reliable signal requires a much higher number of seed runs

N

compared to traditional benchmarking setups. Evaluating across 5 games and 10 LLMs, we show a model's strategic competence varies significantly depending on the game setup and the opponent faced. Building on this, we implement a suite of selector methods and find that approaches that rely on query embeddings or LLM self-reflection do not consistently capture the underlying strategic signal. A lightweight multi-armed bandit algorithm performs more reliably across games, though it still falls short of the best fixed model. Finally, we show that simple non-stationary and adversarial bandit variants fail to outperform a stationary bandit baseline, pointing to a gap in how current methods handle a shifting opponent in strategic settings.

Abstract

LLM Agent Evaluation

Explore similar work

Choosing LLMs for Strategic Agent Interactions

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

Not Yet: Humans Outperform LLMs in a Colonel Blotto Tournament