The Token Games: Evaluating Language Model Reasoning with Puzzle Duels

Simon Henniger; Gabriel Poesia

cs.AIFeb 19, 2026

The Token Games: Evaluating Language Model Reasoning with Puzzle Duels

Authors: Simon Henniger, Gabriel Poesia

Organizations: Harvard University

Abstract

Evaluating the reasoning capabilities of Large Language Models is increasingly challenging as models improve. Human curation of hard questions is highly expensive, especially in recent benchmarks using PhD-level domain knowledge to challenge the most capable models. Even then, there is always a concern about whether these questions test genuine reasoning or if similar problems have been seen during training. Here, we take inspiration from 16th-century mathematical duels to design The Token Games (TTG): an evaluation framework where models challenge each other by creating their own puzzles. We leverage the format of Programming Puzzles - given a function that returns a boolean, find inputs that make it return True - to flexibly represent problems and enable verifying solutions. Using results from pairwise duels, we then compute Elo ratings, allowing us to compare models relative to each other. We evaluate 10 frontier models on TTG, and closely match the ranking from existing benchmarks such as Humanity's Last Exam, spending less than $200 USD and without involving any human effort in creating puzzles. We also find that creating good puzzles is still a highly challenging task for current models. Overall, our work suggests new paradigms for evaluating reasoning that avoid saturation by design, and that allow testing models for other skills like creativity and task creation alongside problem solving.

Figures & tables

Figure 1: Reasoning duel in The Token Games. Two language models take turns between being puzzle proposers or solvers. Proposers score in a turn if they design a puzzle, give a correct solution, and have the opponent fail to solve the challenge. Puzzles are represented as Python functions returning a boolean value, with the challenge consisting of finding inputs to make it return true.

	TTG								HLE		ARC-AGI		SWE-BP		TQ		GPQA-D
Model	Elo	R	Solv%	R	SolvW%	R	Prop%	R	Acc	R	Acc	R	Acc	R	Acc	R	Acc	R
gpt-5.4	1746 ${}^{+213}_{-118}$	1	98.9	1	99.3	1	60.4	1	40.3	5	65.0	4	49.7	6	42.2	2	92.0	4
gpt-5.5	1445 ${}^{+120}_{-85}$	2	97.6	2	98.3	2	38.2	2	43.6	2	77.5	1	53.4	4	42.0	3	93.5	2
claude-opus-4-8	1337 ${}^{+102}_{-85}$	3	88.2	7	92.3	6	28.7	4	42.2	3	67.5	3	65.4	1	40.1	4	92.0	4
gemini-3.1-pro-pre…	1311 ${}^{+103}_{-87}$	4	85.7	8	90.1	8	33.8	3	45.9	1	73.3	2	46.7	7	45.8	1	94.1	1
claude-opus-4-7	1280 ${}^{+102}_{-89}$	5	92.0	3	94.4	3	16.9	6	39.0	6	50.8	6	60.9	2	37.0	5	91.4	6

Table 1: Model performance comparison on The Token Games and, in column order, Humanity’s Last Exam, ARC-AGI, SWE-Bench Pro, TextQuests, and GPQA Diamond. For TTG, we show match Elo with 95% bootstrap confidence intervals as super/subscripts, the clean solver success rate on valid puzzles (Solv%), the duel-scoring solver win rate (SolvW%), and the proposer win rate (Prop%), as defined in Section 3.2 . For the other benchmarks, we report public-leaderboard accuracies (sources in Appendix C ). Each metric is followed by the model’s rank; the top and bottom ranks are highlighted. The top-2 models are preserved (in order) in 97% of bootstrap resamples, the same top-3 set in 61%, and the same bottom-3 set in 73%; bootstrap details in Appendix A .

	vs HLE		vs ARC-AGI		vs SWE-BP		vs TQ		vs GPQA-D
TTG Metric	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$
Elo	+0.69	0.007	+0.84	0.000	+0.72	0.004	+0.77	0.001	+0.77	0.000
(w/o incomplete rows)	+0.78	0.002	+0.84	0.000	+0.78	0.002	+0.82	0.001	+0.83	0.000
Solver Win Rate	+0.70	0.005	+0.80	0.001	+0.80	0.001	+0.82	0.000	+0.83	0.000
(w/o incomplete rows)	+0.75	0.003	+0.80	0.001	+0.81	0.001	+0.82	0.001	+0.79	0.001
Proposer Win Rate	+0.81	0.000	+0.90	0.000	+0.57	0.034	+0.85	0.000	+0.82	0.000

Table 2: Spearman rank correlation (

\rho

) between model performance on TTG and reference benchmarks. Rows marked “w/o incomplete rows” recompute the correlations after excluding models that are missing scores on one or more of the benchmarks. Bolded results are for estimates with

p<0.05

. Table 8 in the appendix reports bootstrap confidence intervals, multiple-testing corrections, and a leave-one-out sensitivity analysis for these correlations.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Proposer	Distinctive preference	Exclusive tag
claude-haiku-4-5	factorials (1.4%, 12.9 $\times$ )	arithmetic-exponentiation (3)
claude-opus-4-7	string-to-digits (2.1%, 5.9 $\times$ )	involution (2)
claude-opus-4-7:low	range-checking (0.7%, 9.0 $\times$ )	arithmetic-algebra (2)
claude-opus-4-8	modular-square-roots (0.7%, 12.9 $\times$ )	code-points (2)
claude-sonnet-4-6	prime-testing (1.3%, 8.1 $\times$ )	math-isqrt (3)
deepseek-v3.2-thinking	rsa (1.3%, 7.7 $\times$ )	base64-encoding (2)

Appendix

Table 3: Per-proposer puzzle tag profile across all duels. Distinctive preference : among tags appearing

\geq 5

times for the proposer, the one with the largest “lift”, i.e., the ratio of the tag’s per-proposer frequency to its global frequency across all proposers; an

N\times

multiplier means the proposer uses the tag

N

times more often than the average proposer. Exclusive tag : the most-used tag whose appearances all come from that one proposer (puzzle count in parentheses).

Proposer Model	gpt-5.4-nano:high	grok-4-fast-reasoning	qwen3.8-27b	Total
claude-haiku-4-5	680/691 (98%)	633/689 (92%)	673/689 (98%)	1986/2069 (96%)
gpt-4o	177/180 (98%)	161/180 (89%)	178/180 (99%)	516/540 (96%)
grok-4-fast-reasoning	477/506 (94%)	464/503 (92%)	453/503 (90%)	1394/1512 (92%)
qwen/qwen3-coder	31/36 (86%)	31/35 (89%)	34/35 (97%)	96/106 (91%)
z-ai/glm-5.2	444/473 (94%)	351/473 (74%)	412/473 (87%)	1207/1419 (85%)
qwen/qwen3.7-max	615/659 (93%)	487/659 (74%)	563/659 (85%)	1665/1977 (84%)

Appendix

Table 4: Uniform solvers by proposer model. After completing all duels, we mined puzzles from their logs and had three models attempt them outside the duel context: Qwen3.8 27B, Grok 4 Fast Reasoning, and GPT-5.4 Nano. We only used valid puzzles, i.e. those that had correct sample solutions, and we show each solver’s solve rate for puzzles by their original proposer.

Model	Solv S (%)	Solv F	Prop S	Prop U (%)	Prop P (%)	Self S (%)	Either S (%)
gpt-5.4	450 (98.9%)	5	239	431 (64.3%)	44 (6.2%)	555 (82.8%)	571 (85.2%)
gpt-5.5	482 (97.6%)	12	372	273 (42.3%)	69 (9.7%)	618 (96.0%)	626 (97.2%)
claude-opus-4-7	458 (92.0%)	40	558	121 (17.8%)	35 (4.9%)	621 (91.5%)	651 (95.9%)
qwen/qwen3.7-ma…	475 (91.5%)	44	571	88 (13.4%)	55 (7.7%)	614 (93.2%)	645 (97.9%)
gpt-5.5:low	442 (89.3%)	53	172	68 (28.3%)	474 (66.4%)	231 (96.2%)	238 (99.2%)
claude-sonnet-4…	461 (88.5%)	60	602	95 (13.6%)	17 (2.4%)	638 (91.5%)	663 (95.1%)

Appendix

Table 5: Puzzle outcomes by model and role. The left columns show performance in the solver role (number of puzzles solved and failed to solve). The middle columns show performance in the proposer role (number of invented puzzles solved, unsolved (% of all correct (non-penalized) puzzles), puzzles with incorrect sample solutions which the proposer was penalized for). In a second run, we had every model try to solve its own puzzle without the prior context (i.e. without seeing its notes). “Self S” shows the proportion of the model’s proposed puzzles that it could itself solve in the separate context, and “Either S” shows the proportion of puzzles that could be solved by either the model itself in a separate context or the model’s opponent in the duel. This is to ensure that puzzles are solvable.

Turn Number	Solved	Unsolved	Penalty (%)
1	594	117	207 (22.5%)
2	540	133	245 (26.7%)
3	533	139	246 (26.8%)
4	517	135	266 (29.0%)
5	507	146	265 (28.9%)
6	480	160	278 (30.3%)

Appendix

Table 6: Puzzle Outcomes By Turn Number

Model	Pairings	Identical (%)	Reversed (%)	Score var.
gpt-4o	34	97.1	0.0	0.002
qwen/qwen3-coder	34	91.2	0.0	0.005
gpt-5.4	34	88.2	5.9	0.016
deepseek-v3.2-thinking	34	67.6	17.6	0.044
gpt-5.5	34	67.6	17.6	0.044
gemini-3.1-pro-preview	34	64.7	26.5	0.059

Appendix

Table 7: Run-to-run stability of match outcomes. Every ordered pairing (A proposes first, against B) is repeated three times in our tournament. Per model, over the pairings it took part in: Identical is the share of pairings with the same outcome in all three repetitions; Reversed is the share where the model both won and lost the same pairing across repetitions; Score var. is the mean within-pairing variance of the match score (win

=1

, draw

=0.5

, loss

=0

) — a fair coin between win and loss would give

0.25

. Overall, 61% of pairings are perfectly repeatable and only 18% ever reverse; most residual variability is between a win and a draw.

Elo	Bench.	$n$	$\rho$	95% CI	$p$	$p_{\mathrm{Bonf}}$	$p_{\mathrm{BH}}$	LOO $\rho$ range	Most infl. model ( $\rho$ w/o)	sig. all LOO
Match	HLE	14	$+0.688$	$[+0.26,+0.91]$	0.0065	0.0981	0.0070	$[+0.61,+0.78]$	z-ai/glm-5.2 ( $+0.78$ )	yes
	ARC-AGI	13	$+0.835$	$[+0.50,+0.97]$	0.0004	0.0057	0.0007	$[+0.79,+0.89]$	gemini-3-flash… ( $+0.89$ )	yes
	SWE-BP	14	$+0.719$	$[+0.36,+0.90]$	0.0038	0.0567	0.0047	$[+0.65,+0.78]$	gpt-4o ( $+0.65$ )	yes
	TQ	14	$+0.767$	$[+0.31,+0.96]$	0.0014	0.0205	0.0021	$[+0.71,+0.84]$	gemini-3-flash… ( $+0.84$ )	yes
	GPQA-D	17	$+0.771$	$[+0.42,+0.93]$	0.0003	0.0044	0.0007	$[+0.72,+0.81]$	gpt-4o ( $+0.72$ )	yes
Proposer	HLE	14	$+0.688$	$[+0.24,+0.91]$	0.0065	0.0981	0.0070	$[+0.61,+0.78]$	z-ai/glm-5.2 ( $+0.78$ )	yes

Appendix

Table 8: Robustness of the Spearman correlations between TTG Elo ratings (match, proposer, and solver Elo) and external benchmarks. For each pair we report the correlation

\rho

, a bootstrap 95% confidence interval (2000 resamples over models), the raw

p

-value alongside Bonferroni- and Benjamini-Hochberg-adjusted values across all 15 tests, and a leave-one-out (LOO) sensitivity analysis: the range of

\rho

when any single model is removed, the model whose removal moves

\rho

the most (with the resulting

\rho

), and whether the correlation remains significant (

p<0.05

) in every LOO subset.

Model	In $/M	Out $/M	In tokens	Out tokens	Calls	Cost (USD)
claude-haiku-4-5	1.00	5.00	2 011 860	17 406 364	1252	$89.04
claude-opus-4-7	5.00	25.00	2 343 607	8 134 940	1212	$215.09
claude-opus-4-7:low	5.00	25.00	2 176 891	3 328 092	1225	$94.09
claude-opus-4-8	5.00	25.00	2 779 193	8 774 103	1181	$233.25
claude-sonnet-4-6	3.00	15.00	2 323 996	20 482 467	1235	$314.21
deepseek-v3.2-thinking	0.27	0.40	1 674 783	17 535 380	1230	$7.46

Appendix

Table 9: Cost of running our 918 duels.

Language Model Reasoning Evaluation

See all 52 papers

Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning
Mar 2, 2026Justin WaughLLM Evaluation LLM Reasoning
Approximate Labs
MathDuels: Evaluating LLMs as Problem Posers and Solvers
Apr 23, 2026Zhiqiu Xu, Shibo Jin, Shreya Arya +1Mathematical Reasoning Benchmarks LLM Evaluation
Department of Computer and Information Science, University of Pennsylvania · Rabdos AI · Department of Mathematics, University of Pennsylvania
Recurrent Reasoning on Symbolic Puzzles with Sequence Models
Apr 19, 2026Gowrav Mannem, Chowdhury Marzia Mahjabin, Jason Chen +2Symbolic Planning Symbolic Reasoning
Algoverse AI Research · Cornell University

Mar 2, 2026J/K move · Enter open · S save

Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning

Justin Waugh

Approximate Labs

LLM Evaluation LLM Reasoning AI Agent Benchmarks Constraint Satisfaction

Open paper AskPDF arXiv

We introduce Pencil Puzzle Bench, a framework for evaluating large language model reasoning through pencil puzzles, a family of constraint-satisfaction problems closely related to NP-complete problems, with deterministic, step-level verification. From a database of 62,231 puzzles across 94 varieties with verified unique solutions, we select a benchmark of 300 puzzles spanning 20 varieties and evaluate 51 models from 11 providers in two modes: direct ask (single-shot) and agentic (multi-turn with iterative verification). A key differentiator of our benchmark is that every intermediate board state can be checked against variety-specific constraints, localizing errors to the exact rule violated, providing the infrastructure for dense, per-move reward signals for process supervision and reinforcement learning. Our evaluation reveals two distinct axes of capability: (1) reasoning effort scaling, where GPT-5.2 improves 81x from no reasoning to maximum effort; and (2) agentic iteration, where Claude Opus 4.6 rises from 0.3% to 30.0% through iterative checking, while GPT-5.2@xhigh improves from 20.2% to 56.0%. Agentic attempts span a median of 29 turns over 17 minutes, with the longest exceeding 1,221 turns and 14.3 hours - a demanding test of long-context utilization, not just reasoning.

Abstract

Language Model Reasoning Evaluation

Explore similar work

Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning

MathDuels: Evaluating LLMs as Problem Posers and Solvers

Recurrent Reasoning on Symbolic Puzzles with Sequence Models