Do Chess Explanations Reflect Model Decisions? Behavioral and Token-Level Tests of LLM Reasoning Faithfulness

Angelina Parfenova

cs.CLSep 5, 2026

Do Chess Explanations Reflect Model Decisions? Behavioral and Token-Level Tests of LLM Reasoning Faithfulness

Authors: Angelina Parfenova

Organizations: Lucerne University of Applied Sciences and Arts · Technical University of Munich

Abstract

Large language models can produce fluent explanations for chess moves, but plausible language does not necessarily reflect the reasoning behind a decision. We study this question in chess, where the board state is fully observable, legal actions can be enumerated, and move quality can be evaluated independently. Across 200 Lichess endgame puzzles, we test explanations using move recoverability, decoder-side controls, and token-level scoring of legal candidate moves. Unmasked explanations make generated moves easy to recover, but this advantage drops sharply after explicit move hints are removed. Under strict masking, explanations provide only small and decoder-dependent gains over the board state alone. Token-level scoring shows that explanations can nevertheless alter move preferences: random but plausible explanations from other puzzles reduce the probability of the correct move, indicating that irrelevant reasoning text is not simply ignored. We also find that recognizable endgame motifs can make generated moves easier to recover without reliably improving move correctness. Together, these results show that linguistic plausibility, consistency with a generated action, and solution correctness are distinct properties. Fluent chess explanations can influence action preferences and support a coherent move narrative while providing only limited evidence of faithful reasoning.

Figures & tables

Figure 1: Example of explanation recoverability. After masking explicit move mentions, the decoder recovers the model’s generated move ( e5e6 ) but not the reference move ( d5e4 ), showing action-specific information without solution correctness.

Figure 2: Illustration of top- $k$ evaluation in our experiments. In recoverability, a decoder predicts ranked move candidates from a masked explanation, and top- $k$ checks whether the target move appears among the first $k$ outputs. In token-level scoring, legal moves are ranked by summed token log-probability, and top- $k$ checks whether the target move appears among the $k$ highest-scoring candidates.

Prompt	Target	Top-1	Top-3
Brief ( $n=191$ )	Gen.	0.398 [.330,.466]	0.518 [.450,.592]
	Ref.	0.058 [.026,.094]	0.099 [.058,.147]
Calc. ( $n=195$ )	Gen.	0.379 [.313,.446]	0.462 [.390,.533]
	Ref.	0.056 [.026,.092]	0.103 [.062,.149]
Teach. ( $n=197$ )	Gen.	0.391 [.325,.462]	0.503 [.431,.574]
	Ref.	0.091 [.056,.132]	0.137 [.091,.188]

Table 1: Strict-mask recoverability with puzzle-bootstrap 95% intervals. Gen. targets the GPT-4.1-mini generated move; Ref. targets the Lichess solution.

Decoder	Condition	Parsed	Generated@1	Generated@3	Reference@1	Reference@3
Llama-3.3-70B	FEN only	0.510	0.047	0.060	0.000	0.000
	Explanation only	0.201	0.000	0.020	0.000	0.007
	Strict mask	0.436	0.073	0.100	0.053	0.053
	Maximal mask	0.423	0.073	0.073	0.060	0.060
	Random expl.	0.376	0.007	0.013	0.000	0.007
	Unmasked	0.879	0.680	0.740	0.113	0.147

Table 2: Decoder-side controls on the same 50-puzzle subset, pooled across prompt styles. Parsed is the rate at which the decoder returned at least one extractable move. Qwen3 uses hidden/no-reasoning mode.

Prompt	Context	R@1	R@3	G@1	G@3
Brief	FEN only	0.16	0.42	0.08	0.22
	+ maximal mask	0.14	0.36	0.12	0.30
	+ unmasked	0.18	0.36	0.50	0.52
	+ random expl.	0.06	0.26	0.06	0.20
Calculation	FEN only	0.18	0.40	0.10	0.16
	+ maximal mask	0.18	0.42	0.10	0.20

Table 3: Qwen2.5-32B all-legal scoring on 50 puzzles. R and G denote reference and generated moves. Unmasked explanations are a leakage check; random explanations come from another puzzle under the same prompt style.

Figure 3: Token-level preference and recoverability. (A) Recoverability by reference-move rank bucket. (B) Recoverability by reference-versus-top log-probability gap. Both relationships are weak and non-monotonic, showing that recoverable explanations do not necessarily correspond to stronger token-level preference for the reference move. (C) Reference-move top-1 and top-3 rates across open scoring models under scoring-only prompts. Open models place the reference move in the top three more often than top one, but rarely rank it decisively first.

Endgame type	Puzzles	Legal	Exact	Gen. rec.	Ref. rec.	Motif mention
Pawn	40	0.558	0.133	0.483	0.125	0.775
Rook	56	0.467	0.103	0.494	0.062	0.788
Knight	10	0.700	0.067	0.233	0.033	0.867
Bishop	14	0.300	0.025	0.256	0.026	0.600
Queen †	7	0.810	0.190	0.238	0.143	1.000
Queen-rook	65	0.558	0.037	0.323	0.053	0.979

Table 4: Endgame-type analysis over the three GPT-4.1-mini prompt conditions. Recoverability is top-1. † Fewer than ten puzzles; descriptive only.

Group	Seg.	Words	Coord.	Move	Spatial	Concept	Spec.
Human ctrl.	1157	8.1	0.124	0.125	0.435	0.136	0.210
Human nat.	417	17.9	0.142	0.142	0.955	0.612	0.122
LLM brief	6299	12.6	0.326	0.389	0.217	0.177	0.728
LLM calc.	17895	10.6	0.292	0.387	0.174	0.179	0.415
LLM teach.	13826	13.6	0.389	0.430	0.336	0.176	0.458

Table 5: Reasoning-style markers for human and LLM chess explanations. Human ctrl. aggregates the three controlled think-aloud transcripts; Human nat. aggregates the two public commentaries. LLM rows aggregate explanations by prompting style. Coord., Move, Spatial, and Concept report the proportion of units containing explicit coordinates, move notation, spatial/deictic board language, or tactical/strategic vocabulary. Spec. is the normalized text-only specificity score.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Reference rank	$n$	Recoverability	Mean $\Delta$ (cp)
2-3	191	0.321	11192.7
1	113	0.352	14443.0
4-5	172	0.412	12272.2
>5	112	0.297	10501.3

Appendix

Table 6: Recoverability by reference-move rank bucket (Mistral-7B token-prob scoring; brief prompt).

Margin bucket	$n$	Recoverability	Mean $\Delta$ (cp)
-5 to -2	146	0.321	10190.0
>=-0.5	136	0.351	13695.5
< -5	242	0.354	13014.5
-2 to -0.5	64	0.381	8920.5

Appendix

Table 7: Recoverability by reference-vs-top logprob margin bucket. Margin is reference logprob minus top logprob (closer to 0 = more internal uncertainty).

Prompt	Top-1	Top-3	Top-5	Mean rank	Median rank	Mean $\Delta$ (ref-top)
brief	0.215	0.540	0.820	3.46	3.00	-3.70
calc	0.200	0.535	0.800	3.52	3.00	-5.34
teaching	0.175	0.490	0.805	3.64	4.00	-5.41

Appendix

Table 8: Token-probability ablation: reference-move rank under style-conditioned scoring prompts (open-weights model).

\Delta

is reference logprob minus top-prob logprob; more negative means the reference move is less preferred.

Model	Prompt	Top-1	Top-3	Top-5	Mean rank	Median rank	Mean $\Delta$ (ref-top)
Llama-3.1-70B (4-bit)	scoring only	0.285	0.575	0.770	3.35	3.00	-2.14
Qwen2.5-32B	scoring only	0.245	0.555	0.780	3.46	3.00	-2.06

Appendix

Table 9: Reference-move ranking under scoring-only prompts for the two larger open models.

\Delta

is the reference-move log-probability minus the top-move log-probability; values closer to zero indicate a smaller preference gap.

Model	Parsed	Legal	Exact	Avg. words	Avg. specificity
GPT-4.1-mini	0.980	0.512	0.082	102.6	0.242
Llama-3.1-8B-Instant	0.740	0.038	0.007	147.3	0.239
Llama-3.3-70B-Versatile	0.478	0.068	0.005	124.8	0.162
Llama-4-Scout-17B	0.693	0.060	0.002	145.6	0.187
Kimi-K2-Instruct	0.863	0.487	0.072	97.1	0.413
Qwen3-32B	0.060	0.048	0.007	145.0	0.241

Appendix

Table 10: Hosted-model results averaged across brief, calculation, and teaching prompts on 200 puzzles. The full 18-condition table is reported in Appendix 12 .

Prompt	Legal (95% CI)	Exact (95% CI)	Median loss, legal	Mean loss, all
Brief	0.535 [.470,.605]	0.080 [.045,.120]	471	1244.8
Calculation	0.415 [.350,.480]	0.070 [.035,.110]	459	1397.8
Teaching	0.585 [.515,.655]	0.095 [.055,.140]	463	1157.7

Appendix

Table 11: GPT-4.1-mini move quality on 200 puzzles. Engine loss is clipped at 2000 centipawns. “Legal” loss excludes illegal and unparsed outputs; “all” assigns them the 2000-centipawn cap. Exact match is measured over all outputs.

Model	Prompt	$n$	Parsed	Legal	Exact	Avg $\Delta$ (cp)	Avg words	Avg spec.
gpt-4.1-mini	brief	200	0.970	0.535	0.080	11559.5	44.4	0.201
gpt-4.1-mini	calc	200	0.975	0.415	0.070	12364.2	146.1	0.321
gpt-4.1-mini	teaching	200	0.995	0.585	0.095	12304.3	117.4	0.205
llama-3.1-8b-instant	brief	200	0.960	0.015	0.005	66600.3	47.2	0.256
llama-3.1-8b-instant	calc	200	0.270	0.070	0.010	21781.6	200.7	0.231
llama-3.1-8b-instant	teaching	200	0.990	0.030	0.005	50084.5	194.1	0.230

Appendix

Table 12: Model comparison on 200 Lichess endgame puzzles. Parsed = fraction of outputs where a move could be extracted.

Figure 4: Qwen2.5-32B all-legal-move scoring under FEN-only and explanation controls. The scorer conditions on GPT-4.1-mini explanations.

Metric	Value
Mean explanation similarity	0.203
Generated move changed	0.959
Top-prob move changed	1.000
Recoverability predicted move changed	0.816
Recoverability match changed	0.000

Appendix

Table 13: Counterfactual sensitivity summary on 25 puzzles (Mistral-7B). Values are fraction of variants where a change occurred, except explanation similarity (SequenceMatcher ratio).

Figure 5: Counterfactual sensitivity summary for 25 puzzles.

Model	Layers	Mean logprob (first)	Mean logprob (last)
Llama-3.1-70B	0-80	-47.59	-21.46
Qwen2.5-32B	0-64	-48.26	-17.64

Appendix

Table 14: Logit-lens summary (20 puzzles). Mean reference-move logprob (Lichess solution) at the first and last layer.

Figure 6: Exploratory logit-lens reference-move logprob across layers for 20 puzzles.

Figure 7: Explanation length vs. recoverability under strict masking.

Mask	Audited	Residual leakage
Light	60	0.567
Strict	60	0.567
Coordinates	60	0.567
Maximal	60	0.000

Appendix

Table 15: Automatic masking audit on 60 sampled explanations. The residual detector flags UCI/SAN, coordinates, piece-square phrases, and promotion language. Maximal masking removes all audited leakage classes.

Decoder	Prompt	Parsed	Generated@1	Generated@3	Reference@1	Reference@3
GPT-4.1-mini	Brief	0.955 [0.925,0.980]	0.392 [0.325,0.464]	0.510 [0.438,0.577]	0.055 [0.025,0.090]	0.095 [0.060,0.135]
GPT-4.1-mini	Calculation	0.975 [0.950,0.995]	0.379 [0.313,0.446]	0.462 [0.390,0.533]	0.055 [0.025,0.085]	0.100 [0.060,0.145]
GPT-4.1-mini	Teaching	0.985 [0.965,1.000]	0.387 [0.322,0.452]	0.497 [0.427,0.568]	0.090 [0.055,0.135]	0.135 [0.090,0.185]
Llama-3.3-70B	Brief	0.995 [0.985,1.000]	0.067 [0.036,0.103]	0.129 [0.088,0.180]	0.010 [0.000,0.025]	0.035 [0.010,0.065]
Llama-3.3-70B	Calculation	1.000 [1.000,1.000]	0.077 [0.041,0.118]	0.128 [0.082,0.179]	0.005 [0.000,0.015]	0.030 [0.010,0.055]
Llama-3.3-70B	Teaching	1.000 [1.000,1.000]	0.075 [0.040,0.116]	0.126 [0.080,0.176]	0.025 [0.005,0.050]	0.045 [0.020,0.075]

Appendix

Table 16: Strict-mask move recoverability on 200 puzzles. Unparsed decoder outputs count as failures; brackets show puzzle-bootstrap 95% confidence intervals.

Decoder	Prompt	Parsed	Generated@1	Generated@3	Reference@1	Reference@3
GPT-4.1-mini	Brief	1.000 [1.000,1.000]	0.398 [0.330,0.466]	0.518 [0.450,0.592]	0.058 [0.026,0.089]	0.099 [0.058,0.147]
GPT-4.1-mini	Calculation	1.000 [1.000,1.000]	0.379 [0.313,0.446]	0.462 [0.390,0.533]	0.056 [0.026,0.087]	0.103 [0.062,0.149]
GPT-4.1-mini	Teaching	1.000 [1.000,1.000]	0.391 [0.325,0.462]	0.503 [0.431,0.574]	0.091 [0.056,0.137]	0.137 [0.091,0.188]
Llama-3.3-70B	Brief	1.000 [1.000,1.000]	0.067 [0.031,0.104]	0.130 [0.083,0.181]	0.010 [0.000,0.025]	0.035 [0.015,0.065]
Llama-3.3-70B	Calculation	1.000 [1.000,1.000]	0.077 [0.041,0.118]	0.128 [0.082,0.179]	0.005 [0.000,0.015]	0.030 [0.010,0.055]
Llama-3.3-70B	Teaching	1.000 [1.000,1.000]	0.075 [0.040,0.116]	0.126 [0.080,0.176]	0.025 [0.005,0.050]	0.045 [0.020,0.075]

Appendix

Table 17: Parsed-only strict-mask recoverability on 200 puzzles. Rates are conditional on extracting at least one UCI move.

Decoder	Condition	Parsed $n$	Generated@1	Generated@3	Reference@1	Reference@3
Llama-3.3-70B	FEN only	76	0.092	0.118	0.000	0.000
	Explanation only	30	0.000	0.100	0.000	0.033
	Strict mask	65	0.169	0.231	0.123	0.123
	Maximal mask	63	0.175	0.175	0.143	0.143
	Random expl.	56	0.018	0.036	0.000	0.018
	Unmasked	131	0.779	0.847	0.130	0.168

Appendix

Table 18: Parsed-only decoder controls on the 50-puzzle subset, pooled across prompt styles. Rates condition on at least one extracted UCI move; end-to-end rates appear in Table 2 .

Figure 8: Decoder-side top-1 recoverability controls on 50 puzzles, pooled across prompt styles. Unparsed outputs count as failures.

Type	Expected motif	$n$	Exact	Gen. rec.
Pawn	absent	27	0.111	0.259
	present	93	0.140	0.548
Rook	absent	35	0.143	0.400
	present	130	0.092	0.520
Bishop	absent	16	0.063	0.250
	present	24	0.000	0.261

Appendix

Table 19: Move correctness and generated-move recoverability conditional on mentioning an expected endgame motif. Counts are explanation rows, not unique puzzles.

Figure 9: Exact accuracy, recoverability, and expected-motif mention rates by endgame category and prompt style. Small categories should be interpreted cautiously.

Figure 10: Rows are labeled endgame types; columns are motif vocabularies mentioned in explanations.

Outcome	Standardized predictor	Coefficient	Bootstrap 95% interval
Correct move	Reference-move rank	$-.978$	$[-1.552,-.547]$
Correct move	Square mentions	$-.616$	$[-1.279,-.040]$
Generated recovery	Legal generated move	$.403$	$[.136,.652]$
Generated recovery	Clipped engine loss	$-.371$	$[-.715,-.169]$
Reference recovery	Generated move correct	$1.109$	$[.835,1.585]$
Generated recovery	Expected motif mentioned	$.128$	$[-.102,.405]$

Appendix

Table 20: Selected coefficients from exploratory logistic classifiers with puzzle-clustered bootstrap intervals. Models also include length, specificity, prompt, rating, material, theme, endgame type, and endgame-motif interactions. Coefficients are descriptive rather than causal.

Feature	LLM	Human	$\Delta$	$q$
Adverbs	1.61	7.29	-0.92	$<.001$
Pronouns	3.19	7.22	-0.79	$<.001$
Hedges	1.54	4.30	-0.67	$<.001$
Tokens/sentence	36.04	16.43	0.77	$<.001$
tree depth	6.80	3.58	0.78	$<.001$
Determiners	6.67	3.66	0.74	$<.001$

Appendix

Table 21: Exploratory ELFEN comparison of 50-word LLM explanation chunks and five human chess-reasoning transcript sources.

\Delta

is Cliff’s delta, positive when the feature is larger in LLM text.

q

is Benjamini-Hochberg corrected.

Figure 11: Largest ELFEN feature differences. Positive Cliff’s delta denotes a higher feature value in LLM explanations; negative values denote a higher value in human transcripts.

Group	$n$	Correct	Words	Squares	Cond.	Line depth	Spec.
Human pre-solution	3	0.50	560.0	35.0	27.3	5.3	0.299
Human post-solution	3	1.00	209.7	21.7	6.7	7.7	0.606
LLM brief	4	0.50	47.5	1.5	0.0	1.3	0.152
LLM calculation	4	0.00	128.3	7.5	1.5	3.5	0.290
LLM teaching	4	0.00	126.0	6.8	0.8	2.3	0.226

Appendix

Table 22: Pilot comparison of human think-aloud reasoning and LLM explanations on book pawn endings. Human pre-solution rows are spontaneous solving attempts; post-solution rows are explanations after seeing the book line.

Source and outcome	Selected excerpt
Puzzle 2, human; incorrect final answer	“If we were to play king to d6, then we might have h5, c5, h4, c6 …and then both make a queen. This might actually be a draw, so this shouldn’t be right. Okay, so we can’t run immediately.”
Puzzle 1/2, Book-B participant; concrete but wrong candidate	“The first movement is to move the pawn from c3 into c4. …I prefer to keep the king at the corner because I want to bring the white king into also corner.”
Puzzle 1/2, Book-C participant; schema-level plan	“The goal of white can really probably win by promotion …black wants to do is kind of just be in front of this pawn.”
Puzzle 2, model; Kc7 , incorrect	“White plays c7, pushing the pawn closer to promotion. …White’s king on c6 supports the pawn on c7, which is one step away from queening. …Therefore, c6c7 is the winning move.”
Puzzle 4, human; unresolved	“We can’t stop this pawn from queening, I don’t think so. And if a queen lands here then that also attacks our promotion square. So we might need to set it up so that our king protects that.”
Puzzle 4, model; h4 , reference move	“White should push the pawn to h4 to start advancing it towards promotion. Black’s king is too far to stop the pawn …This move initiates a winning plan for White.”

Appendix

Table 23: Qualitative examples from the controlled pawn-ending pilot. Human rows are think-aloud excerpts from the participants; model rows are generated explanations. Puzzle 4 is a drawing study: the model selects the reference move but incorrectly describes it as winning. Excerpts are shortened for space.

Faithfulness of Language Model Explanations

Faithfulness of language model explanations measures whether stated reasons reflect the evidence and processes driving model outputs.

See all 106 papers

Language Models that Play Chess and Explain Their Moves
Oct 2, 2026Adithya Bhaskar, Jeffrey Cheng, Danqi ChenChess
Princeton Language and Intelligence, Princeton University
Explaining Chess and Go in Natural Language
Oct 16, 2026Ori Press, Alexander GolubevConcept-Based Explanations Explainability Evaluation
Nebius AI
Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models
May 17, 2026Ethan TangLLM Evaluation Language Modeling
School of Computing and Augmented Intelligence · Arizona State University · Tempe, AZ 85281

Oct 2, 2026J/K move · Enter open · S save

Language Models that Play Chess and Explain Their Moves

Adithya Bhaskar, Jeffrey Cheng, Danqi Chen

Princeton Language and Intelligence, Princeton University

Chess

Open paper AskPDF arXiv

Modern chess engines are silent experts: they play at a superhuman level, but do not offer explanations for their play. On the other hand, language models (LMs) can generate plausible-sounding explanations, but their weak playing strength limits the utility of their explanations. We introduce Queen, a 4B-parameter chess-language model that can explain its moves and plans while playing at the level of a typical Grandmaster. Our novel framework enables domain-specific reasoning through complementary components: an encoder-decoder architecture and an iterative distillation algorithm. This architecture integrates a silent expert chess encoder with an instruction-tuned LM through cross-attention, which we train via a question-answering curriculum to extract chess concepts from the encoder's representations. Building on this domain-adapted model, we iteratively improve its explanations with a natural-language analog of the Bellman update: the model analyzes the positions after its top candidate moves and consolidates them into an explanation of the current position, which is then distilled back into the model. Over seven iterations, our model gains over 900 Elo points (1782 to 2697), substantially surpassing all frontier models on both playing strength and puzzle accuracy, despite containing three orders of magnitude fewer parameters. Furthermore, LM-based evaluations show that our explanations are fluent and approach GPT-5.6-Sol (high) in coherence. The generality of our architecture and training procedure suggests a recipe for applying language models to domains where silent expert encoders are available, like games, robotics, and computer use.

Abstract

Faithfulness of Language Model Explanations

Explore similar work

Language Models that Play Chess and Explain Their Moves

Explaining Chess and Go in Natural Language

Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models