Feedback That Backfires in Games: Do LLM Players Repeat the Move They Just Watched Fail?

Esmail Gumaan

cs.AIOct 21, 2026

Feedback That Backfires in Games: Do LLM Players Repeat the Move They Just Watched Fail?

Authors: Esmail Gumaan

Organizations: Faculty of Computer Science and Mathematics, University of Passau, Germany

Abstract

When an LLM agent's move fails, the harness writes the move and the failure message into the transcript and asks again. In tool calling, this record makes small models more likely to repeat the failed action. We test whether the same holds in games (Sokoban, Connect Four, Othello), where a move can fail by being illegal or by being legal but losing, both verified exactly. On fixed probe items built by perturbing a correct move, the record backfires: pooled over 8 instruction-tuned models (135M-1.7B) it makes the failed move more likely (corrective gain G = -1.38 nats per token), through the move's surface form, for illegal moves but not for losing ones, and not for every model. An exploratory analysis shows what sets the sign: after the record, the failed move's log-probability barely depends on how likely it was before (slope 0.18). Moves the model would not have played become more likely, and greedy decoding starts to produce them; moves it was about to play become less likely, but greedy decoding still repeats 38% of them. A description of the failure that omits the move string keeps the model's preference instead. Along the OLMo 2 1B post-training ladder the inversion is present in the base model and largely removed at the DPO stage. RAGEN's default models invert in RAGEN's own Sokoban format before any RL. A "do not repeat" instruction does nothing, a clean retry is the worst harness, and a decoder ban stops exact repeats but not illegal play. All measurements ran in float32 on a free Colab GPU, which first reproduced the original paper's number.

Figures & tables

	$G$ per move token, by game			pooled over games
Model	Sokoban	Connect Four	Othello	$G$ /tok	$\Delta_{\mathrm{copy}}$ /tok	$\Delta_{\mathrm{sem}}$ /tok	$G<0$
SmolLM2-135M	−2.35 [−2.84, −1.86]	−1.34 [−1.98, −0.71]	−1.22 [−1.47, −0.98]	−1.64 [−1.93, −1.35]	1.47 [1.16, 1.78]	0.17 [0.09, 0.24]	92%
SmolLM2-360M	−1.14 [−1.74, −0.58]	−0.95 [−1.42, −0.50]	−1.18 [−1.41, −0.98]	−1.09 [−1.36, −0.85]	1.11 [0.86, 1.37]	−0.02 [−0.10, 0.07]	74%
Qwen2.5-0.5B	−0.33 [−1.54, 0.85]	−0.20 [−1.05, 0.64]	0.39 [−0.01, 0.76]	−0.05 [−0.58, 0.44]	−0.33 [−0.86, 0.23]	0.38 [0.13, 0.65]	40%
Qwen3-0.6B	−5.24 [−6.51, −3.97]	−6.68 [−7.75, −5.52]	−4.45 [−4.95, −3.96]	−5.45 [−6.08, −4.85]	5.62 [5.02, 6.25]	−0.17 [−0.21, −0.13]	92%
Llama-3.2-1B	0.48 [−0.18, 1.06]	0.36 [−0.61, 1.28]	1.15 [0.79, 1.49]	0.66 [0.26, 1.05]	−0.52 [−0.95, −0.08]	−0.15 [−0.26, −0.02]	23%

Table 1: RQ1: corrective gain of a failure record, instruction-tuned models in their chat template, short notation, shared items (perturbation and random sources). Nats per move token;

G<0

is feedback inversion. Cells: mean and 95% cluster-bootstrap interval over positions; bold intervals exclude zero. Last column: share of items with

G<0

Figure 1: The decomposition

-G=\Delta_{\mathrm{copy}}+\Delta_{\mathrm{sem}}

by game, pooled over the RQ1 models (shared items, nats per move token, 95% intervals clustered on position). Positive bars push towards repeating the failed move.

Game	Failure	$G$ /tok	$\Delta_{\mathrm{copy}}$ /tok	$\Delta_{\mathrm{sem}}$ /tok	$P(\text{repeat})$	greedy repeat	$n$
Sokoban	illegal	−2.12 [−3.28, −1.04]	1.89 [1.11, 2.76]	0.23 [−0.17, 0.62]	0.25 $\to$ 0.39	18% $\to$ 35%	212
Sokoban	losing	0.23 [−0.26, 0.60]	0.35 [−0.11, 0.79]	−0.58 [−0.87, −0.18]	0.45 $\to$ 0.38	38% $\to$ 19%	68
Connect Four	illegal	−2.40 [−3.53, −1.33]	2.78 [1.78, 3.87]	−0.39 [−0.51, −0.26]	0.20 $\to$ 0.30	7% $\to$ 20%	188
Connect Four	losing	−0.84 [−1.48, −0.22]	0.78 [0.01, 1.53]	0.06 [−0.25, 0.35]	0.26 $\to$ 0.33	16% $\to$ 22%	92
Othello	illegal	−0.70 [−1.10, −0.34]	0.39 [0.07, 0.75]	0.31 [0.20, 0.42]	0.21 $\to$ 0.34	1% $\to$ 18%	280

Table 2: RQ1 by game and failure class, pooled over the instruction-tuned models (intervals clustered on position).

P(\text{repeat})

: probability of the failed move normalised over the four-move candidate set, before

\to

after the failure record. Greedy repeat: share of items on which greedy decoding reproduces the failed move token for token, before

\to

after.

Figure 2: RQ1 by model, game and failure class (chat template, short notation, shared items). Points: mean

G

per move token; bars: 95% cluster-bootstrap intervals over positions. Othello has no losing items.

Figure 3: Exploratory. Left: mean per-token log-probability of the failed move after each kind of record, against its value before the attempt (deciles of the latter; RQ1 models, chat template, all item sources). The dashed line is no change. Right: share of items on which greedy decoding reproduces the failed move, by its prior probability; bars are 95% cluster intervals over positions.

	$G$ /tok, completion format			mean over notations		chat
Stage	short	verbose	index	$\Delta_{\mathrm{copy}}$ /tok	$\Delta_{\mathrm{sem}}$ /tok	$G$ /tok (short)
base	−0.76 [−1.07, −0.48]	−0.20 [−0.29, −0.12]	−0.59 [−0.81, −0.38]	0.49	0.03	–
SFT	−0.56 [−0.90, −0.22]	−0.16 [−0.24, −0.08]	−0.54 [−0.86, −0.24]	0.28	0.14	−0.30 [−0.72, 0.10]
DPO	−0.16 [−0.63, 0.27]	−0.06 [−0.16, 0.03]	−0.13 [−0.53, 0.25]	−0.23	0.35	0.16 [−0.32, 0.62]
RLVR1	−0.24 [−0.71, 0.20]	−0.06 [−0.15, 0.03]	−0.15 [−0.54, 0.23]	−0.20	0.35	−0.06 [−0.49, 0.35]
Instruct	−0.26 [−0.73, 0.19]	−0.06 [−0.15, 0.03]	−0.17 [−0.58, 0.24]	−0.20	0.37	−0.16 [−0.60, 0.27]

Table 3: RQ2: the OLMo 2 1B post-training ladder. Every checkpoint is probed on the same items in one completion format; the post-trained checkpoints also in their chat template. Pooled over the three games; shared items; intervals clustered on position.

Figure 4: RQ2: the OLMo 2 1B post-training ladder in the completion format, by notation (shared items, nats per move token, 95% intervals clustered on position).

	RQ2 ladder	+ prior	RQ1 models	+ prior
notation: verbose (vs. short)	0.48 (0.25)	0.53 (0.15)	−0.16 (0.78)	−2.12 (0.34)
notation: index (vs. short)	−0.19 (0.08)	−0.28 (0.05)	−0.06 (0.20)	−0.44 (0.11)
move length in tokens (z)	−0.36 (0.11)	−0.84 (0.06)	−0.03 (0.28)	−0.03 (0.13)
token overlap with a legal move	0.86 (0.17)	−0.55 (0.11)	1.12 (0.36)	0.05 (0.15)
prior log-prob. of the move, per token		0.81 (0.02)		0.92 (0.01)
losing (vs. illegal)		−0.37 (0.24)		−0.17 (0.13)

Table 4: Mechanism test: mixed-effects regression of

G

per move token on notation, move length and token overlap, with random effects for position and model (shared items). Coefficients in nats per token with standard errors; bold:

p<0.05

. Positive coefficients mean less inversion. The “+ prior” columns add the failed move’s log-probability before the attempt, which also enters

G

itself (see text).

Model	Failure	$G$	$\Delta_{\mathrm{copy}}$	$\Delta_{\mathrm{sem}}$	$G$ , RAGEN’s own feedback	positions
Qwen2.5-0.5B	illegal (wall, box, unknown)	−2.08 [−3.24, −0.97]	2.05 [0.75, 3.36]	0.03 [−0.19, 0.28]	−2.39 [−3.75, −1.02]	45
Qwen2.5-0.5B	deadlock push	0.63 [−0.26, 1.47]	−1.75 [−2.97, −0.49]	1.12 [0.67, 1.56]	–	15
Qwen2.5-1.5B	illegal (wall, box, unknown)	−3.82 [−5.54, −2.22]	3.87 [2.29, 5.54]	−0.05 [−0.13, 0.03]	−3.89 [−5.52, −2.33]	45
Qwen2.5-1.5B	deadlock push	0.50 [0.29, 0.75]	−0.13 [−0.38, 0.14]	−0.37 [−0.59, −0.16]	–	15

Table 5: RQ3: Sokoban in RAGEN’s own prompt format (SimpleSokoban, no-think, one action per turn), RAGEN’s two default models before any training. A RAGEN action is one token, so these are also per-token values. RAGEN’s own feedback is its invalid-action warning, or nothing after a blocked move; it is defined for illegal moves only.

Variant	Sokoban	Connect Four	Othello	pooled
abstract description	−1.18 [−1.65, −0.74]	−0.25 [−0.68, 0.17]	−1.05 [−1.38, −0.72]	−0.83 [−1.08, −0.58]
bare failure marker	−0.95 [−1.45, −0.48]	−0.21 [−0.61, 0.18]	−0.99 [−1.33, −0.66]	−0.72 [−0.97, −0.47]
+ “do not repeat”	0.11 [0.05, 0.17]	0.04 [−0.02, 0.10]	−0.07 [−0.10, −0.04]	0.03 [−0.00, 0.06]
message quotes the move	0.49 [0.35, 0.63]	0.75 [0.67, 0.84]	1.09 [0.96, 1.22]	0.78 [0.70, 0.86]
the move failed twice	0.67 [0.53, 0.82]	1.71 [1.56, 1.85]	2.67 [2.48, 2.85]	1.68 [1.51, 1.86]
a different move failed	−1.10 [−1.55, −0.68]	−0.45 [−0.88, −0.02]	−0.77 [−1.11, −0.45]	−0.77 [−1.02, −0.54]

Table 6: RQ4 in the probe: paired change in

\log P(a^{\times})

, nats, of each harness variant relative to the standard verbatim record, item by item (positive = repetition more likely). Pooled over the remedy-study models; intervals clustered on position.

Figure 5: RQ4 in the probe. Left: change in

\log P(a^{\times})

of each harness variant against the verbatim record, item by item, pooled over the remedy-study models, by game. Right: for every RQ1 model, the gain under the verbatim record and under the abstract description (nats; shared items). 95% intervals clustered on position.

Figure 6: Free play under each harness (greedy decoding, random opponent). Points: mean over episodes; bars: 95% bootstrap intervals over episodes.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hugging Face id	Revision	Formats probed
SmolLM2-135M	HuggingFaceTB/SmolLM2-135M-Instruct	12fd25f773	chat
SmolLM2-360M	HuggingFaceTB/SmolLM2-360M-Instruct	a10cc1512e	chat
SmolLM2-1.7B	HuggingFaceTB/SmolLM2-1.7B-Instruct	31b70e2e86	chat
Qwen2.5-0.5B	Qwen/Qwen2.5-0.5B-Instruct	7ae557604a	chat, ragen
Qwen2.5-1.5B	Qwen/Qwen2.5-1.5B-Instruct	989aa7980e	chat, ragen
Qwen3-0.6B	Qwen/Qwen3-0.6B	c1899de289	chat

Appendix

Table 7: Checkpoints and pinned revisions (first ten hex digits).

Condition	What the transcript records after the successful prefix
pre	position s before the attempt (baseline; also the clean-retry context)
fail	attempt + failure message + s (standard harness, verbatim)
succ	attempt + counterfactual success message + s
neut	attempt + valence-free message + s
abstract	runtime-generated failure description + s, no verbatim move
fail_instr	as fail, plus a do-not-repeat instruction in the system prompt

Appendix

Table 8: Every probe condition. All conditions of an item share the system prompt, goal, successful prefix and position byte for byte; post-attempt conditions re-show the same position after their message.

Game	Failure type	Failure message ( fail )	Gloss in the abstract description
connect4	column_full	Illegal move: that column is already full.	the chosen column was already full
connect4	off_board	Illegal move: there is no such column.	the chosen location is not on the board
connect4	losing_move	That move lost: the opponent completes four in a row on the next turn. The position has been restored; choose again.	it let the opponent win on the next turn
othello	occupied	Illegal move: that square is already occupied.	the chosen square was already occupied
othello	no_flip	Illegal move: a disc there would not flip any of the opponent’s discs.	a disc there would not flip any discs
othello	off_board	Illegal move: that square is not on the board.	the chosen location is not on the board

Appendix

Table 9: Observation messages (version 2). Valence-free: Move received. ; counterfactual success: Move accepted. ; abstract description: [{ordinal} attempt failed: {gloss}; that move is not repeatable] ; bare marker: [{ordinal} attempt failed] ; fail_echo appends Your move was: {move} ; instruction: “Important: some of your earlier moves failed. Do not repeat a move that has already failed; choose a different move.”.

Game	Notation	Example	Invalid example	Rules text
connect4	short	4	0	Columns are numbered 1 to 7 from left to right. Reply with the column number only, for example: 4
connect4	verbose	drop a disc into column 4	drop a disc into column 0	Columns are numbered 1 to 7 from left to right. Reply with a phrase of the form: drop a disc into column 4
connect4	index	3	-1	Columns are indexed 0 to 6 from left to right. Reply with the column index only, for example: 3
othello	short	d3	a9	Squares are named by column letter (a-h, left to right) and row number (1-8, top to bottom). Reply with the square only, for example: d3
othello	verbose	place a disc on column d, row 3	place a disc on column a, row 9	Columns are lettered a-h from left to right and rows numbered 1-8 from top to bottom. Reply with a phrase of the form: place a disc on column d, row 3
othello	index	19	64	Squares are indexed 00 to 63 row by row: index = 8 x (row - 1) + column position, with a=0 … h=7. Reply with the two-digit index only, for example: 19

Appendix

Table 10: Move notations. The RAGEN word form is used only in the RAGEN format.

Game	Failure type	Decision points
connect4	column_full	22
connect4	losing_move	29
connect4	off_board	29
othello	no_flip	27
othello	occupied	31
othello	off_board	22

Appendix

Table 11: Item pools (seed 20261011; one decision point per position, each rendered in every notation). Sources: connect4: perturbation 60, perturbation+random 3, random 17; othello: perturbation 60, perturbation+random 1, random 19; sokoban: perturbation 80, perturbation+random 7, random 13.

Model	Game	Harness	won/solved	exact repeat	illegal	wasted	loop
qwen2.5-0.5b	Connect Four	verbatim	62%	96%	32%	9.6	38%
qwen2.5-0.5b	Connect Four	verbatim+instr	62%	96%	21%	6.4	25%
qwen2.5-0.5b	Connect Four	drop	62%	96%	32%	9.6	38%
qwen2.5-0.5b	Connect Four	abstract	75%	89%	22%	5.9	25%
qwen2.5-0.5b	Connect Four	verbatim+ban	62%	12%	28%	8.4	25%
qwen2.5-0.5b	Connect Four	abstract+ban	88%	0%	5%	0.5	0%

Appendix

Table 12: Free play under each harness (greedy decoding). Exact repeat: share of failed moves that repeat an earlier failed move byte for byte; illegal: share of all moves that the runtime rejected; wasted: rejected moves per episode; loop: episodes that hit the turn limit with a repeat.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 711 papers

Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
Aug 24, 2026Esmail GumaanLLM Agents LLM Agent Reliability
Faculty of Computer Science and Mathematics University of Passau, Germany
Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions
Apr 30, 2026Jan Sobotka, Mustafa O. Karabag, Ufuk TopcuImperfect-Information Games LLM Interpretability
EPFL · The University of Texas at Austin
Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures
Sep 28, 2026Dong Xu, Zhangfan Yang, Jiantao Wu +5LLM Agent Evaluation AI Agent Benchmarks
School of Artificial Intelligence, Shenzhen University · EasternDawn · School of Computer Science, University of Nottingham Ningbo

Aug 24, 2026J/K move · Enter open · S save

Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail

Esmail Gumaan

Faculty of Computer Science and Mathematics University of Passau, Germany

LLM Agents LLM Agent Reliability LLM Tool Use LLM Agent Evaluation

Open paper AskPDF arXiv

Agent harnesses record a failed tool call and its error message in the transcript and ask the model to continue, on the assumption that the error is corrective information. We measure whether it is. Defining the corrective gain of a failure record as the change in log-probability of re-emitting the action that just failed, we find the gain is negative for every instruction-tuned model we tested (6 checkpoints, 135M-1.7B, 4 families) in two environments: simulated tool calling and MBPP program repair. Normalised by action length the effect is about -1.03 nats per action token, a factor of 2.8 in the odds of each token, and holds on 90%-100% of individual items, not only on average. Over a fixed candidate set the probability of repeating the failed call rises from 0.06 to 0.54, and greedy decoding reproduces it token for token on 19% of items after the failure versus 0% before. Counterfactuals pairing the same call with a failure message, a success message, or a neutral acknowledgement separate two effects: the failed call's surface form accounts for 83% of the damage, while the semantic contribution of marking it failed is small and inconsistent in sign across environments. The problem is in the harness, not the model's grasp of error messages, and that predicts which remedies work. Replacing the verbatim call with a runtime-generated description of the failure removes 76% of the inversion at no token cost, and making previously-failed strings unreachable at the decoder acts on the same term. Two plausible remedies do not: an explicit "do not repeat" instruction leaves the measured quantity where it was, and deleting the failed attempt to retry from a clean context, the standard prescription for context contamination, is the worst harness we measured for repetition, because it restores the context that produced the failure. The study runs end to end on a CPU; all artefacts are released.

Abstract

LLM Agents

Explore similar work

Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail

Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions

Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures