Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

Nan Li

cs.CLAug 28, 2026

Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

Authors: Nan Li

Organizations: Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

Abstract

Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.

Figures & tables

Dialogue state	Model action	Local failure
After crane : red c/r/n ; yellow a ; green e	guess: crane	repeats guess
(same)	guess: abid	wrong length
(same)	guess: cable	uses red c

Table 1: Simplified Wordle failure states. The model output can be syntactically close to valid while violating a constraint entailed by the immediately preceding feedback. A real transcript exhibiting this pattern is reproduced in Table 7 .

Phase	Objective	LR / batch / len	Clem
—	Qwen3.5-2B	—	13.05
A. Acquire	broad SFT	2e-4 / 32 / 2048	43.85
B. Repair	turn-DPO: rpt + len	5e-6 / 16 / 1024	46.83
B. Repair	turn-DPO: inv + dup	5e-6 / 16 / 1024	45.52
C. Refine	branch-DPO	5e-6 / 16 / 1024	48.52
D. Preserve	delta scaling, $s{=}0.85$	no gradient	50.43 †

Table 2: Pipeline trajectory on the pinned public-development set (dataset 557d8caf , clembench ed39486 ). Batch denotes effective batch size; all Clem values are the full 14-game suite. Phases B and C train on Wordle data only. Both Phase B passes use

\beta{=}0.1

; Phase C uses

\beta{=}0.2

. All trainable phases use LoRA on Qwen3.5-2B , AdamW, bf16, and one epoch; Phase C branches the model (factor 2, rounds 1–3,

T{=}0.7

) against a frozen reference. rpt/len = repeated guesses and bad length; inv/dup = invalid words and duplicated answer fields. † Mean of two runs (50.82 and 50.03). Per-game and per-phase Wordle detail: Tables 13 and 14 .

System	Base	Public	Stat	Closed ID	Closed OOD	$\Delta$ ID	$\Delta$ OOD
Baseline	Qwen3.5-2B	10.67	44.24	13.41	3.72	—	—
	Qwen3.5-4B	26.66	51.33	34.02	17.99	—	—
	Llama-3.1-8B	19.53	45.59	31.24	22.62	—	—
	Qwen3.5-9B	31.91	53.90	41.12	24.91	—	—
	Qwen3.5-27B	60.30	65.05	64.34	43.51	—	—
LLP llp-final	Qwen3.5-9B	53.39	57.80	59.23	22.09	$+18.11$	$-2.82$

Table 3: Official final evaluation results of the LM Playschool Challenge ( https://github.com/lm-playpen/lm-playschool-2026-final-results , snapshot 2dd5a533 ). Public is the public interactive clemscore; Stat is the static benchmark score; Closed ID and OOD are the organizer’s hidden in-domain and out-of-domain clemscores.

\Delta

ID and

\Delta

OOD are differences against each system’s own matching base. Submissions are sorted by public clemscore.

	Played	Quality	Clem
	(base $\to$ ours)	(base $\to$ ours)	(base $\to$ ours)
Public	27.22 $\to$ 78.63	39.20 $\to$ 49.50	10.67 $\to$ 38.92
Closed ID	47.32 $\to$ 82.62	28.33 $\to$ 49.83	13.41 $\to$ 41.17
Closed OOD	35.19 $\to$ 33.08	10.56 $\to$ 23.81	3.72 $\to$ 7.88

Table 4: Played and Quality decomposition of the official final evaluation. OOD Played falls below the untrained base, indicating that the OOD clemscore gain is driven entirely by higher conditional Quality on the episodes that remain in play.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Ph.	Data	Objective	LR / sched. / warmup	Batch / length / epochs	Other
A	9,522 rows	SFT, AdamW	2e-4 / cosine / 0.03	micro 8 $\times$ accum 4 $=$ 32; 2,048; 1	LoRA $r$ 16/ $\alpha$ 32/dropout 0.05/all-linear; bf16; grad. ckpt. on; seed 42
B1	1,905 pairs	DPO, $\beta{=}0.1$	5e-6 / cosine / 0.03	micro 1 $\times$ accum 16 $=$ 16; 1,024; 1	bf16; grad. ckpt. off; separate frozen reference
B2	1,906 pairs	DPO, $\beta{=}0.1$	5e-6 / cosine / 0.03	micro 1 $\times$ accum 16 $=$ 16; 1,024; 1	bf16; grad. ckpt. off; separate frozen reference
C	77 pairs	DPO, $\beta{=}0.2$	5e-6 / cosine / 0.03	micro 1 $\times$ accum 16 $=$ 16; 1,024; 1	bf16; grad. ckpt. on; separate frozen reference; factor 2, rounds 1–3, $T{=}0.7$ , 256 new tokens
D	—	delta scaling $s{=}0.85$	no optimizer	—	multiply each lora_B by 0.85, then merge in fp32

Appendix

Table 5: Complete training configuration for the four effective phases. All trainable phases continue the same LoRA adapter on Qwen3.5-2B in bf16.

Game	Success	Capped	Train
AdventureGame	429	429	403
Codenames	2,402	700	669
DoND	1,256	700	663
GuessWhat	2,170	700	662
HotAirBalloon	1,102	700	658
ImageGame	1,018	700	662

Appendix

Table 6: Broad-SFT per-game counts at the reconstructed training-time snapshot ( cf23af92 ). Success counts successful episodes; Capped applies the 700-per-game limit; Train is the count after the 95/5 split.

Role	Content
Model	guess: slate
GM	guess_feedback: s<red> l<red> a<red> t<red> e<red>
Model	guess: crane
GM	guess_feedback: c<red> r<green> a<red> n<green> e<red>
Model	The feedback shows that ‘r’ and ‘n’ are correct in their positions, while ‘c’, ‘a’, and ‘e’ are incorrect. … guess: ringer
GM	Error: The length of the guessed word is not 5. game_result = ABORT

Appendix

Table 7: A real Wordle failure from the broad-SFT checkpoint on the pinned public-development set (medium-frequency instance 14, target word grind ). The model reasons correctly about letter positions and then produces a six-letter guess. This is a diagnostic example, not a Phase B training row: Phase B pairs are built from clean successful episodes.

Row	Configuration	Clem	Stat
A	broad SFT only	44.09	39.88
B	A $+$ turn-DPO pass 1	45.58	39.62
C	A $+$ turn-DPO pass 1 (repeat)	45.94	40.96
D	C $+$ turn-DPO pass 2	47.28	40.31
E	D $+$ branch-DPO	45.21	41.54

Appendix

Table 8: Controlled single-host ablation. Rows B and C independently apply the same turn-DPO pass-1 configuration to the same effective parent; their difference (

+0.36

clem,

+1.34

stat) estimates repeat-run variation. Adding branch-DPO to Row D reduces clem by 2.07 and raises stat by 1.23 (Row E). Row A’s adapter is byte-identical to the broad-SFT adapter that scored 43.85 on the pinned rail; the 0.24-point difference is evaluation variation measured in § A.4 . This controlled campaign and Table 2 use separate evaluation rails.

Arm	Pairs	Clem	vs. 50.43
Wordle-only solver	66	48.39	$-2.0$
All failures	675	44.15	$-6.3$
Wordle high-dose	66	44.99	$-5.4$
Conservative SVF	500	48.92	$-1.5$
SVF, then $s{=}0.95$	—	48.08	$-2.4$
SVF, then $s{=}0.90$	—	46.68	$-3.8$

Appendix

Table 9: Ten evaluated late-continuation endpoints. Eight are DPO runs (

\beta{=}0.20

, max length 1,024, micro-batch 1, separate frozen reference); two are training-free rescalings of a preceding DPO arm (marked —). SVF denotes success-versus-failure pairs. Of the eight DPO runs, five start from the selected

s{=}0.85

adapter and three start from other parents (Phase A or unscaled Phase C). The 50.43 comparator is the mean of two evaluations of the selected adapter and serves as the contextual reference for this campaign.

	Baseline		Ours
Benchmark	Played	Quality	Played	Quality
BBH	100.00	0.00	100.00	30.47
CLadder	100.00	48.51	100.00	46.53
EQ-Bench	97.06	64.78	94.12	56.43
IFEval	100.00	68.52	100.00	51.85
MMLU-Pro	100.00	40.71	100.00	38.05

Appendix

Table 10: Public static components of the official evaluation. Applying Eq. 1 gives statscore 44.24 for the baseline and 44.14 for our submission. The aggregate is approximately preserved, but the components are redistributed: BBH Quality rises by 30.47 while IFEval falls by 16.67 and EQ-Bench Quality by 8.35.

Game	Played	Quality	Eff.	Played	Quality	Eff.	$\Delta$ Eff.
	Qwen3.5-2B baseline			playornotplay (ours)
TMW GraphReasoning	0.00	0.00	0.00	100.00	70.99	70.99	$+70.99$
ImageGame	80.00	12.03	9.62	97.50	71.46	69.67	$+60.05$
AdventureGame	0.00	0.00	0.00	100.00	58.33	58.33	$+58.33$
TextMapWorld	0.00	0.00	0.00	100.00	57.91	57.91	$+57.91$
TMW SpecificRoom	0.00	0.00	0.00	100.00	53.33	53.33	$+53.33$

Appendix

Table 11: Official hidden in-domain per-game decomposition, 21 scored configurations, sorted by change in effective score. Eff. is the displayed diagnostic

\mathrm{Played}\times\mathrm{Quality}/100

; these values are not averaged to reconstruct the suite clemscore (Eq. 1 ).

Game	Played	Quality	Eff.	Played	Quality	Eff.	$\Delta$ Eff.
	Qwen3.5-2B baseline			playornotplay (ours)
Wordle-crazy w/ Critic	6.67	0.00	0.00	36.67	90.91	33.34	$+33.34$
Wordle-crazy w/ Clue	6.67	50.00	3.33	36.67	100.00	36.67	$+33.34$
TA Frozen Lake	50.00	26.11	13.05	40.00	55.56	22.22	$+9.17$
TA Mastermind	0.00	0.00	0.00	15.00	33.33	5.00	$+5.00$
Wordle-crazy	3.33	0.00	0.00	80.00	4.17	3.34	$+3.34$

Appendix

Table 12: Official hidden out-of-domain per-game decomposition, 13 scored configurations from eight game families, sorted by change in effective score. The three Wordle-crazy configurations contribute 82.5% of total positive movement. Columns are as in Table 11 .

Game	$n$	Base	Broad SFT	Branch-DPO	Final	$\Delta$ Base $\to$ SFT	$\Delta$ SFT $\to$ Branch	$\Delta$ Branch $\to$ Final
AdventureGame	5	0.0	33.3	40.0	33.3	$+33.3$	$+6.7$	$-6.7$
Codenames	13	0.0	0.0	7.7	7.7	$0.0$	$+7.7$	$0.0$
GuessWhat	6	0.0	50.0	33.3	50.0	$+50.0$	$-16.7$	$+16.7$
ImageGame	4	35.8	81.2	89.8	85.2	$+45.4$	$+8.6$	$-4.6$
MatchIt ASCII	4	75.0	50.0	75.0	75.0	$-25.0$	$+25.0$	$0.0$
PrivateShared	4	0.0	35.1	21.3	18.3	$+35.1$	$-13.8$	$-3.0$

Appendix

Table 13: Matched per-game comparison of four checkpoints on the pinned public-development set (single host, single harness, dataset 557d8caf , clembench ed39486 ). The per-game score is the derived

P_{g}Q_{g}/100

; it is displayed as 0 when Played

=0

, although Quality itself is undefined there (Eq. 1 ). The overall clemscore is the product of macro-averaged Played and macro-averaged Quality over played games, not the mean of the per-game column, so per-game deltas do not sum to the overall delta. The Broad SFT column is the Phase A checkpoint and the Branch-DPO column is the Phase C checkpoint (Table 2 ).

\Delta

Branch

\to

Final includes gradient-free delta scaling and the fp32 merge, not additional training. Each game has 3–13 episodes; the changes are descriptive.

Checkpoint	Training scope	Suite Clem	W Played	W Quality	Rep./played ep.	Viol./ep.	Req. success
Broad SFT weights ∗	all games	43.85	66.7	0	3.00	0.33	88.9
Phase B pass 1	Wordle	46.83	0	—	—	1.00	63.9
Phase B pass 2	Wordle	45.52	66.7	0	0.00	0.33	91.7
Phase C	Wordle	48.52	100	0	0.00	0.00	100.0
$+$ scaling $s{=}.85$	none (no gradient)	50.82 / 50.03	0	—	—	1.00	69.4
Shipped fp32 merge	none (merge only)	48.70 / 49.10	0	—	—	1.00	69.4

Appendix

Table 14: Plain-Wordle behavior after each phase on the three public validation episodes (development evaluations; pinned dataset 557d8caf ). ∗ Three greedy evaluations of byte-identical weights span 43.35–44.09; the difference is evaluation noise (§ A.4 ). Suite Clem is the full 14-game clemscore; for the scaled and shipped artifacts we list the two runs of each. W Quality is undefined (—) when no episode is played. Rep./played ep. is the mean clembench Guess Repetitions over played episodes; Viol./ep. and Req. success are the mean Violated Request Count and Request Success Ratio (%) over all three episodes.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 564 papers

First Make It Playable, Then Make It Good: Staged Interaction Learning for Small Dialogue-Game Agents
Aug 27, 2026Syed Mahbubul Huq, Pranava MadhyasthaLLM Agent Training Game-Playing Agents
City, University of London · The Alan Turing Institute
Investigating Knowledge Transfer Across Interactive Dialogue Games
Aug 25, 2026Filippo Momentè, Mir Nafis Sharear Shopnil, Andrea de Varda +5
University of Trento · Technovative Solutions Ltd · Massachusetts Institute of Technology +2
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford

Aug 27, 2026J/K move · Enter open · S save

First Make It Playable, Then Make It Good: Staged Interaction Learning for Small Dialogue-Game Agents

Syed Mahbubul Huq, Pranava Madhyastha

City, University of London · The Alan Turing Institute

LLM Agent Training Game-Playing Agents

Open paper AskPDF arXiv

We present Qwen-GuidePlay-2B, a 2B-parameter language model for dialogue-game interaction. We fine-tune Qwen3.5-2B using three steps: a) SFT on only successful game trajectories from Playpen, b) weighted turn-level SFT, and c) teacher-guided SFT. The teacher model (which is a larger model) is only used to fix formatting and evaluate examples, but does not create new gold actions. Our final model scores 57.12 clemscore and 42.68 statscore on the public Playpen validation. In the officially released challenge results, our model obtains the second-highest Playpen clemscore delta among submitted systems (which is approximately +36 over its base model). Our findings suggest that imitating full trajectories helps with playability, while turn-level and teacher-guided training usually improve decision-making and increase the overall score. Alternative procedurally heavy approaches like replay-repair and hard-example mining did not help, which suggests that small models are performant simply by using careful curation strategies rather than aggressive changes. We make available both the model and the code for reproducibility.

Abstract

LLM Agent Training

Explore similar work

First Make It Playable, Then Make It Good: Staged Interaction Learning for Small Dialogue-Game Agents

Investigating Knowledge Transfer Across Interactive Dialogue Games

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap