Investigating Knowledge Transfer Across Interactive Dialogue Games

Filippo Momentè; Mir Nafis Sharear Shopnil; Andrea de Varda; Pavel Merinov; Raffaella Bernardi; Oswald Lanz; Alessandro Suglia; Alessandro Torcinovich

cs.CLAug 25, 2026

Investigating Knowledge Transfer Across Interactive Dialogue Games

Authors: Filippo Momentè, Mir Nafis Sharear Shopnil, Andrea de Varda, Pavel Merinov, Raffaella Bernardi, Oswald Lanz, Alessandro Suglia, Alessandro Torcinovich

Organizations: University of Trento · Technovative Solutions Ltd · Massachusetts Institute of Technology · Free University of Bozen Bolzano ·

Abstract

Dialogue games represent a challenging setting where complex cognitive skills are required to accomplish tasks while coordinating with other players. Considering that language represents an interface for both understanding the game rules and executing actions, it is reasonable to assume that training on a specific language game will enhance specific capabilities that might be relevant for other tasks as well. Motivated by this rationale, in this paper, we investigate how knowledge transfers across different dialogue games. We study transferability by finetuning LLM models on games from the clembench suite (Chalamalasetti et al., 2023) and performing two analyses: i) we derive a task-transferability graph using a binary integer optimization program from Zamir et al. (2018), using task performance as the main metric; and ii) we compute task vectors (Ilharco et al., 2022) for each game to study similarities across finetuned models and their task transferability. In our first analysis, we find that some games benefit more from transfer than finetuning, and that the visuospatial family (e.g., exploration games) transfers best. With our task vector analysis instead, we find that similarity-based approaches capture game-role relationships but almost no transferability patterns, suggesting that more complex metrics are required.

Figures & tables

		Target
		adventuregame	codenames_P1	codenames_P2	guesswhat_P1	guesswhat_P2	imagegame_P1	imagegame_P2	matchit_ascii_P1	referencegame_P1	referencegame_P2	taboo_P1	taboo_P2	wordle_withclue	wordle_withcritic_P1	wordle_withcritic_P2
Source	adventuregame	0.50	0.29	0.25	0.10	0.88	0.07	0.10	0.20	0.05
	codenames_P1		0.27	0.30	0.10	0.77	0.01	0.12	0.20							0.01
	codenames_P2		0.10	0.22		0.78	0.04	0.09	0.10			0.02
	guesswhat_P1			0.27		0.82		0.15		0.05		0.05	0.02		0.08
	guesswhat_P2		0.25	0.50		0.90		0.03	0.05	0.10		0.10

Table 1: Filtered transfer modeling matrix. Tasks are divided into visuospatial , and verbal ; P1 / P2 denotes player roles, while empty cells correspond to specialists underperforming the baseline.

Panel of Figure 1 — Figure 1: Taxonomies across different $\gamma$ values. Tasks are divided into visuospatial , and verbal ; blue encodes the transfer set. Edges represent transfers, with edge thickness proportional to the improvement. Dashed edged indicate multiple optimal transfers for the target. Edges depart prevalently from vertices of visuospatial tasks, indicating their versatile transferability across all of the considered games.

Figure 2: Comparison of the collective transfer performance against random transfer sets.

task $A$	task $B$	$\langle\tau_{A},\tau_{B}\rangle$	$\text{ovl}(\tau_{A},\tau_{B})$	$qs(A,B)$	$qs(B,A)$
img_P1	mia_P1	$12.9$	$44.7$	$+10.0$	$+6.9$
img_P1	ref_P1	$2.3$	$16.5$	$+20.0$	$-12.9$
img_P1	ref_P2	$2.0$	$12.7$	$+0.0$	$+3.7$
mia_P1	ref_P2	$2.2$	$13.3$	$+5.0$	$-15.0$
tab_P1	tab_P2	$7.4$	$35.7$	$-13.3$	$-12.5$

Table 2: Task-vector comparison computed as cosine (

\langle\cdot,\cdot\rangle

) and top-

5

subspace overlap ( ovl ) against

qs

transfer in both directions. Rows are restricted to targets where the share of completed episodes is unchanged from Base , so the quality-score gain is unambiguous. All values are reported

\times 10^{2}

. High similarity coincides with mutual harm (row 1) and with mutual help (row 2), so it does not determine transfer. Strong asymmetrical transfers (rows 3–5) imply low similarity.

Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents
Aug 28, 2026Nan LiLLM Agent Training Conversational Agents
Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands
Strategy, Not Payoffs: A Behavioural Embedding of Normal-Form Games
Jul 30, 2026Joshua Caiata, Sreepriya Pulyassary, Xiang Li +1LLM Fine-Tuning Strategic Reasoning Risks in Language Models
University of Waterloo
Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents
Aug 20, 2026Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian +1LLM Agent Skill Learning LLM Agent Skill Evaluation
Computer Science Department, Stony Brook University, Stony Brook, USA

Aug 28, 2026J/K move · Enter open · S save

Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

Nan Li

Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

LLM Agent Training Conversational Agents Multi-Turn Dialogue Management AI Agent Benchmarks Supervised Fine-Tuning

Open paper AskPDF arXiv

Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.

Abstract

Explore similar work

Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

Strategy, Not Payoffs: A Behavioural Embedding of Normal-Form Games

Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents