Investigating Knowledge Transfer Across Interactive Dialogue Games
Authors: Filippo Momentè, Mir Nafis Sharear Shopnil, Andrea de Varda, Pavel Merinov, Raffaella Bernardi, Oswald Lanz, Alessandro Suglia, Alessandro Torcinovich
Organizations: University of Trento · Technovative Solutions Ltd · Massachusetts Institute of Technology · Free University of Bozen Bolzano · University of Edinburgh
Dialogue games represent a challenging setting where complex cognitive skills are required to accomplish tasks while coordinating with other players. Considering that language represents an interface for both understanding the game rules and executing actions, it is reasonable to assume that training on a specific language game will enhance specific capabilities that might be relevant for other tasks as well. Motivated by this rationale, in this paper, we investigate how knowledge transfers across different dialogue games. We study transferability by finetuning LLM models on games from the clembench suite (Chalamalasetti et al., 2023) and performing two analyses: i) we derive a task-transferability graph using a binary integer optimization program from Zamir et al. (2018), using task performance as the main metric; and ii) we compute task vectors (Ilharco et al., 2022) for each game to study similarities across finetuned models and their task transferability. In our first analysis, we find that some games benefit more from transfer than finetuning, and that the visuospatial family (e.g., exploration games) transfers best. With our task vector analysis instead, we find that similarity-based approaches capture game-role relationships but almost no transferability patterns, suggesting that more complex metrics are required.
Figures & tables
Target
adventuregame
codenames_P1
codenames_P2
guesswhat_P1
guesswhat_P2
imagegame_P1
imagegame_P2
matchit_ascii_P1
referencegame_P1
referencegame_P2
taboo_P1
taboo_P2
wordle_withclue
wordle_withcritic_P1
wordle_withcritic_P2
Source
adventuregame
0.50
0.29
0.25
0.10
0.88
0.07
0.10
0.20
0.05
codenames_P1
0.27
0.30
0.10
0.77
0.01
0.12
0.20
0.01
codenames_P2
0.10
0.22
0.78
0.04
0.09
0.10
0.02
guesswhat_P1
0.27
0.82
0.15
0.05
0.05
0.02
0.08
guesswhat_P2
0.25
0.50
0.90
0.03
0.05
0.10
0.10
Table 1: Filtered transfer modeling matrix. Tasks are divided into visuospatial , and verbal ; P1 / P2 denotes player roles, while empty cells correspond to specialists underperforming the baseline.
Figure 1: Taxonomies across different γ values. Tasks are divided into visuospatial , and verbal ; blue encodes the transfer set. Edges represent transfers, with edge thickness proportional to the improvement. Dashed edged indicate multiple optimal transfers for the target. Edges depart prevalently from vertices of visuospatial tasks, indicating their versatile transferability across all of the considered games.
Figure 2: Comparison of the collective transfer performance against random transfer sets.
task A
task B
⟨τA,τB⟩
ovl(τA,τB)
qs(A,B)
qs(B,A)
img_P1
mia_P1
12.9
44.7
+10.0
+6.9
img_P1
ref_P1
2.3
16.5
+20.0
−12.9
img_P1
ref_P2
2.0
12.7
+0.0
+3.7
mia_P1
ref_P2
2.2
13.3
+5.0
−15.0
tab_P1
tab_P2
7.4
35.7
−13.3
−12.5
Table 2: Task-vector comparison computed as cosine ( ⟨⋅,⋅⟩ ) and top- 5 subspace overlap ( ovl ) against qs transfer in both directions. Rows are restricted to targets where the share of completed episodes is unchanged from Base , so the quality-score gain is unambiguous. All values are reported ×102 . High similarity coincides with mutual harm (row 1) and with mutual help (row 2), so it does not determine transfer. Strong asymmetrical transfers (rows 3–5) imply low similarity.
Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.
Nan Li
Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands
Learning a strategic task changes more than what is directly taught: fine-tuning on one game can either enhance or degrade an agent's ability to reason in another. Understanding and predicting this transfer of strategic capabilities, however, remains a key challenge for large language models (LLMs). Normal-form games provide an ideal testbed for analyzing this phenomenon, as they feature explicitly defined payoffs and well-characterized equilibrium behaviours. In this work, we investigate whether game embeddings can explain and predict changes in LLM strategic capabilities following fine-tuning across different games. We propose a lightweight two-feature embedding that captures fundamental behavioural demands: the entropy of the Nash equilibrium and the sensitivity of optimal responses to an opponent's action. We show that while existing published structural embeddings primarily memorize game identities and fail to generalize, our behavioural embedding reliably predicts performance changes on held-out games. These results demonstrate that the transfer of strategic capabilities in LLMs is not dictated by the payoff geometry of a game, but by the underlying structure of the decision-making behaviour it requires.
Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience. In practice, induced skills may transfer unreliably and can even harm the agent that retrieves them. When agent-induced skills transfer reliably across tasks remains an open question. We conduct a comprehensive and controlled study of how the way skills are induced shapes their transfer across tasks. Specifically, we compare task-level with subtask-level skill induction and text with code skill formats, the two axes along which existing methods differ. Task-level skills mostly reduce the agent's performance below its no-memory baseline while subtask-level skills raise it above on average, and text skills transfer better than code skills. To further understand our findings, we examine two complementary properties of the induced skills: specificity, which measures how closely a skill matches real tasks, and abstractness, which measures how evenly its relevance spreads across tasks. Neither property alone predicts task success, but their combined effect does, which we propose as a skill utility score. The score correlates consistently with task success when skills are transferred, and subtask-level and text skills score higher. Computing skill utility only needs the skills and task descriptions, so our score serves as a lightweight diagnostic of a skill memory for practitioners.