MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games

Jacob Eisenstein; Fantine Huot; Adam Fisch; Jonathan Berant; Mirella Lapata

cs.CLFeb 27, 2026

MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games

Authors: Jacob Eisenstein, Fantine Huot, Adam Fisch, Jonathan Berant, Mirella Lapata

Organizations: Google DeepMind

Abstract

We present a scalable and verifiable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communication about private information. This enables an interactive scaling analysis, in which a fixed token budget is divided over a variable number of turns. We find that language models often fail to use interactive collaboration to improve over the non-interactive baseline in which one agent summarizes its information and the other agent immediately acts, despite substantial headroom. This suggests that state-of-the-art models still suffer from significant weaknesses in planning and executing multi-turn collaborative conversations. We analyze the linguistic features of these dialogues, assessing the roles of sycophancy, information density, and discourse coherence. While there is no single linguistic explanation for the collaborative weaknesses of contemporary language models, we note that humans achieve comparable task success at superior token efficiency by producing more coherent dialogues. The proactive management of private information is a defining feature of real-world communication, and this work is designed to drive further progress on this capability.

Figures & tables

Panel of Figure 1 — Figure 1: Isotoken evaluation , showing task accuracy with a constant token budget sharded over varying numbers of turns. In most cases, performance is flat or decreasing as the turn budget increases, indicating that the models fail to exploit interactivity.

Panel of Figure 2 — Figure 2: Successful chess dialogue from Gemini 2.5 Pro with thinking. Chess is the sole task in which the models regularly exceed the token budget, as seen in turns 1, 5, 6, and 7.

Panel of Figure 3 — Figure 3: An example of a successful COVR dialogue from Gemma3-12B.

Panel of Figure 4 — Figure 4: Top: example of successful dialogue for GPT-4o on the MD3 image selection task; the guesser sees all six images, while the describer sees only the highlighted image on the lower right. Bottom: unsuccessful dialogue for Qwen-VL8B on the Tangram task; the guesser sees all four images, while the describer sees the highlighted image on the far right.

Panel of Figure 5 — Figure 5: A successful but lucky dialogue for Gemini 2.5 Flash (thinking) on the name-game.

Panel of Figure 6 — Figure 6: Comparison of the Gemini models against a simple guess-and-check heuristic on name-game.

Figure 7: Isotoken evaluation of Gemini Flash models

Panel of Figure 8 — Figure 8: Tokens and turns per dialogue.

Panel of Figure 9 — Figure 9: Counts of early answer proposals in the chess task for the Gemini 2.5 thinking models. Some incorrect proposals are accepted, but at a far lower rate than correct proposals.

Panel of Figure 10 — Figure 10: Lexical density and coherence for AI models and humans on the MD3 image selection task. The language model scores are averaged over all turn budgets.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure C.11: Full turn/token budget sweep for Gemini 2.5 Flash

Figure C.12: Full turn/token budget sweep for Gemini 3.5 Flash

Panel of Figure 13 — Figure E.13: Example of sycophancy and answer proposal in chess (Gemini 2.5 Flash, thinking). Bob offers a spurious apology in turn 4. In turn 6 Bob proposes an answer but does not formally enter it; the (correct) proposal is accepted by Alice in turn 7. Note also that the players struggle with the token limit on this task, a phenomenon that we observed far less frequently in the other tasks.

Panel of Figure 14 — Figure E.14: Rates of sycophantic features per dialogue across models on the chess task.

Panel of Figure 15 — Figure E.15: Lexical density as a function of dialogue length. Each curve shows a model’s average density at 4, 8, and 16 turns. Higher values reflect more content-bearing and novel lexical choices. Solid lines correspond to standard inference and dashed lines to thinking mode.

Panel of Figure 16 — Figure E.16: Example dialogues and their centering transitions (as identified by our automatic procedure). The top example is Gemini 2.5 Pro (no thinking) on MD3; the bottom example is Qwen-VL8B on name-game. We also highlight the preferred ( $C_{p}$ ) and backward-looking ( $C_{b}$ ) centers as detected by our algorithm. Entities that are both the preferred and backward-looking center are highlighted in orange . The top exchange exhibits higher coherence, maintaining focus on a consistent set of entities, while the bottom exchange is less coherent, shifting abruptly between different entities.

Panel of Figure 17 — Figure E.17: Mean Centering Coherence Score (CS) for five private information games across three turn lengths (max_turns $\in\{4,8,16\}$ ).

Transition	Condition
Continue	$C_{b_{n}}=C_{b_{n-1}}$ and $C_{b_{n}}=C_{p_{n}}$
Retain	$C_{b_{n}}=C_{b_{n-1}}$ and $C_{b_{n}}\neq C_{p_{n}}$
Smooth Shift	$C_{b_{n}}\neq C_{b_{n-1}}$ and $C_{b_{n}}=C_{p_{n}}$
Rough Shift	$C_{b_{n}}\neq C_{b_{n-1}}$ and $C_{b_{n}}\neq C_{p_{n}}$ , or $C_{b_{n}}$ is undefined

Appendix

Table F.5: Centering transition classification rules.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities
Mar 17, 2026Victoria Graf, Valentina Pyatkin, Nouha Dziri +2Multi-Turn Dialogue Evaluation Multi-Turn Dialogue Management
University of Washington · Allen Institute for AI
For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instances
Sep 16, 2026Alexander Shirnin, Aleksey KudelyaDeception in Language Models LLM Security
Be Careful Who You Trust: Coordination Dynamics under Corrupted Communication in LLM Multi-Agent Games
Sep 19, 2026Xuanyi Liu, Niall Dalton, Hairi Amin +2Multi-Agent Coordination Multi-Agent LLM Systems
Department of Computer Science, University College London

Mar 17, 2026J/K move · Enter open · S save

TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

Victoria Graf, Valentina Pyatkin, Nouha Dziri, Nathan Lambert+1

University of Washington · Allen Institute for AI

Multi-Turn Dialogue Evaluation Multi-Turn Dialogue Management LLM Training

Open paper AskPDF arXiv

Multi-turn conversations are a common and critical mode of language model interaction. However, current open training and evaluation data focus on single-turn settings, failing to capture the additional dimension of these longer interactions. To understand this multi-/single-turn gap, we first introduce a new benchmark, TurnWiseEval, for multi-turn capabilities that is directly comparable to single-turn chat evaluation. Our evaluation isolates multi-turn specific conversational ability through pairwise comparison to equivalent single-turn settings. We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data. Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance, and that including as little as 10k multi-turn conversations during post-training can lead to a 12% improvement on TurnWiseEval.

Abstract

LLM Evaluation

Explore similar work

TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instances

Be Careful Who You Trust: Coordination Dynamics under Corrupted Communication in LLM Multi-Agent Games