cs.CLNov 11, 2026

ToolTurns: Benchmarking LLM Agents in Multi-Turn Dialogue

Authors: Milad Gholami, Brendan King, Jeffrey Flanigan

Organizations: University of California, Santa Cruz

Abstract

Tool-using language models are increasingly deployed in conversational settings, yet most large-scale tool-use benchmarks focus on isolated requests. We introduce ToolTurns, a benchmark of 2,620 problems spanning 1,827 tools and 11,004 API endpoints across 47 categories. Each dialogue begins with an intentionally underspecified request and continues through two context-dependent follow-ups, testing clarification, tool use, and retention of conversational context. We also introduce an open-source user simulator that reveals withheld context through dialogue and generates follow-up requests, with a checker--fixer pipeline for improving its turns. Across nine language models, the best agent resolves 69.6% of individual requests but completes only 39.8% of full dialogues. Under this evaluation protocol, every evaluated model receives lower success scores on follow-up requests than on the initial request, and dialogue-quality rubric scores and task success can rank models very differently. Paired turn-level evaluation shows that the checker--fixer reduces detected dialogue defects, and blind human evaluation favors the revised turns. ToolTurns provides a testbed for evaluating whether agents sustain task success across multi-turn dialogue with tools.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TurnFlow: Benchmarking User-State Reasoning in Multi-Turn LLM Agents via Scene-Guided User Simulation

    Nov 7, 2026Zhixun Chen, Ming Li, Anjie Liu +3Multi-Turn Language Model ReasoningMulti-Turn Dialogue Evaluation

  2. TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

    Mar 17, 2026Victoria Graf, Valentina Pyatkin, Nouha Dziri +2Multi-Turn Dialogue EvaluationMulti-Turn Dialogue Management