Tool-using language models are increasingly deployed in conversational settings, yet most large-scale tool-use benchmarks focus on isolated requests. We introduce ToolTurns, a benchmark of 2,620 problems spanning 1,827 tools and 11,004 API endpoints across 47 categories. Each dialogue begins with an intentionally underspecified request and continues through two context-dependent follow-ups, testing clarification, tool use, and retention of conversational context. We also introduce an open-source user simulator that reveals withheld context through dialogue and generates follow-up requests, with a checker--fixer pipeline for improving its turns. Across nine language models, the best agent resolves 69.6% of individual requests but completes only 39.8% of full dialogues. Under this evaluation protocol, every evaluated model receives lower success scores on follow-up requests than on the initial request, and dialogue-quality rubric scores and task success can rank models very differently. Paired turn-level evaluation shows that the checker--fixer reduces detected dialogue defects, and blind human evaluation favors the revised turns. ToolTurns provides a testbed for evaluating whether agents sustain task success across multi-turn dialogue with tools.
Figures & tables
Benchmark
Resource size
Tool/API scope
User interaction during evaluation
Evaluation target
ToolBench
126,486 instruction–solution pairs (corpus)
3,451 tools; 16,464 APIs; 49 categories
Single user instruction; tool steps are not user turns
Pass rate and win rate (ToolEval)
ToolDial
11,111 generated dialogues (corpus)
RapidAPI graph (4,474 APIs)
Recorded dialogue histories
Action, API-argument, and response prediction
ToolTalk
78 annotated conversations
28 callable tools
Fixed user turns; predicted calls executed with feedback
Table 1: How the evaluation setting differs across benchmarks. ToolTurns combines broad API coverage with adaptive clarification and follow-up dialogue, complementing benchmarks centered on stateful application workflows. Counting conventions for each row are in Appendix A .
Metric
Value
Total problems
2,620
Single-tool
1,032 (39.4%)
Multi-tool (2+)
1,588 (60.6%)
Cross-category
1,023 (64.4% of multi)
Distinct tools
1,827
Distinct API endpoints
11,004
Table 2: ToolTurns statistics for the 2,620-problem evaluation set. The last two rows report mean / median.
Proprietary
Open-Weight
Metric
4o-mini
4.1-mini
5.4-mini
Opus 4.8
Qwen3.5
Qwen3.6
Gemma 4
GPT-OSS ‡
GLM-5.2
397B
27B
31B
120B
744B
Overall (weighted)
0.646
0.740
0.817
0.809
0.683
0.709
0.759
0.693
0.685
Task success
57.0%
64.5%
56.3%
69.6%
60.9%
61.0%
67.5%
55.2%
67.9%
all three pass
25.1%
33.9%
26.0%
39.8%
29.3%
29.0%
36.4%
24.9%
37.7%
Consistency
0.839
0.866
0.924
0.898
0.837
0.858
0.899
0.873
0.836
Table 3: Dialogue quality and task success per agent model, over a shared set of 2,620 problems. Bold denotes the best value per row. Open-weight models are labeled by family with total parameter count below; checkpoints and quantization are in Appendix E . The weighted aggregate, the pooled subtask pass rate, and the stricter all-three-pass rate are defined in § 3.4 ; dialogues and subtasks for which the judge returned an error are excluded. ‡ GPT-OSS is also the judge model, so its scores are not judge-independent.
Model
Initial
FU 1
FU 2
Δ
GPT-4o-mini
65.2
55.7
50.3
−14.9
GPT-4.1-mini
67.7
63.9
62.0
−5.7
GPT-5.4-mini
57.6
55.9
55.5
−2.1
Opus 4.8
74.7
65.4
68.6
−6.1
Qwen3.5-397B
67.6
57.8
57.5
−10.1
Qwen3.6-27B
65.8
57.2
60.1
−5.7
Table 4: Subtask success rate (%) by position of the main query within the dialogue: the initial query and the two follow-ups (FU 1, FU 2). Δ is the change from the initial query to the second follow-up. Every model degrades as the dialogue lengthens.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
GPT-4o-mini
Qwen3-32B (single-call)
Qwen3-32B (checker–fixer)
Overall (weighted)
0.858
0.862
0.871
Realism
0.807
0.805
0.812
Contextual awareness
0.863
0.869
0.880
Relevance
0.904
0.900
0.911
Natural language
0.874
0.877
0.880
Conversational flow
0.808
0.827
0.839
Appendix
Table 5: User-simulator quality scores under three conditions (322 dialogues each). Bold denotes the best value per row. Qwen3 refers to Qwen3-32B-AWQ; checker–fixer denotes the full checker–fixer pipeline ( § 3.3.1 ). Dimensions and weights follow Appendix B.1 .
Checker reported
Revision preferred
p
Contextual unawareness
26/34 (76.5%)
0.003
Repetition
13/17 (76.5%)
0.049
Lack of progress
34/45 (75.6%)
8×10−4
Any non-templating
49/73 (67.1%)
0.005
Templating alone
27/47 (57.4%)
0.38
Verbosity
5/19 (26.3%)
0.064
Appendix
Table 6: Blind human preference for the revised user turn over the original draft, by the violation type the checker reported. Categories overlap, since a turn may trigger several. Two-sided sign tests.
Dimension
GPT-4o-mini user
Qwen3-32B user
Human (all 622)
Human (excl. 54)
Overall (weighted)
0.845
0.851
0.737
0.772
Realism
0.762
0.766
0.576
0.610
Contextual awareness
0.900
0.888
0.915
0.909
Relevance
0.864
0.885
0.618
0.677
Natural language
0.861
0.867
0.776
0.823
Conversational flow
0.783
0.807
0.710
0.772
Appendix
Table 7: User-simulator quality scores for our simulator vs. human-written API-Bank dialogues. Each simulator condition and the full human reference contain 622 records; the filtered human reference contains 568. All are scored by the same LLM judge with the same prompt and dimensions (Appendix B.1 ). Qwen3-32B user uses the checker–fixer pipeline. Human (excl. 54) removes 54 purely one-line items from the reference set. Bold denotes the best value per row. Simulator and human dialogues differ in the assistant and tool environment as well as in the user, so observed gaps reflect protocol differences and not user behaviour alone.
Type
What the checker flags
Repetition
Repeating prior requests or information already given
Hallucination
Fabricating facts attributed to the assistant or tools
Contextual unawareness
Ignoring what the assistant actually said or asking for already-provided information
Verbosity
Unnecessary or overly long text
Conversational flow
Unnatural or disconnected phrasing
False agreement
Claiming to have received information that was not provided
Appendix
Table 8: The eleven violation types detected by the user-turn checker ( § 3.3.1 ).
4o-mini
4.1-mini
5.4-mini
Opus 4.8
Qwen3.5
Qwen3.6
Gemma 4
GPT-OSS
GLM-5.2
Initial
✗
✗
✗
✗
✗
✓
✓
✗
✗
FU 1
✗
✓
✓
✓
✗
✓
✓
✗
✓
FU 2
✓
✓
✓
✗
✓
✓
✓
✗
✓
Rubric overall
0.55
0.75
0.68
0.80
0.53
0.85
0.80
0.43
0.70
Appendix
Table 9: Subtask outcomes and weighted rubric score for problem 2610. Follow-up queries differ across agents because each is generated from that agent’s own dialogue.
Consistency
Relevance
Efficiency
Correctness
Adaptability
Overall
Task success
GPT-5.4-mini
1.00
1.00
0.75
1.00
0.75
0.925
0/3
Opus 4.8
1.00
0.25
0.50
0.50
1.00
0.600
3/3
Appendix
Table 10: Rubric aspects and task success for two agents on problem 140.
4o-mini
4.1-mini
5.4-mini
Opus 4.8
Qwen3.5
Qwen3.6
Gemma 4
GPT-OSS
GLM-5.2
Task success (%)
57.0
64.5
56.3
69.6
60.9
61.0
67.5
55.2
67.9
finish calls that give up (%)
7.1
1.6
6.9
0.1
7.9
6.1
1.1
12.4
0.6
Initial queries with no API call (%)
1.7
4.0
28.3
3.3
2.3
2.2
4.1
9.2
1.1
API calls per dialogue
15.6
8.3
3.6
7.2
14.0
10.3
10.6
10.7
17.4
message_user calls per dialogue
3.05
1.99
0.73
0.86
1.38
1.78
0.92
1.36
1.29
Dialogues with an answer outside finish (%)
0.3
0.0
0.5
< 0.1
< 0.1
0.3
< 0.1
56.1
< 0.1
Appendix
Table 11: Agent behaviour over the 2,620 ToolTurns dialogues. Give up : finish called with return_type give_up_and_restart . No API call : the initial subtask, including any clarification turns, contains no call other than finish or message_user . Answer outside finish : at least one assistant turn that contains a final_answer payload as plain text instead of a finish call; the runtime treats such a turn as a message to the user.
Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.
Large Language Models (LLMs) are increasingly deployed through conversational interfaces, yet strong single-turn performance does not imply reliability in multi-turn settings. Multi-turn assistance requires a model to maintain an evolving representation of user requirements and to reason over it before complying with later requests. Failures can therefore be subtle: models may fail to recognize a changed constraint or hidden conflict, or recognize it without enforcing it in the final response. Existing benchmarks primarily test fully specified prompts, long-context recall, or controlled domain environments, leaving open-domain requirement reconciliation undermeasured. We introduce TurnFlow, an interactive benchmark for evaluating whether models preserve, revise, and act on user requirements across turns. TurnFlow combines adaptive user simulation with verifiable judging, allowing conversations to unfold naturally while keeping hidden requirements and pass/fail criteria controlled. In our evaluation, nine LLMs achieve overall pass rates from 42.86% to 59.59% on 490 scenarios. Performance varies substantially across challenge types: models perform well on Stateful Multi-Constraint Following but remain considerably less reliable on Scope Creep and Goal Shift. These results reveal persistent limitations in reasoning over evolving user requirements across turns.
Zhixun Chen, Ming Li, Anjie Liu +3
Hong Kong University of Science and Technology (Guangzhou) · University of Maryland, College Park · Peking University +1
Multi-turn conversations are a common and critical mode of language model interaction. However, current open training and evaluation data focus on single-turn settings, failing to capture the additional dimension of these longer interactions. To understand this multi-/single-turn gap, we first introduce a new benchmark, TurnWiseEval, for multi-turn capabilities that is directly comparable to single-turn chat evaluation. Our evaluation isolates multi-turn specific conversational ability through pairwise comparison to equivalent single-turn settings. We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data. Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance, and that including as little as 10k multi-turn conversations during post-training can lead to a 12% improvement on TurnWiseEval.