Tool-using language models are increasingly deployed in conversational settings, yet most large-scale tool-use benchmarks focus on isolated requests. We introduce ToolTurns, a benchmark of 2,620 problems spanning 1,827 tools and 11,004 API endpoints across 47 categories. Each dialogue begins with an intentionally underspecified request and continues through two context-dependent follow-ups, testing clarification, tool use, and retention of conversational context. We also introduce an open-source user simulator that reveals withheld context through dialogue and generates follow-up requests, with a checker--fixer pipeline for improving its turns. Across nine language models, the best agent resolves 69.6% of individual requests but completes only 39.8% of full dialogues. Under this evaluation protocol, every evaluated model receives lower success scores on follow-up requests than on the initial request, and dialogue-quality rubric scores and task success can rank models very differently. Paired turn-level evaluation shows that the checker--fixer reduces detected dialogue defects, and blind human evaluation favors the revised turns. ToolTurns provides a testbed for evaluating whether agents sustain task success across multi-turn dialogue with tools.
Figures & tables
Benchmark
Resource size
Tool/API scope
User interaction during evaluation
Evaluation target
ToolBench
126,486 instruction–solution pairs (corpus)
3,451 tools; 16,464 APIs; 49 categories
Single user instruction; tool steps are not user turns
Pass rate and win rate (ToolEval)
ToolDial
11,111 generated dialogues (corpus)
RapidAPI graph (4,474 APIs)
Recorded dialogue histories
Action, API-argument, and response prediction
ToolTalk
78 annotated conversations
28 callable tools
Fixed user turns; predicted calls executed with feedback
Table 1: How the evaluation setting differs across benchmarks. ToolTurns combines broad API coverage with adaptive clarification and follow-up dialogue, complementing benchmarks centered on stateful application workflows. Counting conventions for each row are in Appendix A .
Metric
Value
Total problems
2,620
Single-tool
1,032 (39.4%)
Multi-tool (2+)
1,588 (60.6%)
Cross-category
1,023 (64.4% of multi)
Distinct tools
1,827
Distinct API endpoints
11,004
Table 2: ToolTurns statistics for the 2,620-problem evaluation set. The last two rows report mean / median.
Proprietary
Open-Weight
Metric
4o-mini
4.1-mini
5.4-mini
Opus 4.8
Qwen3.5
Qwen3.6
Gemma 4
GPT-OSS ‡
GLM-5.2
397B
27B
31B
120B
744B
Overall (weighted)
0.646
0.740
0.817
0.809
0.683
0.709
0.759
0.693
0.685
Task success
57.0%
64.5%
56.3%
69.6%
60.9%
61.0%
67.5%
55.2%
67.9%
all three pass
25.1%
33.9%
26.0%
39.8%
29.3%
29.0%
36.4%
24.9%
37.7%
Consistency
0.839
0.866
0.924
0.898
0.837
0.858
0.899
0.873
0.836
Table 3: Dialogue quality and task success per agent model, over a shared set of 2,620 problems. Bold denotes the best value per row. Open-weight models are labeled by family with total parameter count below; checkpoints and quantization are in Appendix E . The weighted aggregate, the pooled subtask pass rate, and the stricter all-three-pass rate are defined in § 3.4 ; dialogues and subtasks for which the judge returned an error are excluded. ‡ GPT-OSS is also the judge model, so its scores are not judge-independent.
Model
Initial
FU 1
FU 2
Δ
GPT-4o-mini
65.2
55.7
50.3
−14.9
GPT-4.1-mini
67.7
63.9
62.0
−5.7
GPT-5.4-mini
57.6
55.9
55.5
−2.1
Opus 4.8
74.7
65.4
68.6
−6.1
Qwen3.5-397B
67.6
57.8
57.5
−10.1
Qwen3.6-27B
65.8
57.2
60.1
−5.7
Table 4: Subtask success rate (%) by position of the main query within the dialogue: the initial query and the two follow-ups (FU 1, FU 2). Δ is the change from the initial query to the second follow-up. Every model degrades as the dialogue lengthens.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
GPT-4o-mini
Qwen3-32B (single-call)
Qwen3-32B (checker–fixer)
Overall (weighted)
0.858
0.862
0.871
Realism
0.807
0.805
0.812
Contextual awareness
0.863
0.869
0.880
Relevance
0.904
0.900
0.911
Natural language
0.874
0.877
0.880
Conversational flow
0.808
0.827
0.839
Appendix
Table 5: User-simulator quality scores under three conditions (322 dialogues each). Bold denotes the best value per row. Qwen3 refers to Qwen3-32B-AWQ; checker–fixer denotes the full checker–fixer pipeline ( § 3.3.1 ). Dimensions and weights follow Appendix B.1 .
Checker reported
Revision preferred
p
Contextual unawareness
26/34 (76.5%)
0.003
Repetition
13/17 (76.5%)
0.049
Lack of progress
34/45 (75.6%)
8×10−4
Any non-templating
49/73 (67.1%)
0.005
Templating alone
27/47 (57.4%)
0.38
Verbosity
5/19 (26.3%)
0.064
Appendix
Table 6: Blind human preference for the revised user turn over the original draft, by the violation type the checker reported. Categories overlap, since a turn may trigger several. Two-sided sign tests.
Dimension
GPT-4o-mini user
Qwen3-32B user
Human (all 622)
Human (excl. 54)
Overall (weighted)
0.845
0.851
0.737
0.772
Realism
0.762
0.766
0.576
0.610
Contextual awareness
0.900
0.888
0.915
0.909
Relevance
0.864
0.885
0.618
0.677
Natural language
0.861
0.867
0.776
0.823
Conversational flow
0.783
0.807
0.710
0.772
Appendix
Table 7: User-simulator quality scores for our simulator vs. human-written API-Bank dialogues. Each simulator condition and the full human reference contain 622 records; the filtered human reference contains 568. All are scored by the same LLM judge with the same prompt and dimensions (Appendix B.1 ). Qwen3-32B user uses the checker–fixer pipeline. Human (excl. 54) removes 54 purely one-line items from the reference set. Bold denotes the best value per row. Simulator and human dialogues differ in the assistant and tool environment as well as in the user, so observed gaps reflect protocol differences and not user behaviour alone.
Type
What the checker flags
Repetition
Repeating prior requests or information already given
Hallucination
Fabricating facts attributed to the assistant or tools
Contextual unawareness
Ignoring what the assistant actually said or asking for already-provided information
Verbosity
Unnecessary or overly long text
Conversational flow
Unnatural or disconnected phrasing
False agreement
Claiming to have received information that was not provided
Appendix
Table 8: The eleven violation types detected by the user-turn checker ( § 3.3.1 ).
4o-mini
4.1-mini
5.4-mini
Opus 4.8
Qwen3.5
Qwen3.6
Gemma 4
GPT-OSS
GLM-5.2
Initial
✗
✗
✗
✗
✗
✓
✓
✗
✗
FU 1
✗
✓
✓
✓
✗
✓
✓
✗
✓
FU 2
✓
✓
✓
✗
✓
✓
✓
✗
✓
Rubric overall
0.55
0.75
0.68
0.80
0.53
0.85
0.80
0.43
0.70
Appendix
Table 9: Subtask outcomes and weighted rubric score for problem 2610. Follow-up queries differ across agents because each is generated from that agent’s own dialogue.
Consistency
Relevance
Efficiency
Correctness
Adaptability
Overall
Task success
GPT-5.4-mini
1.00
1.00
0.75
1.00
0.75
0.925
0/3
Opus 4.8
1.00
0.25
0.50
0.50
1.00
0.600
3/3
Appendix
Table 10: Rubric aspects and task success for two agents on problem 140.
4o-mini
4.1-mini
5.4-mini
Opus 4.8
Qwen3.5
Qwen3.6
Gemma 4
GPT-OSS
GLM-5.2
Task success (%)
57.0
64.5
56.3
69.6
60.9
61.0
67.5
55.2
67.9
finish calls that give up (%)
7.1
1.6
6.9
0.1
7.9
6.1
1.1
12.4
0.6
Initial queries with no API call (%)
1.7
4.0
28.3
3.3
2.3
2.2
4.1
9.2
1.1
API calls per dialogue
15.6
8.3
3.6
7.2
14.0
10.3
10.6
10.7
17.4
message_user calls per dialogue
3.05
1.99
0.73
0.86
1.38
1.78
0.92
1.36
1.29
Dialogues with an answer outside finish (%)
0.3
0.0
0.5
< 0.1
< 0.1
0.3
< 0.1
56.1
< 0.1
Appendix
Table 11: Agent behaviour over the 2,620 ToolTurns dialogues. Give up : finish called with return_type give_up_and_restart . No API call : the initial subtask, including any clarification turns, contains no call other than finish or message_user . Answer outside finish : at least one assistant turn that contains a final_answer payload as plain text instead of a finish call; the runtime treats such a turn as a message to the user.