Solving problems through tool use under explicit constraints constitutes a highly challenging yet unavoidable scenario for large language models (LLMs), requiring capabilities such as function calling, instruction following, and self-refinement. However, progress has been hindered by the absence of dedicated evaluations. To address this, we introduce CCTU, a benchmark for evaluating LLM tool use under complex constraints. CCTU is grounded in a taxonomy of 12 constraint categories spanning four dimensions (i.e., resource, behavior, toolset, and response). The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios, each involving an average of seven constraint types and an average prompt length exceeding 4,700 tokens. To enable reliable evaluation, we develop an executable constraint validation module that performs step-level validation and enforces compliance during multi-turn interactions between models and their environments. We evaluate nine state-of-the-art LLMs in both thinking and non-thinking modes. Results indicate that when strict adherence to all constraints is required, no model achieves a task completion rate above 20%. Further analysis reveals that models violate constraints in over 50% of cases, particularly in the resource and response dimensions. Moreover, LLMs demonstrate limited capacity for self-refinement even after receiving detailed feedback on constraint violations, highlighting a critical bottleneck in the development of robust tool-use agents. To facilitate future research, we release the data and code.
Figures & tables
Figure 1: Comparison between general tool use and constrained tool use. The constrained setting introduces a constraint validation module that performs step-level verification across multi-turn interactions and provides feedback when violations occur.
Benchmarks
Basic Information
Constraint Dimensions
Evaluated Capabilities
#Number
Avg. Tools
Avg. Len.
Precise Eval.
Resource
Behavior
Toolset
Response
Func. Call.
Ins. Follow.
Self-Refine.
IFEval ( Zhou et al., 2023 )
541
0
54
✓
✗
✗
✗
✓
✗
✓
✗
IFBench ( Pyatkin et al., 2025 )
300
0
90
✓
✗
✗
✗
✓
✗
✓
✗
MultiChallenge ( Deshpande et al., 2025 )
273
0
1636
✗
✗
✗
✗
✓
✗
✓
✗
BFCL v4 ( Patil et al., 2024 )
5088
7.08
1446
✓
✗
✗
✓
✓
✓
✓
✗
τ -bench ( Yao et al., 2025 )
165
12.36
4245
✓
✗
✗
✓
✗
✓
✗
✗
Table 1: Comparison of benchmarks across basic information, constraint dimensions, and evaluated capabilities.
Figure 2: Distribution of CCTU queries across 28 domains.
Figure 3: The number of samples associated with each constraint category in CCTU.
Figure 4: Distribution of the number of constraint categories per sample in CCTU.
Figure 5: Distribution of data lengths in CCTU, measured using the Qwen3 tokenizer.
Models
Single-Hop
Parallel Single-Hop
Multi-Hop
Parallel Multi-Hop
Overall
SR
PSR
SR
PSR
SR
PSR
SR
PSR
SR
PSR
Thinking Mode
GPT-5.2
32.671.89
24.67 3.40
24.671.89
17.33 0.94
25.332.49
20.672.49
15.330.94
10.001.63
24.500.82
18.17 1.03
GPT-5.1
25.335.25
20.007.12
20.672.49
16.000.00
22.671.89
20.671.89
22.670.94
9.332.49
22.832.01
16.502.83
Claude Opus 4.6
34.67 4.99
10.001.63
30.67 2.49
13.330.94
38.67 3.40
23.33 1.89
32.67 1.89
12.67 0.94
34.17 2.25
14.831.03
Seed-2.0-Pro
22.673.77
19.334.99
20.675.25
12.673.40
22.670.94
18.670.94
15.331.89
8.672.49
20.332.62
14.832.87
Table 2: Performance in thinking and non-thinking modes, ranked by overall PSR score. Standard deviations are shown in smaller font at the lower right of each mean. The best result in each column is highlighted in bold .
Figure 6: The probability of different LLMs violating each category of constraints.
Figure 7: Self-refinement probability of different LLMs under different constraint categories.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Stage
One-Round
Two-Round
Total
Revision Rate
Data Verification
38
6
44
21%
Code Verification
19
35
54
27%
Appendix
Table 3: Statistics of manual revisions during the quality control process. The revision rate represents the proportion of revised instances relative to the total dataset (200 instances).
Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.
Despite great advances in tool-use capabilities of large language models (LLMs), existing evaluation benchmarks struggle to fully align with real-world scenarios. Such benchmarks mostly rely on simulated idealized user assumptions and lacks experience-oriented evaluation. These limitations fail to account for the ambiguity, uncooperative behaviors, and shifting intentions characteristic of real-world users. To fill this gap, we propose RUT-Bench, a dedicated benchmark designed to assess LLMs under diverse Real-world User Tool calling scenarios. RUT-Bench supports high-fidelity simulations covering both ideal rational patterns and heterogeneous non-ideal behaviors across single-turn and multi-turn dialogues. We conduct comprehensive evaluations on 19 widely adopted open-source and proprietary LLMs using our benchmark. Experimental results reveal that no tested LLMs achieve an overall success rate above 40%, and nearly all of them experience noticeable performance drops when facing more complicated non-ideal user inputs. Our code and data is available at https://github.com/Miaow-Lab/RUT-Bench.
Xuan Yang, Hao Xu, Tingfeng Hui +4
Department of Data Science, City University of Hong Kong · 3Li Auto Inc. · 2Hong Kong Institute of AI for Science, City University of Hong Kong +1
Tool-use capability is a fundamental component of LLM agents, enabling them to interact with external systems through structured function calls. However, existing research exhibits inconsistent interaction representations, largely overlooks the structural distribution of tool-use trajectories, and relies on incompatible evaluation benchmarks. We present UniToolCall, a unified framework for tool learning that standardizes the entire pipeline from toolset construction and dataset generation to evaluation. The framework curates a large tool pool of 22k+ tools and constructs a hybrid training corpus of 390k+ instances by combining 10 standardized public datasets with structurally controlled synthetic trajectories. It explicitly models diverse interaction patterns, including single-hop vs. multi-hop and single-turn vs. multi-turn, while capturing both serial and parallel execution structures. To support coherent multi-turn reasoning, we further introduce an Anchor Linkage mechanism that enforces cross-turn dependencies. Furthermore, we convert 7 public benchmarks into a unified Query--Action--Observation--Answer (QAOA) representation with fine-grained evaluation at the function-call, turn, and conversation levels. Experiments show that fine-tuning Qwen3-8B on our dataset substantially improves tool-use performance. Under the distractor-heavy Hybrid-20 setting, achieves 93.0% single-turn Strict Precision, outperforming commercial models including GPT, Gemini, and Claude.
Yijuan Liang, Xinghao Chen, Yifan Ge +5
University of Science and Technology of China · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, Eastern Institute of Technology, Ningbo · Department of Computing, The Hong Kong Polytechnic University