In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Yet existing benchmarks largely assume cooperative and behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. We introduce τ-Emotion-Bench, a ten-domain benchmark for evaluating tool-calling agents under controlled emotion-conditioned user interactions while preserving the underlying task objective. To scale both benchmark construction and agent training, we introduce Tracer, a scalable agentic data-generation pipeline that constructs grounded, policy-valid tasks and collects trajectories through an interactive agent--user--environment harness. By allowing dissatisfied users to terminate inefficient interactions, the harness provides an interaction-aware signal that filters trajectories with excessive user dependence and favors effective tool use. Experiments with two user simulators show that frontier agents remain unreliable, with substantial gaps between average task success and consistent task completion. Tracer-generated supervision consistently improves Qwen3 models across scales, generalizes to unseen domains, and transfers to τ-Bench and τ-Trait. Behavioral analyses further show that higher task success is associated with a better balance between tool execution and necessary user follow-up. Overall, τ-Emotion-Bench and Tracer provide a scalable framework for evaluating agent reliability under diverse user behaviors and improving tool-calling agents through interaction-aware supervision.
Figures & tables
Figure 1: Average accuracy of LLM agents on τ -Emotion-Bench over three runs.
Figure 2: Overview of τ -Emotion-Bench and the Tracer data-generation pipeline. Stage 1: Tracer builds policy-valid dependency graphs from tools, environments, and policies. Stage 2: Valid subgraphs are sampled to form executable multi-step traces. Stage 3: Traces are reverse-refined into natural user instructions while preserving task goals, context, preferences, and ambiguity. Stage 4: Task-specific feelings are mapped to structured emotion conditions and integrated with the task instruction. Stage 5: Tasks are verified and filtered for executability, policy consistency, and validity before inclusion in τ -Emotion-Bench.
Benchmark
Task Complexity
Tool Complexity
Interaction Complexity
Diversity
Scalability
Goal Clarity
Avg. Ground-Truth Tool Calls per Task
# Tools
Inter-tool Dependency
# Turns (approx.)
User Preferences
Emotion Behavioral
# Domains
Automated Generation
Scalable
ToolTalk ( Farn and Shin, 2023 )
✓
3.4
28
✓
[2,10]
✗
✗
1
✓ –
✗
IN3 ( Qian et al., 2024 )
✗
-
-
✗
[2,10]
✓
✗
1
✓ –
✓ –
MINT ( Wang et al., 2023 )
✓ –
-
13
✗
[2,10]
✗
✗
8
✗
✗
ToolSandbox ( Lu et al., 2025 )
✓ –
3.8
34
✓
[10,30]
✗
✗
11
✗
✗
DialogTool ( Wang et al., 2025 )
✓
4.7
30
✓
[10,30]
✗
✗
15
✓ –
✓ –
Table 1: Comparison of existing user interaction benchmarks across task, tool, interaction, diversity, and scalability dimensions. “ ✓ ” indicates fully addressed, “ ✓ – ” indicates partially addressed, and “ ✗ ” indicates not addressed. A “-” indicates that the corresponding information was not found.
Domain
Read Tools
Write Tools
Emotions (Family/Subfamily)
Test Tasks
Airline
8
6
5/16
70
Retail
9
7
5/17
70
Telecom
12
6
5/13
70
Telehealth
18
5
3/8
70
School
9
4
4/10
50
Basketball
17
3
1/2
50
Table 2: Benchmark statistics for τ -Emotion-Bench domains.
Model
Airline
Retail
Telecom
Telehealth
School
Basketball
Restaurant
Travel
Movie
Bank
Avg.
Non-thinking Models
Qwen3-32B
10.0
21.4
12.9
10.0
20.0
4.0
6.0
0.0
0.0
10.0
9.4
Qwen3-235B-A22B
17.1
15.7
10.0
21.4
22.0
24.0
4.0
0.0
0.0
24.0
13.8
DeepSeek-V4.1-Flash
32.9
44.3
17.1
31.4
42.0
38.0
26.0
30.0
22.0
38.0
32.2
Thinking Models
Qwen3-32B
8.6
21.4
8.6
15.7
14.0
12.0
2.0
2.0
2.0
0.0
8.6
Table 3: Performance comparison across ten τ -Emotion-Bench domains using Pass^3 (%).
Method
OOD Domains
School
Basketball
Restaurant
Travel
Movie
Bank
Avg. Pass^1
Avg. Pass^3
Pass^1
Pass^3
Pass^1
Pass^3
Pass^1
Pass^3
Pass^1
Pass^3
Pass^1
Pass^3
Pass^1
Pass^3
Qwen3-1.7B
Base
3.3
0.0
6.7
0.0
0.7
0.0
0.0
0.0
0.0
0.0
9.3
0.0
3.3
0.0
AReaL
4.0
2.0
4.0
0.0
0.7
0.0
0.0
0.0
0.0
0.0
2.7
0.0
1.9
0.3
EnvScaler
8.0
0.0
5.3
0.0
4.7
0.0
1.3
0.0
4.7
0.0
14.7
4.0
6.5
0.7
Table 4: Performance comparison of Tracer with existing agent-trained models across six OOD domains on τ -Emotion-Bench using Pass^1 and Pass^3. Values are in %.
Figure 3: Performance comparison of Qwen3 base, Tracer-Low, and Tracer-High models across τ -Bench and τ -Trait using Pass^1, Pass^2, and Pass^3.
Figure 4: Comparison of model groups by policy violation rate across domains and emotions.
Figure 5: Avg@3 versus tool-call-to-follow-up ratio across emotion conditions and model groups.
Figure 10
Figure 8: Distribution of tool calls and follow-up actions across AReaL, Simia, Tracer-Low, and Tracer-High training trajectories.
Method
In-Domain
Airline
Retail
Telecom
Avg. Pass^1
Avg. Pass^3
Pass^1
Pass^3
Pass^1
Pass^3
Pass^1
Pass^3
Qwen3-1.7B
Base
1.0
0.0
0.5
0.0
23.8
8.6
8.4
2.9
AReaL
11.0
1.4
10.0
1.4
13.3
2.9
11.4
1.9
Tracer-Low (Ours)
29.5
10.0
40.9
18.6
32.4
10.0
34.3
12.9
Table 5: Performance comparison of Tracer with existing agent-trained models across three in-domain domains on τ -Emotion-Bench using Pass^1 and Pass^3.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Tasks
Rollouts
R=1
R=0
R=−1
Success (%)
High
Low
Airline
922
922
143
680
99
15.5
143
143
Retail
898
898
585
288
25
65.1
250
585
Telecom
1,519
1,519
186
1,118
215
12.2
186
186
Telehealth
1,314
1,314
584
718
12
44.4
250
584
School
333
325
133
97
95
40.9
133
–
Basketball
876
457
346
72
39
75.7
250
–
Appendix
Table 6: Domain-level statistics for generated training tasks, rollout outcomes, success rates, and trajectories selected for Tracer-High and Tracer-Low.
Figure 9: Distribution of unsuccessful training trajectories across emotion families.
Figure 10: Distribution of training trajectories across emotion families.
Domain
R=1
Avg. User Turns
Avg. Assistant Turns
Avg. Tool Calls
Avg. Gold Actions
Avg. Reasoning Tokens
Airline
143
5.5
10.7
6.2
4.0
243
Retail
585
7.7
15.0
8.3
4.8
105
Telecom
186
7.0
12.5
6.5
4.4
166
Telehealth
584
6.7
15.0
9.3
5.5
121
School
133
6.7
14.5
8.8
4.5
180
Basketball
346
7.8
21.5
14.7
6.2
112
Appendix
Table 7: Statistics of successful ( R=1 ) training trajectories.
Hyperparameter
Value
Learning Rate
5×10−5
Scheduler
Cosine
Warmup Ratio
0.03
Epochs
3
Optimizer
Paged AdamW 8-bit
Weight Decay
0.0
Appendix
Table 8: Fine-tuning hyperparameters used for Qwen3-1.7B, Qwen3-4B and Qwen3-8B.
Model
Thinking
Serving
Hugging Face
Evaluated models
Qwen3-1.7B
On
Local
Qwen/Qwen3-1.7B
Qwen3-4B
On
Local
Qwen/Qwen3-4B
Qwen3-8B
On
Local
Qwen/Qwen3-8B
Qwen3-32B
On / Off
Local
Qwen/Qwen3-32B
Qwen3-235B-A22B-Instruct-2507
Off
ASU
Qwen/Qwen3-235B-A22B-Instruct-2507
Appendix
Table 9: Open-weight models we evaluate. Thinking : whether the model’s reasoning mode is on. Serving : ASU’s hosted inference cluster or our own vLLM server (Local).
Model
Thinking
Temp.
Top- p
Top- k
Min- p
Other
Evaluated models
Qwen3-1.7B
On
0.6
0.95
20
0
–
Qwen3-4B
On
0.0
–
–
–
–
Qwen3-8B
On
0.0
–
–
–
–
Qwen3-32B (thinking)
On
0.6
0.95
20
0
–
Qwen3-32B (non-thinking)
Off
0.7
0.8
20
0
–
Appendix
Table 10: Decoding configuration used in the largest number of domains for each model. A dash means the parameter is not set (the server default).
Category
Fault type
Description
Missing or Extra Action
Missed Action
A required action is not taken.
Wrong Order
Actions are taken out of the required order.
Extra Action
An action not needed for the task is taken.
Repeated Action
The same action is taken again without need.
Incomplete
Misread Intent
The user’s request or a tool result is misunderstood.
Early Stop
The conversation ends before the task is complete.
Appendix
Table 11: Fault taxonomy used in the error analysis.
Figure 11: Failure analysis across domains on τ -Emotion-Bench.
Figure 12: Share of failed trajectories with a critical agent error in each fault category for Qwen3 base and Tracer models, with Qwen3-235B-A22B-Thinking and DeepSeek-V4.1-Flash as references.
Figure 13: Task coverage across domains on τ -Emotion-Bench.
Model
Airline
Retail
Telecom
Telehealth
School
Basketball
Restaurant
Travel
Movie
Bank
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
Non-thinking Models
Qwen3-32B
20.0
13.3
10.0
46.2
29.5
21.4
30.9
18.1
12.9
33.3
18.1
10.0
40.7
28.0
20.0
16.0
7.3
4.0
16.7
8.0
6.0
4.7
1.3
0.0
3.3
0.7
0.0
27.3
15.3
10.0
Qwen3-235B-A22B
35.2
22.4
17.1
40.0
23.8
15.7
28.6
16.7
10.0
42.9
28.1
21.4
47.3
30.7
22.0
46.7
30.7
24.0
22.0
10.7
4.0
10.7
2.7
0.0
7.3
0.7
0.0
45.3
31.3
24.0
DeepSeek-V4.1-Flash
57.6
42.4
32.9
71.9
56.2
44.3
38.1
24.3
17.1
52.4
39.0
31.4
61.3
48.0
42.0
60.0
45.3
38.0
55.3
35.3
26.0
49.3
36.0
30.0
51.3
31.3
22.0
61.3
44.7
38.0
Thinking Models
Appendix
Table 12: Performance comparison of frontier open-source models and Tracer models across ten domains on τ -Emotion-Bench, evaluated using Pass^1 (p^1), Pass^2 (p^2), and Pass^3 (p^3). Values are reported as percentages (%). Qwen3-235B-A22B-Instruct-2507 is used as the user simulator. The highest values are bolded, and the second highest values are underlined.
Method
Airline
Retail
Telecom
Telehealth
School
Basketball
Restaurant
Travel
Movie
Bank
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
Qwen3-1.7B
Base
1.0
0.0
0.0
0.5
0.0
0.0
23.8
12.9
8.6
15.2
5.2
2.9
3.3
1.3
0.0
6.7
2.0
0.0
0.7
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
9.3
2.7
0.0
AReaL
11.0
3.3
1.4
10.0
2.9
1.4
13.3
5.7
2.9
2.4
0.0
0.0
4.0
2.7
2.0
4.0
0.0
0.0
0.7
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
2.7
7.0
0.0
EnvScaler
10.0
4.8
1.4
7.1
1.4
0.0
37.6
22.4
14.3
35.7
20.9
11.4
8.0
1.3
0.0
5.3
0.7
0.0
4.7
0.7
0.0
1.3
0.0
0.0
4.7
0.7
0.0
14.7
7.3
4.0
Qwen3-4B
Appendix
Table 13: Performance comparison of existing agent-trained models across ten domains on τ -Emotion-Bench using Pass^1 (p^1), Pass^2 (p^2), and Pass^3 (p^3). Values are in %. The highest values are bolded, and the second highest values are underlined.
Model
Airline
Retail
Telecom
Telehealth
School
Basketball
Restaurant
Travel
Movie
Bank
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
p^1
p^2
p^3
Non-thinking Models
Qwen3-32B
28.1
21.9
20.0
63.3
54.3
50.0
66.7
58.6
54.3
63.3
54.3
48.6
51.3
38.7
34.0
30.7
16.7
10.0
24.0
13.3
8.0
8.0
1.3
0.0
6.0
1.3
0.0
37.3
22.7
14.0
Qwen3-235B-A22B
48.6
40.9
37.1
63.3
50.5
42.9
63.3
53.8
48.6
68.6
56.7
50.0
70.7
60.0
54.0
53.3
42.0
36.0
40.7
34.7
32.0
32.7
16.7
10.0
17.3
9.3
6.0
58.7
44.7
38.0
DeepSeek-V4.1-Flash
86.2
81.4
78.6
96.2
93.8
92.9
65.7
55.7
50.0
75.2
64.3
60.0
68.0
59.3
54.0
71.3
60.7
54.0
80.0
71.3
66.0
94.7
90.0
86.0
60.7
46.0
36.0
76.7
71.3
68.0
Thinking Models
Appendix
Table 14: Performance comparison of frontier open-source models and Tracer models across ten domains on τ -Emotion-Bench, evaluated using Pass^1 (p^1), Pass^2 (p^2), and Pass^3 (p^3). Values are reported as percentages (%). GLM-5.2 is used as the user simulator. The highest values are bolded, and the second highest values are underlined.
Figure 14: Tool dependency graphs of all ten domains. Each panel puts read tools in the left column and write tools in the right. Gold nodes are root-eligible, i.e. valid starting points for a trace. A blue arrow marks a transition valid in both directions, drawn once per direction; an orange arrow marks one valid in a single direction only. Grey arcs are transitions within a column, read to read or write to write.
Domain
Core Entities
Airline
500 users, 300 flights, 2,000 reservations
Retail
527 users, 50 products, 1,075 orders
Telecom
500 customers, 15 services, 240 tickets
Telehealth
239 patients, 114 providers, 214 appointments
School
1,000 students, 120 courses, 1,345 registrations
Basketball
1,000 players, 240 clubs, 1,468 games
Appendix
Table 15: Core entities per domain database for evaluation set
Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, but this assumption has rarely been validated against human annotation. We introduce AgentProp-Bench, a 2,000-task benchmark with 2,300 traces across four domains, nine production LLMs, and a 100-label human-validated subset. We quantify judge reliability, characterize error propagation, and evaluate a runtime mitigation. Substring-based judging agrees with human annotation at kappa=0.049 (chance-level); a three-LLM ensemble reaches kappa=0.432 (moderate) with a conservative bias. Under validated evaluation, a parameter-level injection propagates to a wrong final answer with human-calibrated probability approximately 0.62 (range 0.46-0.73 across models). Rejection (catching bad parameters) and recovery (correcting after acceptance) are independent model capabilities (Spearman rho=0.126, p=0.747). A tuned runtime interceptor reduces hallucination on GPT-4o-mini by 23.0 percentage points under a concurrent n=600 control, but shows no significant effect on Gemini-2.0-Flash, whose aggressive parameter rejection eliminates the target failure mode. All code, data, traces, and human labels are released at https://github.com/bhaskargurram-ai/agenthallu-bench.
As LLM-based agents increasingly rely on external tools, it is important to evaluate their ability to sustain tool-grounded reasoning beyond familiar workflows and short-range interactions. We introduce AgentEscapeBench, an escape-room-style benchmark that tests whether agents can infer, execute, and revise novel tool-use procedures under explicit long-range dependency constraints. Each task defines a directed acyclic dependency graph over tools and items, requiring agents to invoke real external functions, track hidden state revealed incrementally, propagate intermediate results, and submit a deterministically verifiable final answer. AgentEscapeBench includes 270 instances across five difficulty tiers and supports fully automated evaluation. Experiments with sixteen LLM agents and human participants show that performance drops sharply as dependency depth increases: humans decline from 98.3% success at difficulty-5 to 80.0% at difficulty-25, while the best model drops from 90.0% to 60.0%. Trajectory analysis attributes model failures mainly to breakdowns in long-range state tracking, clue adherence, and intermediate-result propagation. These findings suggest that current agents can often handle local tool use but still struggle with deep contextual dependencies. We hope AgentEscapeBench can serve as a diagnostic testbed for measuring current agent capabilities and informing future training efforts toward more robust general-purpose reasoning, action, and adaptation.
Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.