Task-oriented conversational agents remain fragile under realistic conditions, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks underrepresent adversarial conversation trajectories, thereby limiting the training resources available for developing robust agents. We present MADS (Multi-Agent Dialog Simulation), a synthetic data generation pipeline that produces structured, diverse dialogue trajectories. We inject adversarial conversation, producing validated conversation trajectories that capture realistic, unhappy paths. Using MADS, we construct MADS-Bench: a benchmark spanning six domains, 55 personas, and 1095 conversations, with approximately 66% of conversations containing failure-prone conversation scenarios. We evaluate the robustness of the trained models using a two-stage training approach that combines supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR).
Figures & tables
Figure 1: Illustration of an impossible scenario. Success requires explaining the unsuitability and proposing an alternative date, not scheduling the event.
Figure 2: Illustration of an unhappy-path scenario. The injected behavior is partial information: the user cannot provide a service ID or confirmation code. The agent must recover by invoking find_service before rescheduling.
Figure 3: Multi-Agent Dialog Simulation (MADS) pipeline for generating validated tool-augmented dialogue trajectories. Scenario configurations specifying task objectives, user personas, tool definitions, and injected adversarial behaviors are used to initialize a three-agent simulation consisting of a user agent, assistant agent, and tool agent.
Split
Turns
Conversations
Scenarios
Avg Turns
Train
574
106
46
5.42
Test
521
83
54
6.28
Total
1095
189
87
—
Table 1: MADS-Bench dataset summary
Figure 4: MADS-Bench training split distribution by domain (left) and scenario complexity (right). Banking dominates the training set (62.3%), while unhappy-path scenarios account for 42.5% of training conversations, reflecting the dataset’s emphasis on non-cooperative user behaviors.
Figure 5: MADS-Bench test split distribution by domain (left) and scenario complexity (right). Home Services appears exclusively in the test set, serving as a held-out domain for cross-domain generalization evaluation. The test set is predominantly composed of normal scenarios (71.1%), with unhappy-path and impossible scenarios comprising the remaining 28.9%.
Model
Approach
Dataset
Action Recall
Tool F1
Param Match
Response F1
Response Sim.
Qwen2.5-7B
Pre-Act
Base
0.22
0.38
0.38
0.07
0.14
Pre-Act
SFT-APIGen
0.32
0.50
0.38
0.1
0.15
Pre-Act
SFT
0.60
0.70
0.44
0.25
0.51
ReAct
Base
0.51
0.68
0.43
0.12
0.29
ReAct
SFT-APIGen
0.42
0.51
0.43
0.17
0.21
ReAct
SFT
0.54
0.54
0.44
0.15
0.35
Table 2: L1 (turn-level) evaluation on the MADS test set comparing supervision sources . Approach denotes the inference-time prompting format (Pre-Act vs. ReAct). Supervision indicates the dataset used for SFT (APIGen vs. MADS). All models are trained with a Pre-Act system prompt; ReAct is applied only at inference time.
Model
Approach
Stage
Action Recall
Tool F1
Param Match
Response F1
Response Sim.
Qwen2.5-7B
Pre-Act
SFT
0.60
0.70
0.44
0.25
0.51
Pre-Act
SFT+RLVR
0.67
0.77
0.51
0.21
0.46
ReAct
SFT
0.54
0.54
0.44
0.15
0.35
ReAct
SFT+RLVR
0.67
0.80
0.48
0.23
0.50
Qwen2.5-32B
Pre-Act
SFT
0.66
0.70
0.66
0.32
0.57
Pre-Act
SFT+RLVR
0.70
0.82
0.67
0.32
0.57
Table 3: L1 (turn-level) evaluation on the MADS test set comparing training stages on MADS . Stage denotes the optimization stage (SFT → SFT+RLVR). Approach denotes the inference-time prompting format (Pre-Act vs. ReAct). All models are trained with a Pre-Act system prompt; ReAct is applied only at inference time.
Model
Training Stage
Task Success(%)
Qwen2.5-7B
SFT
24.49
SFT+RLVR
39.78
Qwen2.5-32B
SFT
44.08
SFT+RLVR
53.96
Table 4: L2 (trajectory-level) task success on the MADS-bench test set. Training Stage denotes optimization on MADS (SFT vs. RLVR).
Domain
APIGen
MADS-Bench
SFT
SFT
SFT+RLVR
7B
7B
32B
7B
32B
Banking
33.33
44.44
16.67
33.33
66.67
Calendar
45.51
35.00
46.15
57.44
69.23
Home Services (Held Out)
40.92
28.87
49.95
42.50
53.87
Online Shopping
37.64
19.44
39.17
37.64
43.10
Table 5: L2 task success rate (%) by domain. Models are evaluated on the MADS-Bench test set. APIGen models are trained with SFT only. MADS-Bench models are trained with SFT and further refined with RLVR. The Home Services domain was excluded from training to evaluate cross-domain generalization. Travel contains a single test instance.
Complexity
APIGen
MADS-Bench
SFT
SFT
SFT+RLVR
7B
7B
32B
7B
32B
Normal
35.82
23.57
38.06
40.28
48.94
Unhappy Path
29.04
23.25
57.72
39.04
68.86
Impossible
40.0
40.00
63.33
36.67
56.67
Table 6: Task success (%) by scenario complexity for different model configurations (training data and objective). Unhappy-path scenarios involve non-cooperative user behaviors, such as goal changes and tool failures. Impossible scenarios require the agent to recognize task infeasibility and respond gracefully. APIGen models are trained only with SFT. MADS models are trained with SFT and further refined with RLVR. All models are evaluated on the MADS-Bench test set.