Can LLM Agents Resolve Service Failures? Simulating Multi-Turn Complaint Handling under Information Asymmetry

Wailing Ng; Mengze Hong; Chen Jason Zhang; Shiqi Fan

cs.AIOct 17, 2026

Can LLM Agents Resolve Service Failures? Simulating Multi-Turn Complaint Handling under Information Asymmetry

Authors: Wailing Ng, Mengze Hong, Chen Jason Zhang, Shiqi Fan

Organizations: The Hong Kong Polytechnic University

Abstract

LLM agents are widely adopted in service chatbots for rule-guided support and information access, yet emotionally demanding complaint handling still relies heavily on human agents. We introduce ServiceBench, to our knowledge the first benchmark dedicated to complaint handling, with 2,500 scenarios spanning 184 complaint categories and five service industries. We propose Asymmetric Progressive Complaint Simulation (APCS) to simulate realistic complaint handling, together with a justice-based evaluation framework. Our experiments reveal that even capable LLM agents struggle to handle complaints effectively. They lack the ability to proactively gather information and strategically guide conversations toward resolution, often resorting to premature compensation. We further show that targeted post-training with SFT and DPO substantially improves these capabilities, enabling a smaller model to match or surpass much larger models on several outcomes.

Figures & tables

Figure 1: Comparison of complaint handling quality and goodwill expenditure across LLM agents.

Figure 2: Overall framework of ServiceBench . We construct scenarios from real complaint dialogues on X and expert guided industry level customer journey maps to ensure realistic, diverse, and operationally grounded service failure contexts. We then apply Asymmetric Progressive Complaint Simulation (APCS) to generate multi-turn complaint handling interactions under information asymmetry. Finally, an LLM-based judge evaluates the completed interactions based on interactional, procedural, and distributive justice.

Model	Justice Score				Diseng.	Agree.	Goodwill
Model	Interactional	Procedural	Distributive	Total	Diseng.	Agree.	Goodwill
Airline
GPT-5	$3.3050_{\pm 0.8922}$	$3.4950_{\pm 0.7765}$	$3.8200_{\pm 0.5995}$	$3.5400_{\pm 0.6433}$	0.0536	0.9464	91.2
GPT-4o	$2.6500_{\pm 0.7212}$	$2.6400_{\pm 0.6500}$	$3.2600_{\pm 0.7845}$	$2.8500_{\pm 0.6041}$	0.2199	0.7801	72.2
GPT-4o mini	$2.2150_{\pm 0.5100}$	$2.1750_{\pm 0.5531}$	$2.8000_{\pm 0.7957}$	$2.3967_{\pm 0.5235}$	0.4619	0.5330	69.0
DeepSeek-V4-Pro	$3.5000_{\pm 0.8507}$	$3.4050_{\pm 0.8688}$	$3.9150_{\pm 0.5912}$	$3.6067_{\pm 0.6814}$	0.0847	0.9153	70.2

Table 1: Main evaluation results on ServiceBench across five service industries. Justice scores are reported as mean

\pm

standard deviation. The best and second best results are highlighted with bold orange and blue backgrounds.

Figure 3: Performance under different inference-time reasoning strategies.

Model	Guidance	Justice Score				Diseng.	Agree.	Goodwill
Model	Guidance	Interactional	Procedural	Distributive	Total	Diseng.	Agree.	Goodwill
GPT-5	None	3.17	3.35	3.83	3.45	0.025	0.975	53.80
	Case	3.46	3.58	3.97	3.67	0.013	0.987	51.74
	Procedure	3.86	3.83	4.07	3.92	0.012	0.988	52.58
	Case + Procedure	3.28	3.44	4.01	3.58	0.042	0.958	49.62
DeepSeek-V4-Pro	None	3.47	3.13	3.90	3.50	0.111	0.889	49.40

Table 2: Effect of case-based and procedural guidance on complaint handling performance.

Figure 4: Performance under progressive disclosure and full information.

Model	Measure	Null	Turn 1	Turn 2	Turn 3+	Overall
DeepSeek-V4-Pro	First Goodwill (%)	11.80	30.80	48.20	9.20	–
DeepSeek-V4-Pro	Information Coverage (%)	–	15.88	28.56	30.82	25.98
Qwen3-32B	First Goodwill (%)	2.00	92.00	4.80	1.20	–
Qwen3-32B	Information Coverage (%)	–	18.64	19.90	20.00	18.74
GPT-4o mini	First Goodwill (%)	12.00	62.60	17.00	8.40	–
GPT-4o mini	Information Coverage (%)	–	16.86	25.87	36.51	20.83

Table 3: Timing of the first goodwill offer and information coverage at commitment.

Model	Complaint Severity
Model	Low	Medium	High
GPT-4o	0.19 (0.30)	0.24 (0.25)	0.27 (0.27)
GPT-4o mini	0.18 (0.10)	0.21 (-0.10)	0.21 (-0.34)
GPT-5	0.21 (0.57)	0.27 (0.64)	0.30 (0.53)
DeepSeek-V4-Pro	0.18 (0.00)	0.22 (-0.04)	0.29 (0.47)
Gemini 2.5 Flash	0.17 (-0.13)	0.20 (-0.31)	0.24 (-0.02)

Table 4: Normalized goodwill across complaint severity levels. Parentheses report within-severity z-scores.

Figure 5: Model performance across customer communication styles.

Model	Justice Score				Diseng.	Agree.	Goodwill
Model	Interactional	Procedural	Distributive	Total	Diseng.	Agree.	Goodwill
GPT-5	3.11	3.37	3.84	3.44	0.053	0.947	57.95
DeepSeek-V4-Pro	3.27	3.18	3.95	3.47	0.215	0.785	57.40
Qwen3-8B	2.85	2.76	3.23	2.95	0.214	0.786	86.25
+ SFT	3.39	3.34	3.68	3.47	0.093	0.907	51.05
+ SFT + DPO	3.69	3.46	3.60	3.58	0.042	0.958	44.59

Table 5: Effect of targeted post-training on Qwen3-8B complaint handling performance.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Judge	DeepSeek-V4	GPT-5	Llama 70B	Qwen 32B	Gemini 2.5	GPT-4o	GPT-4o mini
DeepSeek-V4-Pro	0.15 ∗∗ (.009)	0.13 ∗ (.014)	-0.18 ∗∗ (.002)	-0.19 ∗∗ (.002)	0.12 ∗ (.019)	-0.04 (.388)	0.01 (.916)
GPT-5	0.07 (.131)	0.09 ∗ (.024)	0.03 (.505)	-0.06 (.088)	-0.03 (.305)	0.03 (.410)	-0.12 ∗∗ (.005)
Llama 3.3 70B	-0.19 ∗∗∗ ( $<$ .001)	-0.22 ∗∗∗ ( $<$ .001)	0.27 ∗∗∗ ( $<$ .001)	0.07 (.158)	0.04 (.428)	-0.03 (.533)	0.06 (.246)
Qwen3-32B	-0.04 (.439)	0.00 (.986)	-0.11 ∗∗ (.007)	0.19 ∗∗∗ ( $<$ .001)	-0.13 ∗∗∗ ( $<$ .001)	0.04 (.351)	0.05 (.258)

Appendix

Table 6: Normalized judge preference advantage. Statistical significance is denoted by

{}^{*}=p<0.05

{}^{**}=p<0.01

, and

{}^{***}=p<0.001

Judge	Service Agent
Judge	GPT-5	DeepSeek-V4-Pro	Qwen3-32B	GPT-4o	Llama 3.3 70B	Gemini 2.5 Flash	GPT-4o mini
DeepSeek-V4-Pro	3.64 $\pm$ 0.62	3.58 $\pm$ 0.66	3.14 $\pm$ 0.73	2.94 $\pm$ 0.65	2.56 $\pm$ 0.58	2.75 $\pm$ 0.70	2.41 $\pm$ 0.51
GPT-5	3.92 $\pm$ 0.47	3.84 $\pm$ 0.51	3.53 $\pm$ 0.69	3.30 $\pm$ 0.69	3.00 $\pm$ 0.72	2.95 $\pm$ 0.73	2.64 $\pm$ 0.71
Llama 3.3 70B	4.35 $\pm$ 0.69	4.29 $\pm$ 0.82	4.22 $\pm$ 0.84	3.78 $\pm$ 1.04	3.65 $\pm$ 1.01	3.46 $\pm$ 1.09	3.15 $\pm$ 1.09
Qwen3-32B	4.48 $\pm$ 0.66	4.36 $\pm$ 0.84	4.25 $\pm$ 0.93	3.76 $\pm$ 1.04	3.25 $\pm$ 0.94	3.22 $\pm$ 1.02	3.03 $\pm$ 1.00
Ensemble	4.10 $\pm$ 0.52	4.02 $\pm$ 0.62	3.79 $\pm$ 0.71	3.44 $\pm$ 0.79	3.11 $\pm$ 0.74	3.10 $\pm$ 0.83	2.81 $\pm$ 0.76

Appendix

Table 7: Cross-judge consistency of service-agent evaluation. Values are reported as mean

\pm

standard deviation.

Justice Dimension	$\boldsymbol{\hat{\beta}}$	$t$ -statistic	$p$ -value
Interactional	$-0.554^{***}$	$-18.06$	$<0.001$
Procedural	$-0.236^{***}$	$-7.58$	$<0.001$
Distributive	$-0.066^{*}$	$-2.06$	$0.040$
Overall	$-0.285^{***}$	$-10.89$	$<0.001$

Appendix

Table 8: Regression estimates for the association between service agent response length and justice scores. Statistical significance is denoted by

{}^{*}=p<0.05

{}^{**}=p<0.01

, and

{}^{***}=p<0.001

Customer Simulator	Service Agent	Justice				Goodwill	Turns	Avg. Length
Customer Simulator	Service Agent	Interactional	Procedural	Distributive	Total	Goodwill	Turns	Avg. Length
DeepSeek-V4-Pro	GPT-4o	2.87	2.65	3.42	2.98	45.0	6.78	66.39
	GPT-4o mini	2.34	2.20	2.93	2.49	40.2	6.38	56.01
	GPT-5	3.25	3.46	3.88	3.53	36.0	9.06	60.17
	DeepSeek-V4-Pro	3.70	3.18	4.04	3.64	44.1	5.69	58.70
	Gemini 2.5	3.22	2.85	3.63	3.23	33.9	6.51	56.56

Appendix

Table 9: Service agent performance under different customer simulation policies.

Top 10 Categories		Bottom 10 Categories
Category	Justice	Category	Justice
Product Hazard Concerns	3.80	Baggage Follow-up Failures	2.50
Delivery Handling Failures	3.52	Room Assignment Failures	2.58
Product Mismatch	3.50	Pest and Infestation Problems	2.60
Replacement Execution Failures	3.48	Room Environment Problems	2.65
Unsafe Consumables	3.45	Specialist Service Delays	2.67

Appendix

Table 10: Highest and lowest performing complaint categories by total justice.

Figure 6: Cumulative resolution outcomes over dialogue turns.

Model	Airline	Hotel	Restaurant	Retail	Ride-Hailing & Delivery
GPT-4o	0.361 (0.076)	0.383 (0.371)	0.066 (0.186)	0.154 (0.330)	0.125 (0.354)
GPT-4o mini	0.345 (-0.065)	0.336 (-0.027)	0.055 (-0.278)	0.149 (0.218)	0.098 (-0.224)
GPT-5	0.456 (0.908)	0.426 (0.731)	0.064 (0.061)	0.133 (-0.116)	0.122 (0.274)
DeepSeek-V4-Pro	0.351 (-0.009)	0.386 (0.393)	0.060 (-0.080)	0.129 (-0.221)	0.090 (-0.381)
Gemini 2.5 Flash	0.337 (-0.134)	0.292 (-0.393)	0.062 (-0.006)	0.142 (0.075)	0.106 (-0.059)
Llama 3.3 70B	0.130 (-1.945)	0.098 (-2.027)	0.023 (-1.655)	0.055 (-1.836)	0.030 (-1.655)

Appendix

Table 11: Normalized goodwill expenditure across service industries. Values are normalized by the USD 200 discretionary limit; parentheses report within-industry standardized scores.

Model	Turns	Length
GPT-5	10.32	61.74
GPT-4o	7.54	68.40
GPT-4o mini	7.77	56.66
DeepSeek-V4-Pro	6.69	63.61
Gemini 2.5 Flash	8.03	51.37
Llama 3.3 70B	10.13	42.18

Appendix

Table 12: Conversation statistics across evaluated service agents.

Model	Run	Justice				Goodwill	Turns
Model	Run	Interactional	Procedural	Distributive	Total	Goodwill	Turns
DeepSeek-V4-Pro	1	3.47	3.42	4.02	3.64	42.9	6.54
	2	3.43	3.37	3.98	3.59	49.2	6.64
	3	3.52	3.28	3.98	3.59	40.9	6.44
	4	3.45	3.28	3.99	3.57	43.3	6.62
	5	3.38	3.25	3.83	3.49	42.2	6.79

Appendix

Table 13: Run-to-run consistency across five independent simulations.

Annotator Pair	Interactional	Procedural	Distributive
Judge 1 – Judge 2	0.870	0.903	0.766
Judge 1 – Judge 3	0.779	0.810	0.758
Judge 2 – Judge 3	0.774	0.763	0.791
Mean pairwise	0.808	0.825	0.772

Appendix

Table 14: Human annotator agreement measured by pairwise Spearman correlation.

Figure 7: Human evaluation of customer simulator realism.

Judge Model	Interactional		Procedural		Distributive
Judge Model	Spearman	MAE	Spearman	MAE	Spearman	MAE
GPT-5	0.831	0.348	0.813	0.420	0.789	0.367
GPT-4o	0.866	0.343	0.778	0.468	0.766	0.477
GPT-4o mini	0.839	0.297	0.787	0.378	0.735	0.632
Llama 3.3 70B	0.793	0.700	0.778	0.755	0.761	0.585
Qwen3-32B	0.795	0.718	0.764	0.728	0.738	0.553

Appendix

Table 15: Validation of candidate LLM judges against human annotations.

Industry	Selected Customer Care Accounts	# Raw Source Dialogues
Airlines	AmericanAir, Delta, JetBlue, SouthwestAir	78,864
Hotels	HiltonHelp, IHGService	2,460
Restaurants	McDonalds, TacoBellTeam, KFC_UKI_Help	13,247
Retail	AskTarget, Walmart	13,541
Ride-Hailing and Delivery	Uber_Support, AskLyft	52,029

Appendix

Table 16: Selected customer care accounts and raw dialogue counts used for customer journey map construction.

Benchmark Scale	Count
Scenarios	2,500
Service industries	5
Customer journey stages	37
Complaint categories	184
Industry Coverage	Stage / Category
Airline	8 / 44

Appendix

Table 17: Summary statistics of ServiceBench . Detailed complaint categories are provided in Appendix C.7 .

Component	Provenance	Description
Source complaint dialogues	Collected	Public customer complaint dialogues collected from X, providing the original facts of service failures.
Customer journey maps	Data driven and manually constructed	Service stages and complaint categories developed through topic discovery and expert guided coding.
Service event sequences	Reconstructed	Plausible sequences of events leading to each complaint, reconstructed from source dialogues while preserving the original facts.
Customer experience background and resolution expectation	Generated with manual refinement	Detailed customer experiences and resolution expectations developed from the event sequences and refined for realism and consistency.
Company case record	Generated with manual refinement	Available company information inferred from the customer experience using industry specific schemas and manually refined for realism and consistency.

Appendix

Table 18: Provenance of the main components used in ServiceBench scenario construction.

Communication Style	Prompt Template
Assertive-Warm	Maintain a clear, cooperative stance and focus on fair resolution. State the problem and what would feel fair, while still showing respect for the agent’s effort. Give relevant details when needed and respond positively when the agent is helpful. Ask for clarification or a better next step when the response is vague or insufficient.
Assertive-Cold	Maintain a firm, skeptical stance and focus on concrete action. State the problem and expected outcome clearly, with little effort to sound friendly or cooperative. Treat vague explanations or unsupported offers as insufficient, and ask for accountability when the agent does not provide a credible path forward. Keep messages controlled but demanding.
Passive-Warm	Maintain a polite, patient, and cooperative stance. Use softer language and avoid direct confrontation. Share concerns gradually, with strong attention to whether the agent understands and treats the issue with care. Respond more openly when the agent shows understanding or provides a credible explanation. When the response feels unclear or insufficient, continue expressing concern gently rather than making strong demands.
Passive-Cold	Maintain a brief, withdrawn, and skeptical stance. Give limited details unless the agent asks specific, low-pressure follow-up questions. Show dissatisfaction through short replies or hesitation rather than direct confrontation. Become less responsive when the agent gives generic or repetitive responses. Engage more only when the agent shows that the concern is taken seriously and provides a credible path forward.

Appendix

Table 19: Communication style prompt.

Panel of Figure 8 — Figure 8: Overview of the airline customer journey map.

Panel of Figure 9 — Figure 9: Overview of the hotel customer journey map.

Panel of Figure 10 — Figure 10: Overview of the restaurant customer journey map.

Panel of Figure 11 — Figure 11: Overview of the retail customer journey map.

Panel of Figure 12 — Figure 12: Overview of the ride-hailing and delivery customer journey map.

Panel of Figure 18 — Figure 18: Case Study 1: DeepSeek-V4-Pro as the service agent. The agent responds with restrained and formulaic language despite the customer describing an unsafe and humiliating experience. Although goodwill is offered, the response does not develop into stronger relational care or concrete protective action that would demonstrate appropriate seriousness.

Panel of Figure 19 — Figure 19: Case Study 2: DeepSeek-V4-Pro as the service agent. The agent provides material recovery through a refund and goodwill credit, but offers limited relational care after the customer describes sleep loss and disrupted plans. It also does not explain what happened or how the issue will be handled, treating the remedy itself as the primary path to closure.

Panel of Figure 20 — Figure 20: Case Study 3: GPT-5 as the service agent. The agent offers premature goodwill for a reservation failure that primarily requires explanation, then continues to request incremental details after the core issue is already clear. The dialogue appears procedurally organized, but becomes overly information seeking rather than focused on clarifying the failure and next steps.

Panel of Figure 21 — Figure 21: Case Study 4: Qwen3-32B as the service agent. The agent provides limited relational care and procedural clarity, instead treating the complaint as a transactional negotiation. As the customer continues to challenge the adequacy of the remedy, the agent repeatedly increases goodwill credits rather than addressing the underlying disclosure failure or offering a consistent resolution.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 681 papers

CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
Jun 1, 2026Danqing Wang, Akshay Sivaraman, Lei LiHuman Behavior Simulation LLM Agent Evaluation
Carnegie Mellon University
CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments
Mar 30, 2026Yi Yu, Guangquan Hu, Chenghuang Shen +15LLM Agent Efficiency LLM Agent Evaluation
School of Mathematics and Sciences, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai, China +2
SAGE: A Service Agent Graph-guided Evaluation Benchmark
Apr 10, 2026Ling Shi, Yuqin Dai, Ziyin Wang +7LLM Agent Evaluation Agent Evaluation
Tianjin University, Tianjin, China · Tsinghua University, Beijing, China · Beihang University, Beijing, China +3

Jun 1, 2026J/K move · Enter open · S save

CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation

Danqing Wang, Akshay Sivaraman, Lei Li

Carnegie Mellon University

Human Behavior Simulation LLM Agent Evaluation AI Agent Benchmarks User Simulation

Open paper AskPDF arXiv

Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.

Abstract

LLM Agents

Explore similar work

CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation

CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments

SAGE: A Service Agent Graph-guided Evaluation Benchmark