Can LLM Agents Resolve Service Failures? Simulating Multi-Turn Complaint Handling under Information Asymmetry

Wailing Ng; Mengze Hong; Chen Jason Zhang; Shiqi Fan

cs.AIOct 17, 2026

Can LLM Agents Resolve Service Failures? Simulating Multi-Turn Complaint Handling under Information Asymmetry

Authors: Wailing Ng, Mengze Hong, Chen Jason Zhang, Shiqi Fan

Organizations: The Hong Kong Polytechnic University

Abstract

LLM agents are widely adopted in service chatbots for rule-guided support and information access, yet emotionally demanding complaint handling still relies heavily on human agents. We introduce ServiceBench, to our knowledge the first benchmark dedicated to complaint handling, with 2,500 scenarios spanning 184 complaint categories and five service industries. We propose Asymmetric Progressive Complaint Simulation (APCS) to simulate realistic complaint handling, together with a justice-based evaluation framework. Our experiments reveal that even capable LLM agents struggle to handle complaints effectively. They lack the ability to proactively gather information and strategically guide conversations toward resolution, often resorting to premature compensation. We further show that targeted post-training with SFT and DPO substantially improves these capabilities, enabling a smaller model to match or surpass much larger models on several outcomes.

Figures & tables

Figure 1: Comparison of complaint handling quality and goodwill expenditure across LLM agents.

Figure 2: Overall framework of ServiceBench . We construct scenarios from real complaint dialogues on X and expert guided industry level customer journey maps to ensure realistic, diverse, and operationally grounded service failure contexts. We then apply Asymmetric Progressive Complaint Simulation (APCS) to generate multi-turn complaint handling interactions under information asymmetry. Finally, an LLM-based judge evaluates the completed interactions based on interactional, procedural, and distributive justice.

Model	Justice Score				Diseng.	Agree.	Goodwill
Model	Interactional	Procedural	Distributive	Total	Diseng.	Agree.	Goodwill
Airline
GPT-5	$3.3050_{\pm 0.8922}$	$3.4950_{\pm 0.7765}$	$3.8200_{\pm 0.5995}$	$3.5400_{\pm 0.6433}$	0.0536	0.9464	91.2
GPT-4o	$2.6500_{\pm 0.7212}$	$2.6400_{\pm 0.6500}$	$3.2600_{\pm 0.7845}$	$2.8500_{\pm 0.6041}$	0.2199	0.7801	72.2
GPT-4o mini	$2.2150_{\pm 0.5100}$	$2.1750_{\pm 0.5531}$	$2.8000_{\pm 0.7957}$	$2.3967_{\pm 0.5235}$	0.4619	0.5330	69.0
DeepSeek-V4-Pro	$3.5000_{\pm 0.8507}$	$3.4050_{\pm 0.8688}$	$3.9150_{\pm 0.5912}$	$3.6067_{\pm 0.6814}$	0.0847	0.9153	70.2

Table 1: Main evaluation results on ServiceBench across five service industries. Justice scores are reported as mean

\pm

standard deviation. The best and second best results are highlighted with bold orange and blue backgrounds.

Figure 3: Performance under different inference-time reasoning strategies.

Model	Guidance	Justice Score				Diseng.	Agree.	Goodwill
Model	Guidance	Interactional	Procedural	Distributive	Total	Diseng.	Agree.	Goodwill
GPT-5	None	3.17	3.35	3.83	3.45	0.025	0.975	53.80
	Case	3.46	3.58	3.97	3.67	0.013	0.987	51.74
	Procedure	3.86	3.83	4.07	3.92	0.012	0.988	52.58
	Case + Procedure	3.28	3.44	4.01	3.58	0.042	0.958	49.62
DeepSeek-V4-Pro	None	3.47	3.13	3.90	3.50	0.111	0.889	49.40

Table 2: Effect of case-based and procedural guidance on complaint handling performance.

Figure 4: Performance under progressive disclosure and full information.

Model	Measure	Null	Turn 1	Turn 2	Turn 3+	Overall
DeepSeek-V4-Pro	First Goodwill (%)	11.80	30.80	48.20	9.20	–
DeepSeek-V4-Pro	Information Coverage (%)	–	15.88	28.56	30.82	25.98
Qwen3-32B	First Goodwill (%)	2.00	92.00	4.80	1.20	–
Qwen3-32B	Information Coverage (%)	–	18.64	19.90	20.00	18.74
GPT-4o mini	First Goodwill (%)	12.00	62.60	17.00	8.40	–
GPT-4o mini	Information Coverage (%)	–	16.86	25.87	36.51	20.83

Table 3: Timing of the first goodwill offer and information coverage at commitment.

Model	Complaint Severity
Model	Low	Medium	High
GPT-4o	0.19 (0.30)	0.24 (0.25)	0.27 (0.27)
GPT-4o mini	0.18 (0.10)	0.21 (-0.10)	0.21 (-0.34)
GPT-5	0.21 (0.57)	0.27 (0.64)	0.30 (0.53)
DeepSeek-V4-Pro	0.18 (0.00)	0.22 (-0.04)	0.29 (0.47)
Gemini 2.5 Flash	0.17 (-0.13)	0.20 (-0.31)	0.24 (-0.02)

Table 4: Normalized goodwill across complaint severity levels. Parentheses report within-severity z-scores.

Figure 5: Model performance across customer communication styles.

Model	Justice Score				Diseng.	Agree.	Goodwill
Model	Interactional	Procedural	Distributive	Total	Diseng.	Agree.	Goodwill
GPT-5	3.11	3.37	3.84	3.44	0.053	0.947	57.95
DeepSeek-V4-Pro	3.27	3.18	3.95	3.47	0.215	0.785	57.40
Qwen3-8B	2.85	2.76	3.23	2.95	0.214	0.786	86.25
+ SFT	3.39	3.34	3.68	3.47	0.093	0.907	51.05
+ SFT + DPO	3.69	3.46	3.60	3.58	0.042	0.958	44.59

Table 5: Effect of targeted post-training on Qwen3-8B complaint handling performance.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Judge	DeepSeek-V4	GPT-5	Llama 70B	Qwen 32B	Gemini 2.5	GPT-4o	GPT-4o mini
DeepSeek-V4-Pro	0.15 ∗∗ (.009)	0.13 ∗ (.014)	-0.18 ∗∗ (.002)	-0.19 ∗∗ (.002)	0.12 ∗ (.019)	-0.04 (.388)	0.01 (.916)
GPT-5	0.07 (.131)	0.09 ∗ (.024)	0.03 (.505)	-0.06 (.088)	-0.03 (.305)	0.03 (.410)	-0.12 ∗∗ (.005)
Llama 3.3 70B	-0.19 ∗∗∗ ( $<$ .001)	-0.22 ∗∗∗ ( $<$ .001)	0.27 ∗∗∗ ( $<$ .001)	0.07 (.158)	0.04 (.428)	-0.03 (.533)	0.06 (.246)
Qwen3-32B	-0.04 (.439)	0.00 (.986)	-0.11 ∗∗ (.007)	0.19 ∗∗∗ ( $<$ .001)	-0.13 ∗∗∗ ( $<$ .001)	0.04 (.351)	0.05 (.258)

Appendix

Table 6: Normalized judge preference advantage. Statistical significance is denoted by

{}^{*}=p<0.05

{}^{**}=p<0.01

, and

{}^{***}=p<0.001

Judge	Service Agent
Judge	GPT-5	DeepSeek-V4-Pro	Qwen3-32B	GPT-4o	Llama 3.3 70B	Gemini 2.5 Flash	GPT-4o mini
DeepSeek-V4-Pro	3.64 $\pm$ 0.62	3.58 $\pm$ 0.66	3.14 $\pm$ 0.73	2.94 $\pm$ 0.65	2.56 $\pm$ 0.58	2.75 $\pm$ 0.70	2.41 $\pm$ 0.51
GPT-5	3.92 $\pm$ 0.47	3.84 $\pm$ 0.51	3.53 $\pm$ 0.69	3.30 $\pm$ 0.69	3.00 $\pm$ 0.72	2.95 $\pm$ 0.73	2.64 $\pm$ 0.71
Llama 3.3 70B	4.35 $\pm$ 0.69	4.29 $\pm$ 0.82	4.22 $\pm$ 0.84	3.78 $\pm$ 1.04	3.65 $\pm$ 1.01	3.46 $\pm$ 1.09	3.15 $\pm$ 1.09
Qwen3-32B	4.48 $\pm$ 0.66	4.36 $\pm$ 0.84	4.25 $\pm$ 0.93	3.76 $\pm$ 1.04	3.25 $\pm$ 0.94	3.22 $\pm$ 1.02	3.03 $\pm$ 1.00
Ensemble	4.10 $\pm$ 0.52	4.02 $\pm$ 0.62	3.79 $\pm$ 0.71	3.44 $\pm$ 0.79	3.11 $\pm$ 0.74	3.10 $\pm$ 0.83	2.81 $\pm$ 0.76

Appendix

Table 7: Cross-judge consistency of service-agent evaluation. Values are reported as mean

\pm

standard deviation.

Justice Dimension	$\boldsymbol{\hat{\beta}}$	$t$ -statistic	$p$ -value
Interactional	$-0.554^{***}$	$-18.06$	$<0.001$
Procedural	$-0.236^{***}$	$-7.58$	$<0.001$
Distributive	$-0.066^{*}$	$-2.06$	$0.040$
Overall	$-0.285^{***}$	$-10.89$	$<0.001$

Appendix

Table 8: Regression estimates for the association between service agent response length and justice scores. Statistical significance is denoted by

{}^{*}=p<0.05

{}^{**}=p<0.01

, and

{}^{***}=p<0.001

Customer Simulator	Service Agent	Justice				Goodwill	Turns	Avg. Length
Customer Simulator	Service Agent	Interactional	Procedural	Distributive	Total	Goodwill	Turns	Avg. Length
DeepSeek-V4-Pro	GPT-4o	2.87	2.65	3.42	2.98	45.0	6.78	66.39
	GPT-4o mini	2.34	2.20	2.93	2.49	40.2	6.38	56.01
	GPT-5	3.25	3.46	3.88	3.53	36.0	9.06	60.17
	DeepSeek-V4-Pro	3.70	3.18	4.04	3.64	44.1	5.69	58.70
	Gemini 2.5	3.22	2.85	3.63	3.23	33.9	6.51	56.56

Appendix

Table 9: Service agent performance under different customer simulation policies.

Top 10 Categories		Bottom 10 Categories
Category	Justice	Category	Justice
Product Hazard Concerns	3.80	Baggage Follow-up Failures	2.50
Delivery Handling Failures	3.52	Room Assignment Failures	2.58
Product Mismatch	3.50	Pest and Infestation Problems	2.60
Replacement Execution Failures	3.48	Room Environment Problems	2.65
Unsafe Consumables	3.45	Specialist Service Delays	2.67

Appendix

Table 10: Highest and lowest performing complaint categories by total justice.

Figure 6: Cumulative resolution outcomes over dialogue turns.

Model	Airline	Hotel	Restaurant	Retail	Ride-Hailing & Delivery
GPT-4o	0.361 (0.076)	0.383 (0.371)	0.066 (0.186)	0.154 (0.330)	0.125 (0.354)
GPT-4o mini	0.345 (-0.065)	0.336 (-0.027)	0.055 (-0.278)	0.149 (0.218)	0.098 (-0.224)
GPT-5	0.456 (0.908)	0.426 (0.731)	0.064 (0.061)	0.133 (-0.116)	0.122 (0.274)
DeepSeek-V4-Pro	0.351 (-0.009)	0.386 (0.393)	0.060 (-0.080)	0.129 (-0.221)	0.090 (-0.381)
Gemini 2.5 Flash	0.337 (-0.134)	0.292 (-0.393)	0.062 (-0.006)	0.142 (0.075)	0.106 (-0.059)
Llama 3.3 70B	0.130 (-1.945)	0.098 (-2.027)	0.023 (-1.655)	0.055 (-1.836)	0.030 (-1.655)

Appendix

Table 11: Normalized goodwill expenditure across service industries. Values are normalized by the USD 200 discretionary limit; parentheses report within-industry standardized scores.

Model	Turns	Length
GPT-5	10.32	61.74
GPT-4o	7.54	68.40
GPT-4o mini	7.77	56.66
DeepSeek-V4-Pro	6.69	63.61
Gemini 2.5 Flash	8.03	51.37
Llama 3.3 70B	10.13	42.18

Appendix

Table 12: Conversation statistics across evaluated service agents.

Model	Run	Justice				Goodwill	Turns
Model	Run	Interactional	Procedural	Distributive	Total	Goodwill	Turns
DeepSeek-V4-Pro	1	3.47	3.42	4.02	3.64	42.9	6.54
	2	3.43	3.37	3.98	3.59	49.2	6.64
	3	3.52	3.28	3.98	3.59	40.9	6.44
	4	3.45	3.28	3.99	3.57	43.3	6.62
	5	3.38	3.25	3.83	3.49	42.2	6.79

Appendix

Table 13: Run-to-run consistency across five independent simulations.

Annotator Pair	Interactional	Procedural	Distributive
Judge 1 – Judge 2	0.870	0.903	0.766
Judge 1 – Judge 3	0.779	0.810	0.758
Judge 2 – Judge 3	0.774	0.763	0.791
Mean pairwise	0.808	0.825	0.772

Appendix

Table 14: Human annotator agreement measured by pairwise Spearman correlation.

Figure 7: Human evaluation of customer simulator realism.

Judge Model	Interactional		Procedural		Distributive
Judge Model	Spearman	MAE	Spearman	MAE	Spearman	MAE
GPT-5	0.831	0.348	0.813	0.420	0.789	0.367
GPT-4o	0.866	0.343	0.778	0.468	0.766	0.477
GPT-4o mini	0.839	0.297	0.787	0.378	0.735	0.632
Llama 3.3 70B	0.793	0.700	0.778	0.755	0.761	0.585
Qwen3-32B	0.795	0.718	0.764	0.728	0.738	0.553

Appendix

Table 15: Validation of candidate LLM judges against human annotations.

Industry	Selected Customer Care Accounts	# Raw Source Dialogues
Airlines	AmericanAir, Delta, JetBlue, SouthwestAir	78,864
Hotels	HiltonHelp, IHGService	2,460
Restaurants	McDonalds, TacoBellTeam, KFC_UKI_Help	13,247
Retail	AskTarget, Walmart	13,541
Ride-Hailing and Delivery	Uber_Support, AskLyft	52,029

Appendix

Table 16: Selected customer care accounts and raw dialogue counts used for customer journey map construction.

Benchmark Scale	Count
Scenarios	2,500
Service industries	5
Customer journey stages	37
Complaint categories	184
Industry Coverage	Stage / Category
Airline	8 / 44

Appendix

Table 17: Summary statistics of ServiceBench . Detailed complaint categories are provided in Appendix C.7 .

Component	Provenance	Description
Source complaint dialogues	Collected	Public customer complaint dialogues collected from X, providing the original facts of service failures.
Customer journey maps	Data driven and manually constructed	Service stages and complaint categories developed through topic discovery and expert guided coding.
Service event sequences	Reconstructed	Plausible sequences of events leading to each complaint, reconstructed from source dialogues while preserving the original facts.
Customer experience background and resolution expectation	Generated with manual refinement	Detailed customer experiences and resolution expectations developed from the event sequences and refined for realism and consistency.
Company case record	Generated with manual refinement	Available company information inferred from the customer experience using industry specific schemas and manually refined for realism and consistency.

Appendix

Table 18: Provenance of the main components used in ServiceBench scenario construction.

Communication Style	Prompt Template
Assertive-Warm	Maintain a clear, cooperative stance and focus on fair resolution. State the problem and what would feel fair, while still showing respect for the agent’s effort. Give relevant details when needed and respond positively when the agent is helpful. Ask for clarification or a better next step when the response is vague or insufficient.
Assertive-Cold	Maintain a firm, skeptical stance and focus on concrete action. State the problem and expected outcome clearly, with little effort to sound friendly or cooperative. Treat vague explanations or unsupported offers as insufficient, and ask for accountability when the agent does not provide a credible path forward. Keep messages controlled but demanding.
Passive-Warm	Maintain a polite, patient, and cooperative stance. Use softer language and avoid direct confrontation. Share concerns gradually, with strong attention to whether the agent understands and treats the issue with care. Respond more openly when the agent shows understanding or provides a credible explanation. When the response feels unclear or insufficient, continue expressing concern gently rather than making strong demands.
Passive-Cold	Maintain a brief, withdrawn, and skeptical stance. Give limited details unless the agent asks specific, low-pressure follow-up questions. Show dissatisfaction through short replies or hesitation rather than direct confrontation. Become less responsive when the agent gives generic or repetitive responses. Engage more only when the agent shows that the concern is taken seriously and provides a credible path forward.

Appendix

Table 19: Communication style prompt.

Panel of Figure 8 — Figure 8: Overview of the airline customer journey map.

Panel of Figure 9 — Figure 9: Overview of the hotel customer journey map.

Panel of Figure 10 — Figure 10: Overview of the restaurant customer journey map.

Panel of Figure 11 — Figure 11: Overview of the retail customer journey map.

Panel of Figure 12 — Figure 12: Overview of the ride-hailing and delivery customer journey map.

Panel of Figure 18 — Figure 18: Case Study 1: DeepSeek-V4-Pro as the service agent. The agent responds with restrained and formulaic language despite the customer describing an unsafe and humiliating experience. Although goodwill is offered, the response does not develop into stronger relational care or concrete protective action that would demonstrate appropriate seriousness.

Panel of Figure 19 — Figure 19: Case Study 2: DeepSeek-V4-Pro as the service agent. The agent provides material recovery through a refund and goodwill credit, but offers limited relational care after the customer describes sleep loss and disrupted plans. It also does not explain what happened or how the issue will be handled, treating the remedy itself as the primary path to closure.

Panel of Figure 20 — Figure 20: Case Study 3: GPT-5 as the service agent. The agent offers premature goodwill for a reservation failure that primarily requires explanation, then continues to request incremental details after the core issue is already clear. The dialogue appears procedurally organized, but becomes overly information seeking rather than focused on clarifying the failure and next steps.

Panel of Figure 21 — Figure 21: Case Study 4: Qwen3-32B as the service agent. The agent provides limited relational care and procedural clarity, instead treating the complaint as a transactional negotiation. As the customer continues to challenge the adequacy of the remedy, the agent repeatedly increases goodwill credits rather than addressing the underlying disclosure failure or offering a consistent resolution.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 681 papers

Jun 1, 2026

CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation

Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.

Danqing Wang, Akshay Sivaraman, Lei Li

Carnegie Mellon University

Human Behavior Simulation LLM Agent Evaluation AI Agent Benchmarks User Simulation

Ask PDF

Mar 30, 2026

CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments

The increasing agentic capabilities of Large Language Models (LLMs) have enabled their deployment in real-world applications, such as cloud services, where customer-assistant interactions exhibit high technical complexity and long-horizon dependencies, making robustness and resolution efficiency critical for customer satisfaction. However, existing benchmarks for LLM-based agents largely rely on synthetic environments that fail to capture the diversity and unpredictability of authentic customer inputs, often ignoring the resolution efficiency essential for real-world deployment. To bridge this gap, we introduce CirrusBench, a novel evaluation framework distinguished by its foundation in real-world data from authentic cloud service tickets. CirrusBench preserves the intricate multi-turn logical chains and realistic tool dependencies inherent to technical service environments. Moving beyond execution correctness, we introduce novel Customer-Centric metrics to define agent success, quantifying service quality through metrics such as the Normalized Efficiency Index and Multi-Turn Latency to explicitly measure resolution efficiency. Experiments utilizing our framework reveal that while state-of-the-art models demonstrate strong reasoning capabilities, they frequently struggle in complex, realistic multi-turn tasks and fail to meet the high-efficiency standards required for customer service, highlighting critical directions for the future development of LLM-based agents in practical technical service applications. CirrusBench evaluation framework is released at: https://github.com/CirrusAI

Yi Yu, Guangquan Hu, Chenghuang Shen +15

School of Mathematics and Sciences, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai, China +2

LLM Agent Efficiency LLM Agent Evaluation LLM Agents Customer Support Automation Long-Horizon LLM Agents

Ask PDF

Apr 10, 2026

SAGE: A Service Agent Graph-guided Evaluation Benchmark

The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominantly rely on static paradigms and single-dimensional metrics, failing to account for diverse user behaviors or the strict adherence to structured Standard Operating Procedures (SOPs) required in real-world deployments. To bridge this gap, we propose SAGE (Service Agent Graph-guided Evaluation), a universal multi-agent benchmark for automated, dual-axis assessment. SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance and comprehensive path coverage. We introduce an Adversarial Intent Taxonomy and a modular Extension Mechanism, enabling low-cost deployment across domains and facilitating automated dialogue data synthesis. Evaluation is conducted via a framework where Judge Agents and a Rule Engine analyze interactions between User and Service Agents to generate deterministic ground truth. Extensive experiments on 27 LLMs across 6 industrial scenarios reveal a significant Execution Gap'' where models accurately classify intents but fail to derive correct subsequent actions. We also observe Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures under high adversarial intensity. Code and resources are available at https://anonymous.4open.science/r/SAGE-Bench-4CD3/.

Ling Shi, Yuqin Dai, Ziyin Wang +7

Tianjin University, Tianjin, China · Tsinghua University, Beijing, China · Beihang University, Beijing, China +3

LLM Agent Evaluation Agent Evaluation AI Agent Benchmarks Customer Support Automation

Ask PDF

Abstract

LLM Agents

Explore similar work

CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation

CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments

SAGE: A Service Agent Graph-guided Evaluation Benchmark