SAGE: A Service Agent Graph-guided Evaluation Benchmark

Ling Shi; Yuqin Dai; Ziyin Wang; Ning Gao; Wei Zhang; Chaozheng Wang; Yujie Wang; Wei He

cs.AIApr 10, 2026

SAGE: A Service Agent Graph-guided Evaluation Benchmark

Authors: Ling Shi, Yuqin Dai, Ziyin Wang, Ning Gao, Wei Zhang, Chaozheng Wang, Yujie Wang, Wei He,

Organizations: Tianjin University, Tianjin, China · Tsinghua University, Beijing, China · Beihang University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China ·

Abstract

The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominantly rely on static paradigms and single-dimensional metrics, failing to account for diverse user behaviors or the strict adherence to structured Standard Operating Procedures (SOPs) required in real-world deployments. To bridge this gap, we propose SAGE (Service Agent Graph-guided Evaluation), a universal multi-agent benchmark for automated, dual-axis assessment. SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance and comprehensive path coverage. We introduce an Adversarial Intent Taxonomy and a modular Extension Mechanism, enabling low-cost deployment across domains and facilitating automated dialogue data synthesis. Evaluation is conducted via a framework where Judge Agents and a Rule Engine analyze interactions between User and Service Agents to generate deterministic ground truth. Extensive experiments on 27 LLMs across 6 industrial scenarios reveal a significant Execution Gap'' where models accurately classify intents but fail to derive correct subsequent actions. We also observe Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures under high adversarial intensity. Code and resources are available at https://anonymous.4open.science/r/SAGE-Bench-4CD3/.

Figures & tables

Figure 1. Service Agent SOP Example (Telecom Scenario).

Figure 2. Overview of SAGE evaluation framework.

Model		AVG Score			OA on 6 Scenarios						Format	Chat
Name	Params	OA	Logic	Chat	ER	TP	PS	LD	AR	OE	Error	Length
Closed-Source Large Language Models
Claude-Sonnet-4.5	-	71.56 2	70.10 3	77.38 1	71.21	75.03	75.07	67.30	66.52	74.24	0.75 %	93.42
Claude-Opus-4.5	-	72.62 1	72.10 1	74.68 2	72.73	73.12	73.25	71.25	70.12	75.22	0.94 %	43.49
GPT-4.1	-	66.79 6	68.31 4	60.70 8	70.32	66.90	65.46	62.69	63.68	71.69	0.04 %	18.59
Gemini-2.5-Pro	-	66.90 5	65.69 8	71.73 4	69.46	66.42	69.60	64.92	60.71	70.28	0.00 %	53.33

Table 1. Main results across 6 scenarios (0-100 Scale). OA : Overall Assessment Score; Logic : Logical Compliance Score; Chat : Conversational Quality Score. The superscripts indicate the ranking within Closed-Source and Open-Source groups, respectively. Format Error : Percentage of outputs failing JSON parsing. Chat Length : Average character count of the response field.

Figure 6. Correlation analysis between the Chat Quality Score and its five sub-dimensions. The deviations from the regression lines (e.g., Doubao-Seed-1.8’s high Linguistic vs. low Instruction scores) highlight specific model characteristics.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	OA Score				Logic Score				Chat Score				Chat Length
Scenario	T1	T5	T10	T15	T1	T5	T10	T15	T1	T5	T10	T15	T1	T5	T10	T15
AirlineRefund	61.1	61.9	59.8	57.2	60.7	61.4	58.7	55.7	63.1	64.1	64.3	63.2	37	36	35	34
EcommerceRefund	66.9	67.9	68.1	66.5	67.5	68.5	68.5	66.6	64.6	65.5	66.5	66.2	37	34	38	36
LogisticsDelivery	59.5	60.5	60.7	60.8	58.9	59.9	60.0	60.7	62.0	63.1	63.3	61.4	40	39	40	45
OnlineEducation	67.0	69.4	71.5	73.1	67.5	69.9	72.6	75.0	65.1	67.3	67.3	65.5	35	35	34	36
PropertyService	64.1	64.7	59.8	56.9	63.9	64.6	58.6	55.5	64.6	65.4	64.4	62.8	30	30	32	35

Appendix

Table 2. Turn-by-Turn Performance Analysis by Scenario. (0-100 Scale).

Scenario	Overall			Logic			Chat Quality			JSON Error (%)
Scenario	Zero	Weak	Strong	Zero	Weak	Strong	Zero	Weak	Strong	Zero	Weak	Strong
AirlineRefund	49.7	63.2	60.9	46.3	63.2	60.3	63.3	63.0	63.4	1.6	1.8	3.4
E-commerceRefund	73.1	68.8	58.9	74.8	69.7	58.1	66.2	65.2	62.0	1.0	1.5	3.9
LogisticsDelivery	78.2	59.9	56.4	84.0	59.5	54.5	55.2	61.6	63.6	0.6	2.0	3.8
OnlineEducation	75.7	74.4	62.6	78.1	76.7	62.0	66.1	65.1	64.9	2.3	2.9	4.5
PropertyService	65.1	61.4	67.0	64.4	60.3	68.8	67.9	65.9	60.0	0.9	1.0	3.1

Appendix

Table 3. Adversarial Intensity Analysis: Average Performance Across All Models (0-100 Scale).

Scenario	Logic Ability			Chat Quality
Scenario	Path	Action	Class.	Ling.	Anth.	Cont.	Satis.	Instr.
AirlineRefund	64.4	29.4	72.5	68.3	56.2	60.6	62.0	73.1
E-commerceRefund	66.0	29.5	88.0	70.1	58.8	62.1	64.0	71.4
LogisticsDelivery	64.3	32.3	66.9	68.9	55.5	58.6	60.0	71.6
OnlineEducation	67.0	43.0	80.2	71.6	62.3	60.0	61.7	73.0
PropertyService	67.2	57.1	64.0	71.3	58.3	60.8	62.1	75.0

Appendix

Table 4. Sub-dimension Analysis: Average Performance Across All Models. Path: Path Correctness; Finals: Finals Correctness; Class.: Classification Accuracy; Ling.: Linguistic Quality; Anth.: Anthropomorphism; Cont.: Content Utility; Satis.: User Satisfaction; Instr.: Instruction Compliance (0-100 Scale).

Panel of Figure 10 — Figure 10. Standard Operating Procedures (SOPs) for Six Industrial Scenarios evaluated in SAGE. These directed graphs define the logical constraints and action spaces for each domain.(continued)

Panel of Figure 11 — Figure 11. Standard Operating Procedures (SOPs) for Six Industrial Scenarios.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

Automated Benchmark Auditing for AI Agents and Large Language Models
May 25, 2026Junlin Wang, Federico Bianchi, Shang Zhu +4LLM Evaluation Benchmark Auditing
Duke University · Together AI · Stanford University
Benchmark Everything Everywhere All at Once
Jun 4, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5LLM Evaluation Benchmark Design
MMLab, The Chinese University of Hong Kong · CPII under InnoHK · The Chinese University of Hong Kong, Shenzhen +3
CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems
Apr 14, 2026Jingbo Yang, Guanyu Yao, Bairu Hou +5LLM Evaluation LLM-as-a-Judge
University of California, Santa Barbara · Cresta

May 25, 2026J/K move · Enter open · S save

Automated Benchmark Auditing for AI Agents and Large Language Models

Junlin Wang, Federico Bianchi, Shang Zhu, Fan Nie+3

Duke University · Together AI · Stanford University

LLM Evaluation Benchmark Auditing Software Engineering Benchmarks AI Agent Benchmarks Benchmark Validity

Open paper AskPDF arXiv

Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environment specifications, and brittle evaluation logic that human annotation cannot reliably catch. We introduce Auto Benchmark Audit (ABA), an agentic framework that systematically audits individual benchmark tasks, uncovering issues such as hidden environment dependencies, specification gaps, and limited grading logic. We run ABA on a collection of frontier LLM benchmarks and previous NeurIPS publications, totaling 168 benchmarks across nine domains. Across this corpus, ABA identifies critical issues including ambiguous task design, execution environment conflicts, and incorrect ground truths in over 25.7% of the evaluated tasks. The precision of these automated audits is validated by expert review and independent third-party reports such as upstream PRs. Crucially, we demonstrate that these problematic tasks severely distorts capability assessments for agents and LLMs: filtering out these tasks with issues shifts model rankings and increases average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively. We release the agentic tool and all task annotations to support the future development of frontier benchmarks.

Abstract

LLM Agent Evaluation

Explore similar work

Automated Benchmark Auditing for AI Agents and Large Language Models

Benchmark Everything Everywhere All at Once

CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems