Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation

Hongliu Cao; Ilias Driouich; Eoin Thomas

cs.AIMar 3, 2026

Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation

Authors: Hongliu Cao, Ilias Driouich, Eoin Thomas

Organizations: Amadeus France

Abstract

Large Language Model (LLM)-based agents are increasingly adopted in high-stakes settings, but current benchmarks evaluate mainly whether a task was completed, not how. We introduce Procedure-Aware Evaluation (PAE), a framework that formalizes agent procedures as structured observations and exposes consistency relationships between what agents observe, communicate, and execute. PAE evaluates agents along complementary axes (Utility, Efficiency, Interaction Quality, Procedural Integrity) and applies multi-dimensional gating that categorically disqualifies corrupt outcomes. Evaluating state-of-the-art LLM agents on tau-bench yields findings at the axis, compliance, and benchmark levels. At the axis level, the dimensions capture non-redundant failure modes: utility masks reliability gaps, speed does not imply precision, and conciseness does not predict intent adherence. At the procedural compliance level, 27-78% of benchmark reported successes are corrupt successes concealing violations across interaction and integrity. Furthermore, gating substantially collapses Pass^4 rate and affects model rankings. The analysis of corrupt success cases reveals distinctive per-model failure signatures: GPT-5 spreads errors across policy, execution, and intent dimensions; Kimi-K2-Thinking concentrates 78% of violations in policy faithfulness and compliance; and Mistral-Large-3 is dominated by faithfulness failures. At the benchmark level, our analysis exposes structural flaws in the benchmark design, including task scope gaps, contradictory reward signals, and simulator artifacts that produce accidental successes.

Figures & tables

Utility
	Retail			Airline
Metric	GPT-5	Kimi-K2	Mistral	GPT-5	Kimi-K2	Mistral
Success Rate $\uparrow$	0.79	0.61	0.68	0.60	0.48	0.40
Passˆ4 $\uparrow$	0.58	0.31	0.46	0.44	0.28	0.18
Pass@4 $\uparrow$	0.92	0.83	0.84	0.74	0.62	0.62
Efficiency

Table 1: Full PAE results across Retail and Airline domains (

k{=}4

trials).

\uparrow

\downarrow

indicates preferred direction; best per metric in bold . All models achieve

I_{id}=I_{pii}=I_{tone}=1.00

(omitted).

Validated Error Type	Total	Confirmed	Accuracy
Automatic proxy (action set comparison)
UNNECESSARY_CALL	67	60	89.6%
HARMFUL_DISALLOWED_EXECUTION	171	155	90.6%
Human spot-check (random sample)
All other error types	150	134	89.3%

Table 2: LLM judge validation. Top : automatic proxy via action set comparison (excess actions confirm execution-related errors). Bottom : human spot-check on 150 randomly sampled assessments. Accuracy = confirmed / total.

	Retail			Airline
Metric	GPT-5	Kimi-K2	Mistral	GPT-5	Kimi-K2	Mistral
Success Rate (Original)	0.79	0.61	0.68	0.60	0.48	0.40
Success Rate (Gated)	0.58	0.27	0.16	0.44	0.30	0.09
$\Delta$	-0.21	-0.34	-0.52	-0.16	-0.18	-0.31
Passˆ4 (Original)	0.58	0.31	0.46	0.44	0.28	0.18
Passˆ4 (Gated)	0.24	0.04	0.03	0.18	0.06	0.02

Table 3: Standard utility versus gated utility requiring six-dimension compliance.

\Delta

shows the absolute drop; largest drop per metric group in bold .

	GPT-5	Kimi-K2-Think.	Mistral
Judge Precision	93.8%	94.4%	95.2%
Policy Faith. $I_{pf}$	29.7%	47.8%	26.3%
Data Faith. $I_{df}$	5.4%	17.4%	28.4%
Policy Compl. $I_{pc}$	35.1%	30.4%	24.2%
Exec. Consist. $I_{ec}$	18.9%	2.2%	14.7%
Intent/Question	10.8%	2.2%	6.3%

Table 4: Manual validation and error distribution across models (Airline domain). Each model’s dominant error type in bold . Intent/Question combines

I_{intent}

(user intent adherence) and

I_{qf}

(question fulfillment).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Error Types
Intent Adherence $I_{intent}$	USER_CONSTRAINT_VIOLATED , USER_INPUT_MISREAD
Policy Compliance $I_{pc}$	HARMFUL_DISALLOWED_EXECUTION , DISALLOWED_DECISION , MISSING_REQUIRED_CHECK
Policy Faithfulness $I_{pf}$	POLICY_HALLUCINATION
Execution Consistency $I_{ec}$	CLAIMED_NOT_EXECUTED , EXECUTED_NOT_CLAIMED
Data Faithfulness $I_{df}$	DATA_HALLUCINATION
Agent Efficiency $I_{eff}$	REDUNDANT_IDENTICAL_CALL , UNNECESSARY_CALL

Appendix

Table 5: Error taxonomy mapping each LLM-as-judge metric to its typed error codes for failure attribution.

Error Pattern	Cases	Representative Examples
$I_{pc}$ : Policy Compliance	13	Offered unauthorized compensation or proceeded with actions not permitted by policy. Unnecessary transfers to human agents.
$I_{pf}$ : Policy Faithfulness	11	Misstated cancellation rules or insurance coverage scope. Claimed benefits not supported by policy text.
$I_{ec}$ : Exec. Consistency	7	User confirmed transfer or cancellation; agent did not execute the corresponding tool call.
$I_{qf}$ : Question Fulfillment	4	Transferred user to human agent without first answering the pending question.
$I_{df}$ : Data Faithfulness	2	Communicated incorrect data (e.g., wrong fare amount) contradicted by tool output.

Appendix

Table 6: Error pattern distribution and representative examples for GPT-5 corrupt success cases. One case can have multiple errors.

Error Pattern	Cases	Representative Examples
$I_{df}$ : Data Faithfulness	27	Arithmetic: Stated $282 total when tools imply$ 542 (Case 9). Fabrication: Claimed flight canceled and insured; tools show landed, insurance=no (Case 3). Invented flight HAT385 and $20 price without tool data (Case 63).
$I_{pf}$ : Policy Faithfulness	25	Baggage: Stated Gold gets 1 free bag; policy says 3-4 (Case 39). Insurance: Claimed insurance covers refund for missed meeting; policy covers health/weather only (Case 18). Economy: Offered to modify basic economy; policy forbids (Case 13).
$I_{pc}$ : Policy Compliance	23	Compensation: Offered $100 certificate to ineligible regular-tier user (Case 3). Modification: Changed destination LGA$ \to$ JFK; policy forbids origin/destination changes (Case 29). Cancellation: Cancelled despite ineligible reason (Case 14).
$I_{ec}$ : Exec. Consistency	14	User confirmed transfer; agent did not call transfer_to_human (Cases 1, 44, 59). 4 cases involve simulation termination markers, not agent failures.
$I_{intent}$ / $I_{qf}$ : Intent/Fulfillment	6	Offered transfer after user explicitly refused (Case 5). User agreed $69; agent charged$ 209 (Case 10).

Appendix

Table 7: Error pattern distribution and representative examples for Mistral-Large-3 corrupt success cases. One case can have multiple errors.

Error Pattern	Cases	Representative Examples
$I_{pf}$ : Policy Faithfulness	22	Unsourced: Made categorical cancellation eligibility statement without policy reference (Case 5). Incomplete: Stated “basic economy cannot be modified” without mentioning cabin upgrade exception (Case 12). Invented: Claimed “48-hour free cancellation window” not in policy (Case 3).
$I_{pc}$ : Policy Compliance	14	Unauthorized: Offered destination change when policy only permits date/time modifications (Case 8). Transfer: Escalated unnecessarily; request was within policy scope (Case 22). Modification: Proposed flight change for basic economy outside 24-hour window (Case 11).
$I_{df}$ : Data Faithfulness	8	Fabrication: Invented specific departure time not in tool output (Case 9). Incorrect: Stated wrong refund amount (Case 6). Communicated incorrect price difference (Case 16).
$I_{ec}$ / $I_{intent}$ : Exec./Intent	2	User confirmed action; agent did not execute tool call (Case 29). Proceeded with action user did not explicitly confirm (Case 17).

Appendix

Table 8: Error pattern distribution and representative examples for Kimi-K2-Thinking corrupt success cases. One case can have multiple errors.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

Apr 17, 2026

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, but this assumption has rarely been validated against human annotation. We introduce AgentProp-Bench, a 2,000-task benchmark with 2,300 traces across four domains, nine production LLMs, and a 100-label human-validated subset. We quantify judge reliability, characterize error propagation, and evaluate a runtime mitigation. Substring-based judging agrees with human annotation at kappa=0.049 (chance-level); a three-LLM ensemble reaches kappa=0.432 (moderate) with a conservative bias. Under validated evaluation, a parameter-level injection propagates to a wrong final answer with human-calibrated probability approximately 0.62 (range 0.46-0.73 across models). Rejection (catching bad parameters) and recovery (correcting after acceptance) are independent model capabilities (Spearman rho=0.126, p=0.747). A tuned runtime interceptor reduces hallucination on GPT-4o-mini by 23.0 percentage points under a concurrent n=600 control, but shows no significant effect on Gemini-2.0-Flash, whose aggressive parameter rejection eliminates the target failure mode. All code, data, traces, and human labels are released at https://github.com/bhaskargurram-ai/agenthallu-bench.

Bhaskar Gurram

Zasti Inc., Ashburn, VA, USA

AI Agent Reliability Tool-Augmented Language Model Agents LLM-as-a-Judge LLM Agent Evaluation AI Agent Benchmarks LLM Reliability+2

Ask PDF

Jun 1, 2026

From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents

LLM agents can fail silently by asserting task completion when the environment state shows otherwise. We study this failure mode, false success, across two agent benchmarks: 9,876 tau2-bench trajectories from 8 model families and 1,879 AppWorld trajectories from 4 model families with text-independent ground truth. False success is common but varies by setting: 45--48% of failures in single-control tau2-bench domains, 3% in dual-control telecom, and 75.8% among AppWorld self-assessing coding-agent trajectories with explicit status claims. LLM judges fail reliably: no configuration across 5 judges, 5 prompt strategies, and full task specifications exceeds AUROC 0.65 on tau2-bench, and the same judges reach only 0.54 AUROC on AppWorld API-call traces. Judges rely on surface completion proxies -- confident closing language in tau2-bench and coarse action-sequence volume in AppWorld -- rather than verified state changes. Lightweight TF-IDF detectors achieve task-disjoint AUROC 0.83 on tau2-bench and 0.95 on AppWorld, recovering 4--8x more false successes than the best judge at the same flag rate with 3,300x lower latency. These results suggest that production monitoring should use lightweight, domain-calibrated detectors as triage signals rather than relying on LLM judges as the primary monitor for false success.

Laksh Advani

University of Colorado

Agent Failure Analysis AI Agent Reliability LLM Agent Evaluation AI Agent Monitoring

Ask PDF

Sep 10, 2026

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

Comparing and selecting task-oriented LLM agents increasingly relies on a low-cost offline evaluation gate: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. We introduce GAUGE, a reusable offline protocol that measures whether this gate's ranking matches a grounded verifiable reward across 25 agents from six providers on the

τ^2

-bench and SimulatorArena benchmarks, separating two kinds of evaluation validity that release practices conflate: ranking validity and construct validity. First, a satisfaction-success gap: satisfaction carries essentially no information about task success, as conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer's task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated. Second, while the gate's ranking is robust across the broad capability span, it loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from

<

1% on wide-reward pairs to 31% on close pairs. The gate is thus human-validated yet mis-anchored. As a remedy, we propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions.

Umesh Bodhwani, Thanh Tran, Kai Wei

Amazon

LLM-as-a-Judge LLM Agent Evaluation AI Agent Evaluation LLM Agent Reliability Agent Benchmarks

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents