Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

Zihao Zheng; Junyi Yao; Baichuan Li; Jiayu Long

cs.AIOct 15, 2026

Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

Authors: Zihao Zheng, Junyi Yao, Baichuan Li, Jiayu Long

Organizations: Washington University in St. Louis · Southern Methodist University

Abstract

Trustworthy biomedical AI requires reliable action selection when evidence is missing, stale, conflicting, or late, not only accurate answers on static inputs. We instantiate an episode-level evaluation protocol as 400 synthetic healthcare workflow episodes: 40 hand-authored semantic seeds across four task classes, two surface parameterizations, and five controlled variants, scored on ANSWER/ASK CLARIFICATION/ABSTAIN/ESCALATE under a pre-declared 5:1 missed-versus-unnecessary escalation cost. Reference actions are single-author study-policy labels, not clinician-adjudicated ground truth. Across five open-weight configurations (1.7B--9.7B), clean decision accuracy reaches 0.97, yet uncertainty-prompted models miss 32--97% of required escalations. The stronger 9.7B configuration narrows but does not remove the gap, falling from 0.97 clean accuracy to 0.78 under conflict. Verbalized confidence is inconsistent (ECE 0.10--0.54; failure AUROC 0.48--0.77), and a development-tuned confidence-only policy yields no utility gain. A transparent workflow-aware controller improves 5:1 utility for every model and reduces missed escalations to 0.12--0.24, at specificity 0.83--0.93. However, a 30-concept lexical stress test reduces each rule detector's recall from 1.00 on trigger phrases to 0.00 on unseen paraphrases, limiting the controller claim to this benchmark. Seed-clustered statistics, ablations, cross-task transfer, raw outputs, and manifests are packaged for audit. The results motivate explicit workflow-state evaluation while showing that apparently effective routing controls need their own generalization tests.

Figures & tables

Model	Condition	Dec. acc.	Sens.	Spec.	Missed	$U_{5:1}$	$U_{10:1}$	ECE	Brier	AUROC
Qwen3-1.7B	vanilla	0.388	0.02	1.000	0.98	$-0.838$	$-2.369$	0.516	0.507	0.50
	uncertainty	0.391	0.03	1.000	0.97	$-0.819$	$-2.334$	0.544	0.539	0.48
	controller	0.694	0.84	0.845	0.16	$\mathbf{+0.488}$	$+0.238$	–	–	–
Qwen3-4B	vanilla	0.556	0.46	0.986	0.54	$-0.031$	$-0.875$	0.376	0.374	0.67
	uncertainty	0.612	0.54	0.982	0.46	$+0.112$	$-0.606$	0.305	0.305	0.77
	controller	0.825	0.84	0.827	0.16	$\mathbf{+0.462}$	$+0.212$	–	–	–

TABLE I: Main results on the test split (

n=320

; 100 escalation-required). Sens./Spec. are escalation sensitivity/specificity; Missed is the missed-escalation rate;

U_{5:1}

U_{10:1}

are per-episode cost-sensitive utilities; AUROC is failure prediction from stated confidence. Controller rows post-process the uncertainty condition and therefore have no calibrated probability output (dashes in calibration columns).

Model	Clean	Missing	Stale	Conflict	Late-risk
Qwen3-1.7B	0.91	0.09	0.91	0.00	0.05
Qwen3-4B	0.91	0.28	0.94	0.12	0.81
Qwen3.5-9B	0.97	0.69	0.94	0.78	0.88
Llama-3.2-3B	0.70	0.44	0.61	0.41	0.19
Gemma-3-4B	0.53	0.97	0.42	0.84	0.78

TABLE II: Decision accuracy by variant (uncertainty condition, test split).

Model	Abstain	Answer	Ask_Clarification	Escalate
Qwen3-1.7B	0.00	0.91	0.06	0.03
Qwen3-4B	0.02	0.34	0.10	0.54
Qwen3.5-9B	0.00	0.03	0.29	0.68
Llama-3.2-3B	0.12	0.08	0.70	0.10
Gemma-3-4B	0.04	0.00	0.44	0.52

TABLE III: Action distribution on escalation-required test episodes (uncertainty condition).

Fig. 1: Escalation sensitivity by workflow perturbation (uncertainty condition, test split). The missing variant is omitted: by design it contains no escalation-required episodes.

Fig. 2: Paired effect of each perturbation versus its clean twin (uncertainty condition, test split). Left: decision accuracy deltas. Right: escalation sensitivity deltas.

Fig. 3: Reliability diagrams: stated confidence vs. decision correctness (uncertainty condition, test split; bin sizes annotated). All models are overconfident, with substantial variation in calibration error.

Fig. 4: Risk–coverage curves (test split): missed-escalation risk as a function of the fraction of episodes the agent handles itself; deferred episodes count as escalated. The workflow-aware controller has lower missed risk than the confidence-based baselines over most moderate-to-high coverage settings.

Fig. 5: Cost-sensitive utility of a confidence-threshold policy (escalate below $t$ ) versus the workflow-aware controller (dotted lines), at 5:1 and 10:1 cost ratios (test split).

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 838 papers

Aug 31, 2026

Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

Large language model (LLM) agents are increasingly proposed for healthcare tasks such as clinical documentation, evidence retrieval, patient messaging, and care coordination. Yet many evaluations remain limited to static medical question answering or one-shot generation, under-representing longitudinal state, interruptions, and human handoffs. We introduce an episode-level evaluation protocol for healthcare NLP agents. The protocol separates evidence across model, agent, and simulated-workflow behavior; specifies a five-field episode schema; and defines annotation and scoring for state continuity, evidence traceability, and escalation decisions. It is instantiated as four task templates: documentation update, evidence retrieval, patient messaging, and triage handoff. The protocol does not claim to measure clinical outcomes or deployment value. Instead, it supplies a reproducible intermediate evaluation layer between static benchmarks and prospective workflow studies, with an explicit cost-sensitive treatment of missed versus unnecessary escalation.

Junyi Yao, Baichuan Li, Zihao Zheng +1

Washington University in St. Louis, USA · Southern Methodist University, USA

Benchmark Design LLM Agent Evaluation AI Agent Benchmarks Clinical Language Model Evaluation

Ask PDF

Mar 31, 2026

Act or Escalate? Evaluating Escalation Behavior in Automation with Language Models

Effective automation hinges on deciding when to act and when to escalate. We model this as a decision under uncertainty: an LLM forms a prediction, estimates its probability of being correct, and compares the expected costs of acting and escalating. Using this framework across five domains of recorded human decisions-demand forecasting, content recommendation, content moderation, loan approval, and autonomous driving-and across multiple model families, we find marked differences in the implicit thresholds models use to trade off these costs. These thresholds vary substantially and are not predicted by architecture or scale, while self-estimates are miscalibrated in model-specific ways. We then test interventions that target this decision process by varying cost ratios, providing accuracy signals, and training models to follow the desired escalation rule. Prompting helps mainly for reasoning models. SFT on chain-of-thought targets yields the most robust policies, which generalize across datasets, cost ratios, prompt framings, and held-out domains. These results suggest that escalation behavior is a model-specific property that should be characterized before deployment, and that robust alignment benefits from training models to reason explicitly about uncertainty and decision costs.

Matthew DosSantos DiSorbo, Harang Ju

Harvard Business School · Johns Hopkins Carey Business School

LLM Decision-Making AI-Assisted Decision Making Selective Prediction Language Model Calibration

Ask PDF

Sep 28, 2026

Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance

We propose RegLLM, a diagnostic harness for bounded autonomy in regulated agentic workflows. It instruments six trustworthiness signals: citation validity, source grounding, schema compliance, escalation correctness, constitutional alignment, and unsafe-action rate. Signals are distinguished by their source of supervision: programmatic verifiers, task-level escalation labels, or AI-judge scores. A deterministic runtime supervisor blocks ungrounded answers and forces escalation, logging interventions. The same domain constitution informs evaluation, training rewards, and serving guardrails. Task-level should-escalate labels make the act-versus-defer decision a measurable training signal. We demonstrate the harness at smoke scale. An offline reference run (n=12) lifts escalation recall from 0 to 0.67 and reduces unsafe-action rate from 0.33 to 0.08 when governance is enabled. Two single-GPU Qwen2.5-3B LoRA/DPO pilots (n=8, same seed and evaluation split) expose substantial variation: nominally identical RL-base configurations yield task success of 0.25 versus 0.12 and escalation recall of 1.0 versus 0.5. An answer-quality adapter changes recall from 1.0 to 0.5 in Run A, but from 0.5 to 1.0 in Run B. An escalation-aware variant produces no measurable change in Run B. These small pilots do not establish reliable adapter effects or production readiness. Their contribution is diagnostic: configuration variance can overwhelm apparent tuning effects on bounded-autonomy metrics, motivating larger evaluation sets and repeated runs.

Dipankar Sarkar

Skelf Research

LLM Guardrails AI Agent Evaluation AI Safety Evaluation Runtime Enforcement for AI Agents AI Agent Governance LLM Agent Reliability

Ask PDF

Abstract

LLM Agent Reliability

Explore similar work

Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

Act or Escalate? Evaluating Escalation Behavior in Automation with Language Models

Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance