Asclepius: An Adaptive Harness for Long-Horizon Clinical Agents

Grace Chang Yuan; Xiaoman Zhang; Sung Eun Kim; Luyang Luo; Pranav Rajpurkar

cs.AISep 11, 2026

Asclepius: An Adaptive Harness for Long-Horizon Clinical Agents

Authors: Grace Chang Yuan, Xiaoman Zhang, Sung Eun Kim, Luyang Luo, Pranav Rajpurkar

Organizations: Massachusetts Institute of Technology, Cambridge, MA · Department of Biomedical Informatics, Harvard Medical School, Boston, MA

Abstract

LLM agents are predominantly benchmarked on short, single-task trajectories, yet real deployments run for hours under contention, surfacing a different class of failures. We use the Clinical Environment Simulator (CES), in which an agent manages an entire emergency-department shift under continuous time and resource pressure, as a testbed: long-horizon execution failures manifest measurably in a single rollout under structured, multi-dimensional grading. On CES, current agents reach the correct diagnosis in most cases yet fail to deliver complete and timely critical actions, revealing an execution gap. We attribute this gap to three long-horizon failure modes, each operationalized as a per-trace counter: instruction-adherence drift, treatment incompleteness, and a severity-equity gap in timeliness. We then introduce Asclepius, an adaptive agent scaffolding with a self-evolving harness that rewrites the operating manual between shifts from trace-level feedback, an externalized clinical skills library for high-stakes regimen knowledge, and three isolated subagents that partition per-turn decisions across the patient queue. On held-out batches never observed during harness evolution, Asclepius improves critical-action correctness by 22% (p = 0.024) over a strong baseline agent framework while preserving diagnostic accuracy, with consistent gains across five LLM judges from three model families; on the full ten-batch set, improvements reach 25% on critical actions and 13% on timeliness. The three failure modes form a coupled bottleneck: decisive reductions appear only when all three components act together.

Figures & tables

Figure 1: Overview of Asclepius. Two coupled loops wrap a baseline agent. Within a shift (inner loop), the Main Agent uses an MCP tool interface, a clinical skills library, and three isolated subagents. Across shifts (outer loop), a proposer agent reads the shift trace and rewrites the operating manual for the next shift.

Configuration	Overall	Diagnosis	Critical Actions	Timeliness	Disposition
Codex (raw)	2.74	3.74	1.59	1.58	4.03
Gemini (raw)	2.06	2.47	1.43	1.42	2.93
Claude Code (raw)	3.52	4.22	2.69	2.65	4.53
Baseline framework	3.80	4.39	2.94	3.34	4.52
Asclepius	4.10	4.38	3.67	3.79	4.54
$\Delta$ vs. baseline	$+0.30$ **	$-0.02$ ns	$+0.73$ ***	$+0.45$ **	$+0.03$ ns

Table 1: Main results across all ten patient batches. Mean scores (out of 5) for the three raw single-agent variants, the baseline agent framework, and Asclepius (harness + skills + acting subagents). Scores pool the six search batches, over which the operating manual was evolved, with the four held-out batches. Significance for the

\Delta

row is from a patient-level mixed-effects model, BH-FDR corrected across all 20 system-by-dimension tests (*:

p<0.05

, **:

p<0.01

, ***:

p<0.001

, ns: not significant). Bold marks the best score in each column.

Dimension	$\Delta$	$p$
Critical Actions	$+0.625$	$0.024$ *
Timeliness	$+0.312$	ns
Overall	$+0.193$	ns

Table 2: Held-out significance (batches 7–10,

n=48

). Mixed-effects mean differences of Asclepius vs. the baseline framework, fit on the held-out batches alone. Diagnosis and disposition are unchanged (both ns).

Judge	$\Delta$ Overall	$\Delta$ Critical Actions	$\Delta$ Timeliness	$\Delta$ Diagnosis	$\Delta$ Disposition
GPT-4.1 (primary)	$+0.30$ **	$+0.73$ ***	$+0.45$ **	$-0.02$ ns	$+0.03$ ns
Claude Sonnet 4.5	$+0.30$ *	$+0.65$ ***	$+0.52$ **	$-0.04$ ns	$+0.09$ ns
Claude Sonnet 5	$+0.36$ **	$+0.76$ ***	$+0.72$ ***	$-0.02$ ns	$-0.03$ ns
Gemini 3.1 Pro	$+0.30$ *	$+0.67$ ***	$+0.67$ ***	$-0.02$ ns	$-0.12$ ns
GPT-5	$+0.31$ **	$+0.65$ ***	$+0.71$ ***	$-0.05$ ns	$-0.09$ ns

Table 3: Multi-judge validation.

\Delta

= Asclepius

-

Baseline under each of the five judges. (*:

p<0.05

, **:

p<0.01

, ***:

p<0.001

, ns: not significant).

Figure 2: Component ablation across all ten patient batches. Per-dimension scores for the baseline framework, baseline + skills, baseline + subagents, baseline + harness, and Asclepius (Full). The baseline, $+S$ , and $+SA$ use the hand-crafted manual $M_{0}$ ; $+H$ and full Asclepius use the evolved manual $M^{*}$ .

Figure 3: Per-component reduction of targeted failure modes. Three panels: (left) naked-disposition rate early (hours 0–2) versus late (hours 4–6); (middle) zero-treatment rate among correctly-diagnosed patients; (right) timeliness gap between non-severe and severe patients. Each compares the baseline, single-component variants ( $+S$ , $+SA$ , $+H$ ), and full Asclepius. Lower is better.

Figure 4: Self-evolving harness search trajectory. Best-so-far mean scores on the six search batches across 10 proposer iterations, one line per CES dimension. v7 is selected on overall search-batch score and used in Asclepius; v6–v8 form a high-performing plateau.

Variant	Overall	Dx	CA	Time	Disp
Advisory	4.00	4.39	3.41	3.69	4.49
Acting (default)	4.10	4.38	3.67	3.79	4.54

Table 4: Subagent advisory vs. acting variants. Mean scores under the full Asclepius configuration. The acting variant gives each subagent direct MCP access within its scope; the advisory variant returns structured recommendations parsed by the Main Agent. We use the acting variant as the default.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Overall		Diagnosis		Critical Actions		Timeliness		Disposition
Configuration	1–6	7–10	1–6	7–10	1–6	7–10	1–6	7–10	1–6	7–10
Baseline	3.81	3.78	4.36	4.44	2.97	2.90	3.32	3.38	4.58	4.42
+ $S$	3.93	3.91	4.50	4.42	3.18	3.13	3.54	3.58	4.51	4.50
+ $SA$	3.76	3.97	4.39	4.50	2.94	3.35	3.28	3.63	4.42	4.40
+ $H$	4.08	3.82	4.51	4.38	3.50	3.04	3.86	3.42	4.43	4.46
Full Asclepius	4.17	3.97	4.39	4.33	3.76	3.54	3.86	3.69	4.68	4.33

Appendix

Table 5: Search-batch and held-out means. Mean overall and per-dimension scores on batches 1–6 (search) and 7–10 (held-out) for the five ablation configurations. Overall is the unweighted mean of the four sub-dimension scores. Bold marks the best score in each column.

Configuration	B1	B2	B3	B4	B5	B6	B7	B8	B9	B10
Baseline	3.54	3.85	4.06	3.58	3.73	4.08	3.56	3.79	3.94	3.83
Full Asclepius	4.25	4.35	3.94	4.23	4.25	4.02	3.88	3.92	4.21	3.90
$\Delta$	$+0.71$	$+0.50$	$-0.12$	$+0.65$	$+0.52$	$-0.06$	$+0.31$	$+0.12$	$+0.27$	$+0.06$

Appendix

Table 6: Per-batch overall scores. Overall (mean across the four dimensions) for the baseline framework and full Asclepius on each of the ten batches.

Configuration	Overall	Dx	CA	Time	Disp
Baseline	3.80	4.39	2.94	3.34	4.52
+ $S$	3.92 ns	4.47	3.16	3.56	4.51
+ $SA$	3.84 ns	4.43	3.11	3.42	4.41
+ $H$	3.98 ns	4.46	3.32 *	3.68 *	4.44
Full Asclepius	4.10 **	4.38	3.67 ***	3.79 **	4.54

Appendix

Table 7: Component ablation across all ten patient batches. Per-dimension mean scores when adding the harness (

H

), skills (

S

), and acting subagents (

SA

) on top of the baseline framework. Significance markers (vs. baseline, BH-FDR corrected) follow the overall and execution columns. Bold marks the best score in each column.

	+ Skills		+ Subagents		+ Harness		Full
Outcome	$\Delta$	sig.	$\Delta$	sig.	$\Delta$	sig.	$\Delta$	sig.
Overall	$+0.12$	ns	$+0.04$	ns	$+0.18$	ns	$+0.30$	**
Diagnosis	$+0.08$	ns	$+0.04$	ns	$+0.07$	ns	$-0.02$	ns
Critical Actions	$+0.22$	ns	$+0.17$	ns	$+0.38$		$+0.73$
Timeliness	$+0.22$	ns	$+0.08$	ns	$+0.34$		$+0.45$	**
Disposition	$-0.01$	ns	$-0.11$	ns	$-0.08$	ns	$+0.03$	ns

Appendix

Table 8: Significance versus baseline (mixed-effects model, BH-FDR corrected). For each outcome and configuration, we report the estimated mean difference vs. the baseline framework on the 1–5 scale and the BH-FDR-corrected significance (*:

p<0.05

, **:

p<0.01

, ***:

p<0.001

, ns: not significant). Overall is the unweighted mean of the four sub-dimension scores.

Split	$\Delta$ Overall	$p$
All batches	$+0.12$	0.364
Held-out (7–10)	$+0.02$	0.883

Appendix

Table 9: Marginal contribution of the evolved manual. Mixed-effects mean differences of full Asclepius (

M^{*}

) versus S+SA (

M_{0}

), holding skills and acting subagents fixed.

Config.	Drift (early $\to$ late)	Incomplete	Neglect
Baseline	16 $\to$ 57%	19%	1.13
+ $S$	10 $\to$ 28%	14%	1.10
+ $SA$	13 $\to$ 29%	20%	0.98
+ $H$	0 7 $\to$ 39%	7%	0.94
Full	10 $\to$ 12%	0 9%	0.23

Appendix

Table 10: Per-trace failure-mode counters across all ten patient batches. Lower is better for all three counters. Drift : decay in the agent’s adherence to the operating manual over the shift, measured as the early (hours 0–2)

\to

late (hours 4–6) rise in naked dispositions (dispositions placed with

\leq 1

prior treatment order, violating the manual’s treatment-first rule). Incomplete : fraction of correctly-diagnosed patients given no disease-specific treatment at all. Neglect : timeliness equity gap, the mean-timeliness difference between non-severe and severe patients (severe

=

baseline holistic encounter rating

\leq 2

n{=}20

Iteration	Overall	Diagnosis	Critical Actions	Timeliness	Disposition
v0 (baseline)	3.81	4.36	2.97	3.32	4.58
v1	3.95	4.51	3.20	3.50	4.60
v2	3.97	4.40	3.36	3.58	4.52
v3	3.98	4.45	3.29	3.64	4.53
v4	3.89	4.49	3.18	3.46	4.43
v5	3.91	4.43	3.10	3.57	4.53

Appendix

Table 11: Harness search trajectory on search batches (1–6). Mean scores across the six search batches (out of 5) for each harness iteration. Overall is the unweighted mean of the four sub-dimension scores. v7 is selected as the final manual based on overall search-batch performance. Bold marks the best score in each column.

Long-Horizon Agent Tasks

Long-Horizon Agent Tasks study how AI agents plan, remember, and act reliably across extended, multi-step interactions.

See all 195 papers

The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
Apr 13, 2026Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun +7Long-Horizon Agent Tasks Agent Failure Analysis
University of Wisconsin–Madison · University of California, Berkeley · Georgia Institute of Technology
Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents
Sep 10, 2026Ruiqing Yue, Yu Cui, Zhuoyu Sun +11Agent Harness Optimization LLM Agent Harnesses
Chengdu Institute of Computer Applications, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Beijing Institute of Technology +3
Real-Time Detection and Repair of LLM Agent Failures
Aug 3, 2026Sunny DubeyAI Agent Reliability AI Agent Monitoring

Apr 13, 2026J/K move · Enter open · S save

The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break

Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun, Haorui Wang+6

University of Wisconsin–Madison · University of California, Berkeley · Georgia Institute of Technology

Long-Horizon Agent Tasks Agent Failure Analysis Long-Horizon Agent Evaluation LLM Agent Reliability AI Agent Benchmarks

Open paper AskPDF arXiv

Large language model (LLM) agents perform strongly on short- and mid-horizon tasks, but often break down on long-horizon tasks that require extended, interdependent action sequences. Despite rapid progress in agentic systems, these long-horizon failures remain poorly characterized, hindering principled diagnosis and comparison across domains. To address this gap, we introduce HORIZON, an initial cross-domain diagnostic benchmark for systematically constructing tasks and analyzing long-horizon failure behaviors in LLM-based agents. Using HORIZON, we evaluate state-of-the-art (SOTA) agents from multiple model families (GPT-5 variants and Claude models), collecting 3100+ trajectories across four representative agentic domains to study horizon-dependent degradation patterns. We further propose a trajectory-grounded LLM-as-a-Judge pipeline for scalable and reproducible failure attribution, and validate it with human annotation on trajectories, achieving strong agreement (inter-annotator κ=0.61; human-judge κ=0.84). Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures and offer practical guidance for building more reliable long-horizon agents. We release our project website at HORIZON Leaderboard and welcome contributions from the community.

Abstract

Long-Horizon Agent Tasks

Explore similar work

The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break

Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents

Real-Time Detection and Repair of LLM Agent Failures