MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation

Rui Tang; Hongqiu Wang

cs.AIOct 11, 2026

MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation

Authors: Rui Tang, Hongqiu Wang

Organizations: The University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Healthcare agents must transform longitudinal clinical evidence into reliable actions through heterogeneous tools and stateful workflows, conditions under which model-centric systems may lose context, misuse tools, fail to verify consequential transitions, or cross authorization boundaries. Existing medical-agent benchmarks primarily emphasize terminal task outcomes, making it difficult to determine whether execution failures arise from the agent model, the runtime harness, or their interaction. We present the Medical Harness Protocol, an outcome-preserving framework that independently evaluates seven healthcare-specific process dimensions: Care Action Fulfillment, Clinical Resource Competence, Patient-Evidence Alignment, Clinical Workflow Continuity, Clinical Auditability, Closed-Loop Verification, and Clinical Conduct Boundaries. A normalized intervention trace further tracks whether runtime support is available, activated, delivered, adopted by the agent, and propagated into subsequent behavior. We instantiate the protocol as MedHarnessBench, comprising 342 executable tasks across FHIR/EHR operations, healthcare-administration interfaces, and multimodal clinical tools. Across more than 16,000 controlled task executions, MedHarness yields broad improvements in terminal task success, intermediate subtask completion, and healthcare-specific process quality. We further evaluate MedHarness against multiple general-purpose harnesses and use controlled single-fault perturbations to verify the sensitivity and discrimination of the seven-dimensional evaluation. These results demonstrate that healthcare-specific runtime support can substantially improve both task completion and execution quality, while revealing failures that terminal outcomes alone cannot capture. Code and resources are available at https://github.com/Tomry1114/Awesome-Agent-Harness-for-Healthcare.

Figures & tables

Figure 1: Overview of MedHarnessBench. Healthcare agents operate over heterogeneous evidence and interfaces, while MedHarness mediates their interactions with clinical APIs, general tools, databases, and medical knowledge through monitoring, verification, recovery, routing, and safety control. The benchmark evaluates harness-mediated execution through seven healthcare-specific process dimensions: Care Action Fulfillment, Clinical Resource Competence, Patient–Evidence Alignment, Clinical Workflow Continuity, Clinical Auditability, Closed-Loop Verification, and Clinical Conduct Boundaries. It integrates 342 executable tasks from PhysicianBench, HealthAdminBench, and MedCTA, spanning FHIR/EHR operations, clinical GUI workflows, and multimodal medical tool use.

Benchmark	#Tasks	Execution setting	Harness demands
PhysicianBench	100	FHIR API; clinical retrieval, ordering, and documentation	Evidence grounding; structured writes; authorization; write verification
HealthAdminBench	135	Clinical GUI; prior authorization, appeals, and submission	State tracking; field completion; recovery; submission verification
MedCTA	107	Multimodal tools; image and report analysis, retrieval, and calculation	Tool selection; argument validity; evidence integration; result use
Total	342	Three executable healthcare settings	Process control across API, GUI, and multimodal workflows

Table 1: Executable healthcare environments in MedHarnessBench.

Category	Dimension	Operational focus
Efficiency	Care Action Fulfillment	Completion of required care actions to their intended state.
	Clinical Resource Competence	Authorized, semantically valid, and productive use of clinical resources.
	Patient–Evidence Alignment	Acquisition of required evidence and binding to the intended clinical subject.
	Clinical Workflow Continuity	Ordered workflow progression, recovery from failed obligations, and appropriate termination.
Safety	Clinical Auditability	Reconstructability of execution state, failures, recovery, and reliability.
	Closed-Loop Verification	Confirmation of decision-critical preconditions and post-action state changes.

Table 2: Healthcare-specific process dimensions in MedHarnessBench.

Native

Medical Harness

Model

Pass@3

Subtask

Pass@3

Subtask

HealthAdminBench (

n=135

)

Claude Sonnet 5

0.341

0.794

0.904

0.944

0.602

0.941

0.910

0.351

0.568

0.363

\mathbf{(+0.022)}

0.805

\mathbf{(+0.011)}

0.917

\mathbf{(+0.013)}

0.946

\mathbf{(+0.002)}

0.622

\mathbf{(+0.020)}

0.942

\mathbf{(+0.001)}

0.912

\mathbf{(+0.002)}

0.649

\mathbf{(+0.298)}

0.615

\mathbf{(+0.047)}

GPT-5.4

0.267

0.543

0.648

0.696

0.522

0.824

0.916

0.283

0.774

0.105

\mathbf{(-0.162)}

0.633

\mathbf{(+0.090)}

0.868

\mathbf{(+0.220)}

0.913

\mathbf{(+0.217)}

0.604

\mathbf{(+0.082)}

0.902

\mathbf{(+0.078)}

0.906

\mathbf{(-0.010)}

0.589

\mathbf{(+0.306)}

0.670

\mathbf{(-0.104)}

Qwen3.7-Max

0.252

0.762

0.907

0.933

0.632

0.898

0.920

0.375

0.709

0.481

\mathbf{(+0.229)}

0.786

\mathbf{(+0.024)}

0.944

\mathbf{(+0.037)}

0.934

\mathbf{(+0.001)}

0.637

\mathbf{(+0.005)}

0.896

\mathbf{(-0.002)}

0.906

\mathbf{(-0.014)}

0.677

\mathbf{(+0.302)}

0.739

\mathbf{(+0.030)}

GPT-5.5

0.244

0.760

0.922

0.949

0.727

0.962

0.909

0.389

0.741

0.267

\mathbf{(+0.023)}

0.792

\mathbf{(+0.032)}

0.937

\mathbf{(+0.015)}

0.963

\mathbf{(+0.014)}

0.821

\mathbf{(+0.094)}

0.964

\mathbf{(+0.002)}

0.912

\mathbf{(+0.003)}

0.701

\mathbf{(+0.312)}

0.756

\mathbf{(+0.015)}

Table 3: Native and MedHarness performance across eight agents and three healthcare environments. Pass@3 measures terminal task success, Subtask measures intermediate task completion, and F/R/E/W/A/V/B denote the seven healthcare-specific process dimensions. Values in parentheses indicate changes relative to Native. Bold denotes the best score within each environment and runtime configuration.

Method	Pass@3	Subtask	F	R	E	W	A	V	B	Avg.
PhysicianBench ( $n=20$ )
Native	$0.310$	$0.677$	$0.833$	$0.934$	$0.740$	$0.884$	$0.966$	$0.337$	$0.529$	$0.746$
MedHarness	0.720	0.963	0.908	0.946	0.800	0.921	0.968	0.390	$0.638$	0.796
Claude Code	$0.280$	$0.322$	$0.558$	$0.769$	$0.650$	$0.804$	$0.961$	$0.317$	0.748	$0.687$
NeMo Guardrails	$0.430$	$0.185$	$0.442$	$0.744$	$0.630$	$0.709$	$0.931$	$0.300$	$0.435$	$0.599$
OpenAI Agents SDK	$0.250$	$0.192$	$0.392$	$0.731$	$0.550$	$0.670$	$0.938$	$0.296$	$0.513$	$0.584$

Table 4: Comparison of MedHarness with general-purpose harnesses using GPT-5.5 as the target agent. For each healthcare environment, 20 tasks are selected by stratified sampling over native task categories. Pass@3 measures terminal task success, Subtask measures intermediate task completion, and F/R/E/W/A/V/B denote the seven healthcare-specific process dimensions. Avg. is the mean of the seven process scores. Bold denotes the best score within each environment.

Dim.	Injected failure	Example transformation
F	Incomplete care action	Remove evidence that an applicable care action or required artifact was completed while preserving the surrounding execution trace.
R	Resource-use failure	Corrupt the selected clinical resource, authorization state, parameter semantics, or use of the returned result while leaving the intended task unchanged.
E	Evidence misalignment	Remove required evidence or alter its subject binding such that the evidence no longer supports the current patient, state, or decision.
W	Workflow discontinuity	Remove evidence required to maintain valid progression, recovery, readiness, or termination across successive workflow states.
A	Missing audit evidence	Remove state, failure, recovery, provenance, or reliability evidence required to reconstruct the execution and its consequential decisions.
V	Missing closed-loop verification	Remove an applicable precondition confirmation or post-action verification while preserving the corresponding action itself.

Table 5: Single-fault interventions used to evaluate the seven process dimensions. F/R/E/W denote the four Efficiency dimensions, while A/V/B denote the three Safety dimensions. Each intervention is adapted to the benchmark environment while preserving the same failure semantics.

Panel of Figure 2 — Figure 2: Process-failure sensitivity and discrimination under controlled single-fault perturbations. (a) Benchmark-balanced mean score decrease for each injected failure and responding dimension, computed over the applicable support of the responding dimension. The targeted dimension is the largest response in all seven rows. (b) Benchmark-balanced fault-localization confusion matrix obtained by predicting the injected failure from the dimension with the largest observed score decrease. Rows are normalized by injected failure class.

Dimension	Target drop (95% CI)	Selectivity margin (95% CI)
Care Action Fulfillment (F)	0.373 [0.302, 0.440]	0.326 [0.258, 0.387]
Clinical Resource Competence (R)	0.087 [0.071, 0.104]	0.086 [0.066, 0.105]
Patient–Evidence Alignment (E)	0.109 [0.067, 0.155]	0.087 [0.042, 0.143]
Clinical Workflow Continuity (W)	0.160 [0.132, 0.186]	0.168 [0.134, 0.201]
Clinical Auditability (A)	0.302 [0.262, 0.340]	0.285 [0.243, 0.324]
Closed-Loop Verification (V)	0.095 [0.090, 0.101]	0.092 [0.084, 0.100]

Table 6: Sensitivity and selectivity under controlled single-fault perturbations. Target drop denotes the benchmark-balanced decrease in the targeted process dimension. Selectivity margin is computed for each clean–perturbed pair as the targeted-dimension drop minus the largest drop among jointly comparable non-target dimensions, and is then aggregated across benchmarks. Brackets report 95% bootstrap confidence intervals.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 525 papers

MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks
Oct 5, 2026Ziqing Wang, Lili Zhao, Kaize DingLLM Agent Evaluation AI Agent Benchmarks
Northwestern University
HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
Jun 30, 2026Qianchu Liu, Sheng Zhang, Guanghui Qin +18Long-Horizon Agent Evaluation AI Agent Benchmarks
Microsoft Research
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +10Computer-Use Agent Benchmarks AI Agent Reliability
Peking University · Qiyuan Tech

Oct 5, 2026J/K move · Enter open · S save

MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks

Ziqing Wang, Lili Zhao, Kaize Ding

Northwestern University

LLM Agent Evaluation AI Agent Benchmarks Clinical Language Model Evaluation LLM Agent Harnesses Agent Benchmarks

Open paper AskPDF arXiv

LLM agents are increasingly built for medical work and scored on clinical benchmarks. Each such score, however, comes from a model running inside an agent harness, the system that controls the loop between the model and its environment. An agent's score is therefore a property of a model--harness pair. For medical agents, how much outcomes change with the harness has rarely been measured. Measuring this change, and explaining it, raises two challenges. First, a harness comparison must change nothing but the harness and be repeated across models and kinds of task. Second, comparing whole harnesses leaves their mechanisms bundled together, so it cannot show when an individual mechanism helps. To address these challenges, we present MedicalHarness, a controlled study of models and agent harnesses on medical tasks. We first build MedicalHarnessBench to evaluate agents on

107

tasks across four domains that each test a different harness capability. Using this benchmark, we run five open-weight models under five agent harnesses, changing only the harness within a comparison, and analyze both outcomes and execution traces. To study individual mechanisms, we build MH-Lab, a controlled harness that switches off context management, planning or tool exposure one at a time within a shared execution loop. We find that the harness and its interaction with the model account for about a quarter of the outcome variance, and that no single harness is best across models and tasks. Code and data are available at https://github.com/REAL-Lab-NU/MedicalHarness.

Abstract

AI Agent Evaluation

Explore similar work

MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows