MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation

Rui Tang; Hongqiu Wang

cs.AIOct 11, 2026

MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation

Authors: Rui Tang, Hongqiu Wang

Organizations: The University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Healthcare agents must transform longitudinal clinical evidence into reliable actions through heterogeneous tools and stateful workflows, conditions under which model-centric systems may lose context, misuse tools, fail to verify consequential transitions, or cross authorization boundaries. Existing medical-agent benchmarks primarily emphasize terminal task outcomes, making it difficult to determine whether execution failures arise from the agent model, the runtime harness, or their interaction. We present the Medical Harness Protocol, an outcome-preserving framework that independently evaluates seven healthcare-specific process dimensions: Care Action Fulfillment, Clinical Resource Competence, Patient-Evidence Alignment, Clinical Workflow Continuity, Clinical Auditability, Closed-Loop Verification, and Clinical Conduct Boundaries. A normalized intervention trace further tracks whether runtime support is available, activated, delivered, adopted by the agent, and propagated into subsequent behavior. We instantiate the protocol as MedHarnessBench, comprising 342 executable tasks across FHIR/EHR operations, healthcare-administration interfaces, and multimodal clinical tools. Across more than 16,000 controlled task executions, MedHarness yields broad improvements in terminal task success, intermediate subtask completion, and healthcare-specific process quality. We further evaluate MedHarness against multiple general-purpose harnesses and use controlled single-fault perturbations to verify the sensitivity and discrimination of the seven-dimensional evaluation. These results demonstrate that healthcare-specific runtime support can substantially improve both task completion and execution quality, while revealing failures that terminal outcomes alone cannot capture. Code and resources are available at https://github.com/Tomry1114/Awesome-Agent-Harness-for-Healthcare.

Figures & tables

Figure 1: Overview of MedHarnessBench. Healthcare agents operate over heterogeneous evidence and interfaces, while MedHarness mediates their interactions with clinical APIs, general tools, databases, and medical knowledge through monitoring, verification, recovery, routing, and safety control. The benchmark evaluates harness-mediated execution through seven healthcare-specific process dimensions: Care Action Fulfillment, Clinical Resource Competence, Patient–Evidence Alignment, Clinical Workflow Continuity, Clinical Auditability, Closed-Loop Verification, and Clinical Conduct Boundaries. It integrates 342 executable tasks from PhysicianBench, HealthAdminBench, and MedCTA, spanning FHIR/EHR operations, clinical GUI workflows, and multimodal medical tool use.

Benchmark	#Tasks	Execution setting	Harness demands
PhysicianBench	100	FHIR API; clinical retrieval, ordering, and documentation	Evidence grounding; structured writes; authorization; write verification
HealthAdminBench	135	Clinical GUI; prior authorization, appeals, and submission	State tracking; field completion; recovery; submission verification
MedCTA	107	Multimodal tools; image and report analysis, retrieval, and calculation	Tool selection; argument validity; evidence integration; result use
Total	342	Three executable healthcare settings	Process control across API, GUI, and multimodal workflows

Table 1: Executable healthcare environments in MedHarnessBench.

Category	Dimension	Operational focus
Efficiency	Care Action Fulfillment	Completion of required care actions to their intended state.
	Clinical Resource Competence	Authorized, semantically valid, and productive use of clinical resources.
	Patient–Evidence Alignment	Acquisition of required evidence and binding to the intended clinical subject.
	Clinical Workflow Continuity	Ordered workflow progression, recovery from failed obligations, and appropriate termination.
Safety	Clinical Auditability	Reconstructability of execution state, failures, recovery, and reliability.
	Closed-Loop Verification	Confirmation of decision-critical preconditions and post-action state changes.

Table 2: Healthcare-specific process dimensions in MedHarnessBench.

Native

Medical Harness

Model

Pass@3

Subtask

Pass@3

Subtask

HealthAdminBench (

n=135

)

Claude Sonnet 5

0.341

0.794

0.904

0.944

0.602

0.941

0.910

0.351

0.568

0.363

\mathbf{(+0.022)}

0.805

\mathbf{(+0.011)}

0.917

\mathbf{(+0.013)}

0.946

\mathbf{(+0.002)}

0.622

\mathbf{(+0.020)}

0.942

\mathbf{(+0.001)}

0.912

\mathbf{(+0.002)}

0.649

\mathbf{(+0.298)}

0.615

\mathbf{(+0.047)}

GPT-5.4

0.267

0.543

0.648

0.696

0.522

0.824

0.916

0.283

0.774

0.105

\mathbf{(-0.162)}

0.633

\mathbf{(+0.090)}

0.868

\mathbf{(+0.220)}

0.913

\mathbf{(+0.217)}

0.604

\mathbf{(+0.082)}

0.902

\mathbf{(+0.078)}

0.906

\mathbf{(-0.010)}

0.589

\mathbf{(+0.306)}

0.670

\mathbf{(-0.104)}

Qwen3.7-Max

0.252

0.762

0.907

0.933

0.632

0.898

0.920

0.375

0.709

0.481

\mathbf{(+0.229)}

0.786

\mathbf{(+0.024)}

0.944

\mathbf{(+0.037)}

0.934

\mathbf{(+0.001)}

0.637

\mathbf{(+0.005)}

0.896

\mathbf{(-0.002)}

0.906

\mathbf{(-0.014)}

0.677

\mathbf{(+0.302)}

0.739

\mathbf{(+0.030)}

GPT-5.5

0.244

0.760

0.922

0.949

0.727

0.962

0.909

0.389

0.741

0.267

\mathbf{(+0.023)}

0.792

\mathbf{(+0.032)}

0.937

\mathbf{(+0.015)}

0.963

\mathbf{(+0.014)}

0.821

\mathbf{(+0.094)}

0.964

\mathbf{(+0.002)}

0.912

\mathbf{(+0.003)}

0.701

\mathbf{(+0.312)}

0.756

\mathbf{(+0.015)}

Table 3: Native and MedHarness performance across eight agents and three healthcare environments. Pass@3 measures terminal task success, Subtask measures intermediate task completion, and F/R/E/W/A/V/B denote the seven healthcare-specific process dimensions. Values in parentheses indicate changes relative to Native. Bold denotes the best score within each environment and runtime configuration.

Method	Pass@3	Subtask	F	R	E	W	A	V	B	Avg.
PhysicianBench ( $n=20$ )
Native	$0.310$	$0.677$	$0.833$	$0.934$	$0.740$	$0.884$	$0.966$	$0.337$	$0.529$	$0.746$
MedHarness	0.720	0.963	0.908	0.946	0.800	0.921	0.968	0.390	$0.638$	0.796
Claude Code	$0.280$	$0.322$	$0.558$	$0.769$	$0.650$	$0.804$	$0.961$	$0.317$	0.748	$0.687$
NeMo Guardrails	$0.430$	$0.185$	$0.442$	$0.744$	$0.630$	$0.709$	$0.931$	$0.300$	$0.435$	$0.599$
OpenAI Agents SDK	$0.250$	$0.192$	$0.392$	$0.731$	$0.550$	$0.670$	$0.938$	$0.296$	$0.513$	$0.584$

Table 4: Comparison of MedHarness with general-purpose harnesses using GPT-5.5 as the target agent. For each healthcare environment, 20 tasks are selected by stratified sampling over native task categories. Pass@3 measures terminal task success, Subtask measures intermediate task completion, and F/R/E/W/A/V/B denote the seven healthcare-specific process dimensions. Avg. is the mean of the seven process scores. Bold denotes the best score within each environment.

Dim.	Injected failure	Example transformation
F	Incomplete care action	Remove evidence that an applicable care action or required artifact was completed while preserving the surrounding execution trace.
R	Resource-use failure	Corrupt the selected clinical resource, authorization state, parameter semantics, or use of the returned result while leaving the intended task unchanged.
E	Evidence misalignment	Remove required evidence or alter its subject binding such that the evidence no longer supports the current patient, state, or decision.
W	Workflow discontinuity	Remove evidence required to maintain valid progression, recovery, readiness, or termination across successive workflow states.
A	Missing audit evidence	Remove state, failure, recovery, provenance, or reliability evidence required to reconstruct the execution and its consequential decisions.
V	Missing closed-loop verification	Remove an applicable precondition confirmation or post-action verification while preserving the corresponding action itself.

Table 5: Single-fault interventions used to evaluate the seven process dimensions. F/R/E/W denote the four Efficiency dimensions, while A/V/B denote the three Safety dimensions. Each intervention is adapted to the benchmark environment while preserving the same failure semantics.

Panel of Figure 2 — Figure 2: Process-failure sensitivity and discrimination under controlled single-fault perturbations. (a) Benchmark-balanced mean score decrease for each injected failure and responding dimension, computed over the applicable support of the responding dimension. The targeted dimension is the largest response in all seven rows. (b) Benchmark-balanced fault-localization confusion matrix obtained by predicting the injected failure from the dimension with the largest observed score decrease. Rows are normalized by injected failure class.

Dimension	Target drop (95% CI)	Selectivity margin (95% CI)
Care Action Fulfillment (F)	0.373 [0.302, 0.440]	0.326 [0.258, 0.387]
Clinical Resource Competence (R)	0.087 [0.071, 0.104]	0.086 [0.066, 0.105]
Patient–Evidence Alignment (E)	0.109 [0.067, 0.155]	0.087 [0.042, 0.143]
Clinical Workflow Continuity (W)	0.160 [0.132, 0.186]	0.168 [0.134, 0.201]
Clinical Auditability (A)	0.302 [0.262, 0.340]	0.285 [0.243, 0.324]
Closed-Loop Verification (V)	0.095 [0.090, 0.101]	0.092 [0.084, 0.100]

Table 6: Sensitivity and selectivity under controlled single-fault perturbations. Target drop denotes the benchmark-balanced decrease in the targeted process dimension. Selectivity margin is computed for each clean–perturbed pair as the targeted-dimension drop minus the largest drop among jointly comparable non-target dimensions, and is then aggregated across benchmarks. Brackets report 95% bootstrap confidence intervals.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 525 papers

Oct 5, 2026

MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks

LLM agents are increasingly built for medical work and scored on clinical benchmarks. Each such score, however, comes from a model running inside an agent harness, the system that controls the loop between the model and its environment. An agent's score is therefore a property of a model--harness pair. For medical agents, how much outcomes change with the harness has rarely been measured. Measuring this change, and explaining it, raises two challenges. First, a harness comparison must change nothing but the harness and be repeated across models and kinds of task. Second, comparing whole harnesses leaves their mechanisms bundled together, so it cannot show when an individual mechanism helps. To address these challenges, we present MedicalHarness, a controlled study of models and agent harnesses on medical tasks. We first build MedicalHarnessBench to evaluate agents on

107

tasks across four domains that each test a different harness capability. Using this benchmark, we run five open-weight models under five agent harnesses, changing only the harness within a comparison, and analyze both outcomes and execution traces. To study individual mechanisms, we build MH-Lab, a controlled harness that switches off context management, planning or tool exposure one at a time within a shared execution loop. We find that the harness and its interaction with the model account for about a quarter of the outcome variance, and that no single harness is best across models and tasks. Code and data are available at https://github.com/REAL-Lab-NU/MedicalHarness.

Ziqing Wang, Lili Zhao, Kaize Ding

Northwestern University

LLM Agent Evaluation AI Agent Benchmarks Clinical Language Model Evaluation LLM Agent Harnesses Agent Benchmarks

Ask PDF

Jun 30, 2026

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous evaluation is essential for measuring progress toward real-world healthcare applications. However, existing healthcare benchmarks are often static or narrow in clinical scope, motivating broader evaluation of agents acting on raw patient data. We introduce HealthAgentBench, a suite of 75 agentic healthcare tasks across 9 categories each with its unique environment. The benchmark suite spans diagnosis, specimen assessment, research, and data management workflows and a broad range of modalities. Each task is designed to replicate an end-to-end clinical workflow: given task instructions and supporting resources, an agent must explore raw healthcare data, operate within a complex environment, and execute multi-step solutions that go beyond naive prompting, such as querying large clinical databases or interpreting gigapixel pathology images. All tasks are scored using binary success criteria derived from reference labels, executable rules, or computational baselines. We evaluate six agents (three GPT models through Codex and three Claude models through Claude Code) with one scored attempt per task (450 evaluations). Overall success ranges from 32.0% to 57.3%; Claude Code Opus-5.5 leads with a 57.3% success rate, followed by GPT-6-astra and Sonnet-5.5 at 50.7% each. Performance varies substantially by category with specialized imaging modalities such as CT and MRI being especially difficult. HealthAgentBench provides a challenging and realistic benchmark with substantial room for future progress. We release our benchmark athttps://github.com/microsoft/HealthAgentBench.

Qianchu Liu, Sheng Zhang, Guanghui Qin +18

Microsoft Research

Long-Horizon Agent Evaluation AI Agent Benchmarks Medical AI Agents

Ask PDF

May 27, 2026

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +10

Peking University · Qiyuan Tech

Computer-Use Agent Benchmarks AI Agent Reliability LLM Agent Evaluation AI Agent Benchmarks LLM Agent Harnesses LLM Agent Reliability

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows