cs.AIApr 10, 2026
SaveHealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks
Organizations: Stanford University · Kinetic Systems · Stanford Health Care
Abstract
Healthcare administration accounts for over $1 trillion in annual spending, making it a promising target for LLM-based computer-use agents (CUAs). While clinical applications of LLMs have received significant attention, no benchmark exists for evaluating CUAs on end-to-end administrative workflows. To address this gap, we introduce HealthAdminBench, a benchmark comprising four realistic GUI environments: an EHR, two payer portals, and a fax system, and 135 expert-defined tasks spanning three administrative task types: Prior Authorization, Appeals and Denials Management, and Durable Medical Equipment (DME) Order Processing. Each task is decomposed into fine-grained, verifiable subtasks, yielding 1,698 evaluation points. We evaluate seven agent configurations under multiple prompting and observation settings and find that, despite strong subtask performance, end-to-end reliability remains low: the best-performing agent (Claude Opus 4.6 CUA) achieves only 36.3 percent task success, while GPT-5.4 CUA attains the highest subtask success rate (82.8 percent). These results reveal a substantial gap between current agent capabilities and the demands of real-world administrative workflows. HealthAdminBench provides a rigorous foundation for evaluating progress toward safe and reliable automation of healthcare administrative workflows.
Figures & tables
Figure 1: HealthAdminBench evaluation loop. Each task is executed by an agent through iterative observation, action selection, and interaction with simulated environments (EHR, payer portals, and fax). Task success is determined using a combination of deterministic checks and LLM-based judges.
| Benchmark | Cross-App Workflows | Strict Forms & Commits | Objective Success | Healthcare Admin |
| WebShop (2022) | x | ✓ | ✓ | x |
| Mind2Web (2023) | x | x | x | x |
| WebArena (2023) | x | ✓ | ✓ | x |
| WorkArena (2024) | x | ✓ | ✓ | x |
| OSWorld (2024) | ✓ | ✓ | ✓ | x |
| REAL (2025) | x | ✓ | ✓ | x |
Table 1: Comparison of HealthAdminBench with related benchmarks. ✓ = present, x = absent. Columns denote: (1) workflows spanning multiple disjoint applications, (2) UI environments with rigid schema validation and terminal commit actions, (3) verifiable, programmatic success criteria, and (4) tasks grounded in healthcare administration.
Figure 2: HealthAdminBench contains four environments which mimic commonly utilized applications for administrative healthcare tasks – (a) an EHR inspired by Epic, (b) a payer portal inspired by Anthem, (c) a portal inspired by Availity, and an eFax inspired by RightFax. These environments are implemented as websites following the REAL framework Garg et al. (2025)
| Subtask | Prior Authorization | Appeals & Denials Management | DME |
| Information Retrieval | 93.3% | 100.0% | 33.3% |
| Documentation | 100.0% | 100.0% | 100.0% |
| Form Completion | 58.3% | 3.3% | 66.7% |
| Task Resolution | 98.3% | 96.7% | 100.0% |
| Document Handling | 53.3% | 23.3% | 100.0% |
| Clinical Reasoning | 60.0% | 71.7% | 6.7% |
Table 2: For each task type, the percentage of tasks that include at least one subtask from each subtask type.
Figure 3: Performance of evaluated agents on HealthAdminBench , reported as (a) task success rate and (b) subtask success rate. 95% test-set bootstrap confidence intervals are noted with error bars.
| Agent | Task Type | ||
| Prior Authorization | Appeals & Denials Management | DME | |
| Claude Opus 4.6 CUA | 38.3% | 25.0% | 73.3% |
| GPT-5.4 CUA | 26.7% | 18.3% | 60.0% |
| Kimi K2.5 | 18.3% | 13.3% | 13.3% |
| Claude Opus 4.6 | 16.7% | 16.7% | 0.0% |
| Qwen 3.5 | 16.7% | 5.0% | 33.3% |
Table 3: End-to-end task success rates, split by task type.
| Agent | Subtask | |||||
| Information Retrieval | Documentation | Form Completion | Task Resolution | Document Handling | Clinical Reasoning | |
| Claude Opus 4.6 CUA | 96.5% | 74.3% | 88.9% | 70.7% | 92.3% | 61.3% |
| GPT-5.4 CUA | 96.1% | 84.4% | 84.9% | 66.2% | 94.3% | 77.0% |
| Kimi K2.5 | 87.7% | 45.6% | 53.2% | 45.3% | 58.1% | 39.3% |
| Claude Opus 4.6 | 88.5% | 43.0% | 23.4% | 40.5% | 49.3% | 32.0% |
| Qwen 3.5 | 87.4% | 54.0% | 44.6% | 51.6% | 52.0% | 44.5% |
Table 4: Granular success rate for individual subtasks, split by subtask type. Note that one task can include many subtasks.
Figure 4: Performance of evaluated agents on HealthAdminBench across prompting and observation settings, reported as task success rate.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Context | Action | Description |
| Screenshot Only | click_coord(x, y) | Left-click at pixel coordinates (origin at top-left). |
| double_click_coord(x, y) | Double-click at pixel coordinates . | |
| triple_click_coord(x, y) | Triple-click at pixel coordinates . | |
| right_click_coord(x, y) | Right-click at pixel coordinates . | |
| move_coord(x, y) | Move the cursor to pixel coordinates . | |
| type_text(text) | Type text at the current cursor focus. |
Table 5: Complete action space for HealthAdminBench. Agents with access to the accessibility tree operate on symbolic UI elements referenced by data-testid identifiers, while screenshot-only agents interact with the environment via coordinate-based mouse and keyboard actions.
Figure 5: Violin plots indicating the distribution of steps taken by each agent for easy, medium, and hard tasks. The average number of steps for each agent is printed to the right of each violin. A dotted red line indicates the maximum steps each model was allowed to take before being terminated.
Figure 6: Violin plots indicating the cost taken by each agent for easy, medium, and hard tasks on HealthAdminBench. The average cost is printed to the right of each violin. CUA agents have not been plotted due to harness limitations.
Figure 7: Subtask success rate of evaluated agents on HealthAdminBench across prompting strategy (rows) and observation modality (columns). Error bars show 95% bootstrap confidence intervals on the test set. Native CUA agents (dark bars) are evaluated only in screenshot mode. This figure complements Figure 4 , which reports the same experiments by task success rate.
| Baseline Compare | Claude Opus 4.6 CUA | GPT-5.4 CUA | Kimi K2.5 | Claude Opus 4.6 | Qwen 3.5 | Gemini 3.1 Pro |
| GPT-5.4 CUA | 9.6% (2.2% - 17.8%) | |||||
| Kimi K2.5 | 20.7% (11.8% - 29.6%) | 11.1% (3.7% - 18.5%) | ||||
| Claude Opus 4.6 | 21.5% (13.3% - 29.6%) | 11.9% (3.0% - 20.7%) | 0.7% (-5.2% - 7.4%) | |||
| Qwen 3.5 | 23.0% (14.1% - 31.1%) | 13.3% (5.9% - 20.7%) | 2.2% (-4.4% - 8.9%) | 1.5% (-5.2% - 8.2%) | ||
| Gemini 3.1 Pro | 24.4% (16.3% - 33.3%) | 14.8% (7.4% - 23.0%) | 3.7% (-3.7% - 11.1%) | 3.0% (-4.4% - 11.1%) | 1.5% (-5.2% - 8.1%) | |
| GPT-5.4 | 30.4% (22.2% - 39.3%) | 20.7% (13.3% - 28.9%) | 9.6% (3.0% - 17.0%) | 8.9% (2.2% - 15.6%) | 7.4% (0.7% - 13.3%) | 5.9% (0.0% - 11.9%) |
Table 6: Head-to-head differences in task success rate (percentage points). Positive numbers indicate that the agent in the column is better than the agent in row. 95% test-set bootstrap confidence intervals are reported under the overall difference between agents. Bolded rows are statistically significant ( ).
| Baseline Compare | Claude Opus 4.6 CUA | GPT-5.4 CUA | Kimi K2.5 | Claude Opus 4.6 | Qwen 3.5 | Gemini 3.1 Pro |
| GPT-5.4 CUA | -4.4% (-8.8% - -0.3%) | |||||
| Kimi K2.5 | 20.9% (14.5% - 27.1%) | 25.3% (19.9% - 30.7%) | ||||
| Claude Opus 4.6 | 23.0% (15.4% - 30.5%) | 27.4% (21.0% - 33.7%) | 2.1% (-4.9% - 9.4%) | |||
| Qwen 3.5 | 16.4% (9.9% - 22.9%) | 20.8% (15.7% - 25.9%) | -4.5% (-10.8% - 1.7%) | -6.6% (-13.4% - 0.1%) | ||
| Gemini 3.1 Pro | 6.9% (2.3% - 11.4%) | 11.4% (8.2% - 14.6%) | -13.9% (-19.4% - -8.4%) | -16.0% (-23.1% - -8.9%) | -9.5% (-15.2% - -4.1%) | |
| GPT-5.4 | 32.8% (26.6% - 38.9%) | 37.2% (31.9% - 42.5%) | 11.9% (5.6% - 18.3%) | 9.8% (2.5% - 17.2%) | 16.4% (10.8% - 22.1%) | 25.9% (19.9% - 31.7%) |
Table 7: Head-to-head differences in subtask success rate (percentage points). Positive numbers indicate that the agent in the column is better than the agent in row. 95% test-set bootstrap confidence intervals are reported under the overall difference between agents. Bolded rows are statistically significant ( ).
| Model | Task Success Rate | Subtask Success Rate | Fine Tuned? |
| Claude Opus 4.6 | 25.7% | 75.0% | - |
| Qwen-3.5 | 17.1% | 65.4% | - |
| Qwen-3.5-Kinetic-SFT | 40.0% | 83.3% | ✓ |
Table 8: Results of domain-specific finetuning on 35 held-out examples using accessibility tree observations and task description + portal guidance prompting.
| Baseline Compare | Qwen -3.5-Kinetic-SFT | Claude Opus 4.6 | Qwen-3.5 |
| Claude Opus 4.6 | 14.3% (-2.9% - 31.4%) | ||
| Qwen-3.5 | 22.9% (8.6% - 37.1%) | 8.6% (-2.9% - 20.0%) |
Table 9: Head-to-head differences in task success rate for the 35-task finetuning comparison (percentage points).
| Baseline Compare | Qwen -3.5-Kinetic-SFT | Claude Opus 4.6 | Qwen-3.5 |
| Claude Opus 4.6 | 8.4% (1.2% - 15.4%) | ||
| Qwen-3.5 | 17.9% (7.2% - 29.8%) | 9.5% (-1.2% - 20.1%) |
Table 10: Head-to-head differences in subtask success rate for the 35-task finetuning comparison (percentage points).
More in this topic
Computer-Use Agent Benchmarks
Computer-use agent benchmarks test whether AI systems can operate graphical interfaces, terminals, and applications reliably, safely, and efficiently.
See all 120 papers