MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles

Ziyang Long; Xinqi Li; Lujing Xing; Hsin-Jung Yang

cs.AIOct 3, 2026

MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles

Authors: Ziyang Long, Xinqi Li, Lujing Xing, Hsin-Jung Yang

Organizations: Biomedical Imaging Research Institute, Cedars-Sinai Medical Center, Los Angeles, CA, USA · University of California, Los Angeles, Los Angeles, CA, USA · Technical University of Munich · Berlin Ultrahigh Field Facility, Max Delbrück Center for Molecular Medicine in the Helmholtz Association, Berlin, Germany

Abstract

Graphical consoles offer a practical interface for medical acquisition assistance, allowing agents to work through the controls and visual feedback used by human operators. Reliable assistance requires linking on-screen anatomy to acquisition decisions that determine what image evidence becomes available next. We introduce MedImageOSWorld, a benchmark for evaluating this capability in simulated CT, MR, and ultrasound consoles. Using screenshots and mouse-and-keyboard actions, agents configure protocols, plan acquisitions, inspect the resulting images, and make corrective adjustments across seven capability levels, from console operation to feedback-driven control. Evaluation combines task-specific workflow checks with hidden anatomical ground truth to assess procedural completion and acquisition outcomes separately. A common evaluation protocol specifies episode conditions and interaction budgets, while recorded trajectories support analysis of how agents observe, act, and respond to acquisition feedback. Across eleven open-weight agents, success rates range from 3.0 to 25.0 on a 0-100 scale while workflow-progress rates reach 21.5-74.5: agents complete much of the console workflow but rarely acquire the intended anatomy. Success collapses between perception and planning, from 79-90% at the lowest three levels for the best agent to at most 12% for millimetre-level planning and 0% for closed-loop control among open-weight agents. Two proprietary agents reach success rates of about 34 and exceed the best open-weight agent mainly in acquisition quality (61 versus 42). MedImageOSWorld provides a controlled setting for studying whether general-purpose GUI agents can translate visual observations into effective medical acquisition decisions.

Figures & tables

Figure 1: MedImageOSWorld evaluation loop. (a) A GUI agent receives console screenshots and acts through mouse and keyboard inputs, without interface or simulator state. (b) Changes to CT scan range, MR plane, or ultrasound probe pose alter the image evidence acquired next. (c) A hidden oracle evaluates the final acquisition in the subject’s coordinate frame, separating workflow completion from acquisition quality. The displayed measurements illustrate the scoring interface and are not experimental results.

Work	Setting	Pixels-only obs.	Plans acq.	Action-dep. image	Subject-frame GT
OSWorld 2.0 ( Yuan et al., 2026 )	Desktop and web computer use	$\checkmark$	—	—	—
MedAgentBench ( Jiang et al., 2025 )	FHIR-based EHR tasks	—	—	—	—
RadABench ( Zheng et al., 2024 )	Prompt-driven radiology tools	—	—	—	—
ABRA ( Maksudov et al., 2026 )	DICOM viewer and reporting tools	$\circ$	—	—	—
MaRGE/MaRCoS ( Algarín et al., 2024 )	Research MRI scanner console	—	$\checkmark$	$\checkmark$	—
Autonomous MRI ( Ravi & Geethanath, 2020 )	Automated brain MRI workflow	—	$\checkmark$	$\checkmark$	—

Table 1: Comparison of environments related to GUI agents and medical image acquisition.

Figure 2: From acquisition procedures to executable benchmark tasks. (a) The shared L0–L6 capability ladder is defined first; three medical imaging experts, assisted by a frontier AI model, propose, cross-review, and finalize each task. Each task specifies an initial state, an acquisition goal, an interaction budget, and scoring conditions. (b) A CT lung-coverage task links scout-based planning to the acquired volume, with workflow completion and anatomical coverage scored separately. Numbers in (b) are illustrative.

Figure 3: Performance across tasks and capability levels. (a) Overall success rate (SR; filled blue circles) is the primary metric: each scored episode receives 1 for PASS and 0 otherwise. Workflow progress (PR; open orange circles) measures completed workflow stages on a five-point scale (0, 0.25, 0.5, 0.75, or 1), without assessing acquisition quality. Acquisition quality (AQ; green diamonds) is a continuous score from 0 to 1 for the 28 eligible geometry and parameter tasks. All three measures are shown as percentages; PR and AQ are auxiliary measures and are not combined with SR. Overall SR uses capability-level weights $w_{i}=i+1$ across 49 tasks (weighted denominator 181). (b) Mean task-level SR within each capability level, pooled across CT, MR, and ultrasound; column headings show task counts. The divider separates locally served open-weight models from API-served proprietary models. API results use the first five assets per task.

Explore similar work

Jun 2, 2026

MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents

Computer-use agents could automate repetitive screen-based clinical work, but their reliability in medical graphical user interfaces remains largely unvalidated. Existing benchmarks focus on general web or desktop tasks and underrepresent medical software, which requires domain knowledge, exhibits markedly different UI design from mainstream applications, lacks public testing environments, and demands safety validation beyond task completion. We introduce MedCUA-Bench, an interactive benchmark for clinical computer-use agents. It covers 18 clinical scenarios across 10 medical domains, reconstructed from real product manuals and open-source medical systems to capture authentic clinical interfaces while avoiding licensing and privacy constraints. Each task ships with paired intent- and step-level goals to disentangle clinical reasoning from UI execution, and is evaluated by a deterministic checker over task completion and five clinical safety dimensions. Across 23 agents, the best closed-source model reaches 54.2% strict success, while all models remain below 9% on the real OpenEMR. Open-source agents average only 2.5%, with the best reaching 16.2%. MedCUA-Bench exposes the gap between current agents and reliable clinical software use, providing a reproducible testbed for future research.

Jia Yu, Zilong Wang, Xinyang Jiang +2

Microsoft Research Asia, Shanghai, China · Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China · Shanghai Key Laboratory of MICCAI, Shanghai, China

Computer-Use Agent Benchmarks Computer-Use Agents LLM Agent Evaluation AI Agent Benchmarks Workflow Automation

Ask PDF

Mar 25, 2026

MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows

Medical imaging benchmarks often evaluate VLMs on pre-selected 2D images, slices, crops, or patches, making evaluation closer to visual recognition. Real clinical workflows impose a different burden: readers must search through complete studies, operate imaging software, navigate across slices and magnifications, and document visual evidence that can be audited. We argue that this evidence-producing workflow is a critical missing evaluation axis for medical imaging agents. To study it, we introduce MedFlowBench, a full-study benchmark for VLM agents, together with MedOpenClaw, a controlled and replayable runtime in which agents operate medical imaging viewers such as 3D Slicer and QuPath. In each episode, an agent inspects a complete radiology study or whole-slide pathology image, returns a task answer, and submits structured evidence, including key slices, coordinates, regions of interest, or lesion-state fields. This evidence is automatically checked against withheld masks, annotations, and labels. Across evaluated models, final answer-only scoring gives an overly optimistic picture: when answers must also be supported by correct evidence, performance drops substantially on complex workflows. We further find that adding image-analysis tools does not by itself solve the problem. Tools help when they make a complex procedure simple and reliable, but agents still struggle when they must choose inputs, manage viewer state, and verify intermediate outputs over multiple steps. MedFlowBench exposes whether medical imaging agents can produce auditable evidence from complete studies, rather than plausible answers from selected images.

Weixiang Shen, Chengzhi Shen, Yanzhu Hu +12

Technical University of Munich (TUM) · TUM University Hospital · LMU Munich +6

Medical VLMs Medical Image Benchmarks AI Agent Auditing Medical Image Grounding

Ask PDF

Jun 10, 2026

MedCTA: A Benchmark for Clinical Tool Agents

To make clinically grounded decisions, medical AI agents are expected to go beyond simple recognition and be capable of tool retrieval, evidence acquisition, and integration. Existing benchmarks largely evaluate isolated perception or single-turn question answering, and therefore provide limited visibility into failures of planning, tool recruitment, and rollout reliability. We introduce MedCTA, a benchmark for evaluating medical tool agents on clinician-validated, step-implicit tasks grounded in realistic multimodal clinical inputs, including radiology images, pathology slides, and reports. MedCTA comprises 107 real-world clinical tasks with clinician-verified executable trajectories over 5 deployed tools, and supports process-aware evaluation of tool selection, argument validity, execution stability, trajectory fidelity, and outcome quality. We benchmark 18 open- and closed-source multimodal models and find that even frontier systems remain brittle in multi-step clinical tool use: autonomous rollouts are dominated by protocol failures, premature stopping, and incorrect tool recruitment, while gold-standard tool routing yields large but still incomplete gains. These results show that strong backbone perception does not translate into reliable agentic behavior in clinical settings. MedCTA provides a rigorous testbed for auditing, diagnosing, and advancing trustworthy medical AI agents. The dataset and evaluation suite are available at https://ivul-kaust.github.io/MedCTA/

Tajamul Ashraf, Hyewon Jeong, Fida Mohammad Thoker +1

King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Massachusetts Institute of Technology (MIT), USA

Agent Evaluation AI Agent Benchmarks Tool-Using Agents Multimodal Agents

Ask PDF