MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI
Authors: Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq, Tausifa Jan Saleem, Tajamul Ashraf
Organizations: Gaash Research Lab, National Institute of Technology Srinagar, India · e& Group, UAE · Mohammad Bin Zayed University of Artificial Intelligence (MBZUAI), UAE · King Abdullah University of Science and Technology (KAUST), Saudi Arabia
Despite the rapid progress of Multimodal Large Language Models (MLLMs), their ability to perform reliable visual grounding in high-stakes clinical software environments remains underexplored. Existing GUI benchmarks largely focus on isolated, single-step grounding queries, overlooking the sequential, workflow-driven reasoning required in real-world medical interfaces, where tasks evolve across independent steps and dynamic interface states. We introduce MedSPOT, a workflow-aware sequential grounding benchmark for clinical GUI environments. Unlike prior benchmarks that treat grounding as a standalone prediction task, MedSPOT models procedural interaction as a sequence of structured spatial decisions. The benchmark comprises 216 task-driven videos with 597 annotated keyframes, in which each task consists of 2 to 3 interdependent grounding steps within realistic medical workflows. This design captures interface hierarchies, contextual dependencies, and fine-grained spatial precision under evolving conditions. To evaluate procedural robustness, we propose a strict sequential evaluation protocol that terminates task assessment upon the first incorrect grounding prediction, explicitly measuring error propagation in multi-step workflows. We further introduce a comprehensive failure taxonomy, including edge bias, small-target errors, no prediction, near miss, far miss, and toolbar confusion, to enable systematic diagnosis of model behavior in clinical GUI settings. By shifting evaluation from isolated grounding to workflow-aware sequential reasoning, MedSPOT establishes a realistic and safety-critical benchmark for assessing multimodal models in medical software environments. Code and data are available at: https://github.com/Tajamul21/MedSPOT.
Figures & tables
Benchmark
Size
Multi-Step
Step Eval.
Sequential
Failure
Medical
Tasks
Protocol
Taxonomy
Software
ScreenSpot [ 7 ]
1,272
×
×
×
×
×
ScreenSpot-Pro [ 20 ]
1,581
×
✓
×
×
×
OSWorld-G [ 48 ]
564
✓
×
×
×
×
OmniACT [ 18 ]
9,802
✓
✓
×
×
×
AssistGUI [ 11 ]
100
✓
✓
×
×
×
Table 1: Comparison of MedSPOT against existing GUI grounding benchmarks. Green cells indicate the presence of a capability, red cells indicate absence. MedSPOT is the only benchmark that jointly evaluates workflow-aware sequential grounding, early-termination protocol, structured failure taxonomy, and medical software environments.
Figure 1: Comparison of interface complexity. (a) Medical GUI environments are densely structured, hierarchically organized, and semantically rich with domain-specific terminology. (b) General GUI environments typically involve simpler layouts and short-horizon navigation tasks.
Figure 2: Overview of the MedSPOT dataset construction and sequential evaluation pipeline.
Figure 3: Overview of the MedSPOT annotation pipeline. GUI interaction videos are recorded from real clinical software, segmented into causally consistent decision frames, and annotated with step-level instructions, semantic targets, bounding boxes, and action types to construct sequential grounding tasks.
Figure 4: MedSPOT dataset statistics illustrating overall scale, task category distribution, and per-software coverage across ten clinical platforms.
Model Name
Params
SHR(%) ↑
S1A(%) ↑
WPS ↑
TCA(%) ↑
Closed Source Models
GPT 4o mini [ 28 ]
-
4.91
5.14
0.051
0.0
GPT 5 [ 35 ]
-
16.4
16.5
0.185
2.8
Open Source Models
Llama 3.2 Vision-Instruct [ 13 ]
11B
0.0
0.0
0.0
0.0
Qwen2-VL-Instruct [ 41 ]
7B
1.83
1.87
0.02
0.0
Table 2: Comparison of state-of-the-art MLLMs on MedSPOT-Bench under the strict sequential evaluation protocol. We report Step Hit Rate ( SHR ), Step-1 Accuracy ( S1A ), Weighted Prefix Score ( WPS ), and Task Completion Accuracy ( TCA ), where higher values indicate stronger spatial precision and sustained workflow consistency. Best results are highlighted in bold green and second-best results are underlined in blue.
Figure 5: Comparison of representative models across Step-1 Accuracy ( S1A ), Step Hit Rate ( SHR ), and Task Completion Accuracy ( TCA ). The consistent gap between S1A and TCA highlights compounding sequential errors under strict workflow constraints.
Model Name
Params
Edge Bias
Far Miss
Toolbar Confusion
Near Miss
No Prediction
Small Target
Open Source Models
Llama 3.2 Vision-Instruct [ 13 ]
11B
49
7
4
0
134
19
Qwen2-VL-Instruct [ 41 ]
7B
129
26
28
0
0
27
DeepSeek-VL2 [ 47 ]
16B
30
152
0
2
0
30
Gemma 3 [ 37 ]
27B
34
44
110
0
0
25
Mistral-3-Instruct [ 21 ]
8B
109
64
13
2
0
26
Table 3: Failure taxonomy distribution on MedSPOT-Bench. Open-source models are shaded in blue and closed-source models in red. Worst failure counts are highlighted in bold red , while lowest values are underlined blue .
Figure 6: Per-software performance analysis. (a) Average TCA across models, ranking clinical applications by difficulty. (b) Per-model TCA breakdown across ten medical software platforms, highlighting interface-dependent performance variability.
Figure 7: Ablation studies on evaluation design and spatial stratification. (a) Sequential evaluation significantly lowers task-level scores due to error compounding. (b) Small UI elements consistently exhibit lower accuracy across all threshold configurations.
Figure 8: Qualitative comparison on the task "Delete the selected study from the browser menu" (DICOMscope). GUI-Actor completes all steps, while other models fail due to toolbar confusion or near-miss predictions, resulting in early termination. Green box = ground truth; blue = correct prediction; red = incorrect prediction. Further examples are in the Supplementary material H .
Appendix Section
Description
Section A
MedSPOT Dataset Card
Section B
Inference Protocol
Section C
Statistical Robustness
Section D
Data Provenance & Ethics
Section E
Failure Taxonomy Definitions
Section F
Insights
Table 12
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Value
Tasks
216 task-driven interaction sequences
Annotated Keyframes
597
Software Platforms
10
Average Steps per Task
2–3
Imaging Modalities
CT, MRI, PET, X-Ray, Ultrasound
Interface Categories
Appendix
Table 4: MedSPOT dataset composition.
Software
Type
Tasks
Modalities
License
Orthanc [ 17 ]
Server
10
CT, MRI, Ultrasound, X-ray, PET
GPL
ITK-SNAP [ 52 ]
Segmentation
20
CT, MRI, CBCT, NIfTI
GPL
Weasis [ 33 ]
Viewer
30
CT, MRI, Ultrasound, X-ray, PET
EPL 2.0
Bluelight [ 6 ]
Web Viewer
12
CT, MRI, Ultrasound, X-ray
MIT
MITK [ 43 ]
Viewer / Seg.
25
CT, MRI, Ultrasound, PET
BSD
RadiAnt [ 25 ]
Viewer
27
CT, MRI, US, X-ray, PET
Freeware
Appendix
Table 5: Dataset curation and benchmark software platforms included in MedSPOT .
Figure 9: Window-level adjustment workflow. This example illustrates a three-step interaction for modifying image windowing parameters. (1) The model must first locate and click the c/w icon in the viewer toolbar to open the window-level adjustment dialog. (2) Once the dialog appears, the model must confirm the selected center and width parameters by clicking the OK button. (3) Finally, the model applies the changes by clicking the Apply button. This example demonstrates a multi-stage GUI workflow in which each step reveals a new interface state.
Figure 10: File loading workflow. The task proceeds through three sequential actions. (1) The model must first click the Open File icon located in the viewer toolbar, which launches the system file explorer. (2) Within the file explorer interface, the model must identify and select the desired DICOM file from the list of available files. (3) Finally, the model confirms the selection by clicking the Open button to load the image into the viewer. This example highlights cross-interface grounding between application windows.
Figure 11: Software update notification workflow. This task involves navigating a menu hierarchy. (1) The model first identifies and clicks the Help menu in the top menu bar. (2) From the dropdown options that appear, it selects Check for Software Updates . (3) After the update status dialog appears, the model confirms the notification by clicking the OK button. This example illustrates hierarchical menu navigation within a GUI.
Figure 12: Lookup table (LUT) selection workflow. This task demonstrates modification of visualization settings in a medical imaging viewer. (1) The model first opens the LUT selection menu by clicking the LUT icon in the viewer toolbar. (2) From the available color mapping options in the dropdown palette, the model selects the Fire lookup table, which updates the rendering of the medical image.
Figure 13: Example annotation structure in MedSPOT. Each annotated step contains the GUI screenshot, the associated natural language instruction, a semantic description of the target GUI element, and the ground-truth bounding box identifying the element to be interacted with. Bounding boxes are represented using normalized coordinates, enabling consistent annotation across screenshots with different resolutions.
Model
Checkpoint
Precision
Coord. Format
K
Temp
GUI-Actor
microsoft/GUI-Actor-7B-Qwen2.5-VL
BF16
Norm [0,1]
5
0
UI-TARS
bytedance/UI-TARS-7B-SFT
BF16
Absolute px
1
0.3
OS-Atlas
OS-Copilot/OS-Atlas-Base-7B
BF16
[0,1000]
1
0
CogAgent
CogAgent-9B
BF16
[0,1000]
1
0
AGUVIS
AGUVIS-7B
BF16
Norm [0,1]
1
0
SeeClick
SeeClick
BF16
Norm [0,1]
1
0
Appendix
Table 6: Per-model inference configuration used for evaluation on MedSPOT .
Output Format
Models
Normalization Strategy
Normalized [0,1]
AGUVIS, UGround
No normalization required
[0,1000] range
OS-Atlas, CogAgent, SeeClick, Qwen3-VL
Divide coordinates by 1000
Absolute pixels
GPT-4o-mini, GPT-5, Gemma-3, LLaMA
Divide by image width W and height H
Native pixels
Qwen2-VL, Qwen2.5-VL, UI-TARS
Divide by W and H
Reasoning trace outputs
UI-TARS, Qwen2-VL, Qwen2.5-VL, Qwen3-VL
Apply think-block stripping before regex parsing
Appendix
Table 7: Model-specific coordinate extraction and normalization procedures used in evaluation.
Parameter
Value
Description
Tolerance δ
14 px patch overlap
Applied only to GUI-Actor; other models require the point to lie inside Bt
Top-K predictions
K=5 (GUI-Actor), K=1 (others)
GUI-Actor outputs multiple candidate coordinates
Retries
2 (UI-TARS), 1 (others)
UI-TARS retries once with temperature 0.3 on parse failure
Maximum tokens
512
Applied uniformly across all models
Appendix
Table 8: Shared inference hyperparameters used across all evaluated models.
Model
TCA (%)
95% CI (TCA)
SHR (%)
95% CI (SHR)
GUI-Actor
43.46
[36.92, 50.00]
49.58
[42.88, 56.24]
Qwen3-VL
35.05
[28.97, 41.13]
46.59
[40.61, 52.68]
UI-TARS
30.84
[24.30, 37.38]
41.93
[35.89, 48.68]
OS-Atlas
26.64
[20.56, 32.71]
55.01
[48.93, 60.76]
AGUVIS
26.64
[21.03, 32.71]
54.76
[48.75, 60.63]
CogAgent
15.42
[10.75, 20.56]
37.80
[30.48, 44.85]
Appendix
Table 9: Bootstrapped 95% confidence intervals for TCA and SHR across all models ( n=216 tasks, 1000 bootstrap resamples).
Figure 14: Example of a No Prediction (NP) failure. Although the target element “Add DICOM Data” is visible in the interface, the model output does not contain a valid coordinate pair, resulting in an empty prediction set P^t=∅ . This reflects a decoding or instruction-following failure rather than a spatial localization error.
Figure 15: Example of a Small Target (ST) failure. The actionable interface element occupies a very small portion of the GUI, requiring extremely precise localization. Even when the instruction is understood, predicted coordinates may fall outside the target due to the resolution limits of the vision encoder.
Figure 16: Example of a Near Miss (NM) failure. The predicted coordinate lies close to the correct UI element but outside the ground-truth bounding box. This reflects correct semantic grounding with insufficient localization precision.
Figure 17: Example of an Edge Bias (EB) failure. The model predicts a coordinate near the image boundary rather than the correct interface element, indicating a systematic bias in the spatial prediction distribution.
Figure 18: Example of a Toolbar Confusion (TC) failure. The predicted coordinate lies in the global toolbar region rather than the task-relevant element within the workspace, indicating a contextual grounding error.
Figure 19: Example of a Far Miss (FM) failure. The predicted coordinate is far from the correct UI element and corresponds to an unrelated interface control, indicating a breakdown in instruction understanding and visual grounding.
Figure 20: Scale and task distribution of MedSPOT .
Figure 21: Example of a slider-based interaction requiring drag input.
Figure 22: Example of multiple visually distinct controls leading to the same operation.
Figure 23: Example illustrating early termination in sequential evaluation.
Figure 24: Example of session-state changes affecting GUI layout and accessibility of target elements.
Figure 25: Easy Interface (ITK-SNAP): “Check for software updates”. The interface contains a small number of clearly separated toolbar elements. GUI-Actor, UI-TARS, and CogAgent correctly ground all three steps of the task, while GPT-5 fails only at Step 2. The minimal visual clutter and clear spatial separation of controls facilitate reliable grounding.
Figure 26: Hard Interface (RadiAnt): “Toggle DICOM tags in image overlay”. The dense toolbar layout causes all models to fail at Step 1. Predictions collapse toward the top boundary of the interface, producing edge-bias and toolbar-confusion failures.
Figure 27: Unsolved Task (Ginkgo CADx): “Add a text annotation to the image”. All evaluated models fail at Step 1. The target element is a small icon embedded within a dense toolbar, making it both visually ambiguous and difficult to localize precisely.
Pair
Mean IoU
Mean GIoU
AR (%)
Value
95% CI
Value
95% CI
≥ 0.50
≥ 0.75
A1 vs. A2
0.909
[0.899, 0.919]
0.898
[0.887, 0.910]
100.0
99.3
A1 vs. A3
0.842
[0.799, 0.885]
0.832
[0.790, 0.874]
100.0
97.8
A2 vs. A3
0.869
[0.858, 0.880]
0.864
[0.852, 0.876]
100.0
98.6
Overall
0.874
[0.840, 0.899]
0.865
[0.829, 0.892]
100.0
98.6
Appendix
Table 10: Pairwise Bounding Box Inter-Annotator Agreement. Mean IoU and GIoU are reported with 95% bootstrap confidence intervals (1,000 resamples, seed = 42). Agreement Rate (AR) is the proportion of frames where IoU exceeds the stated threshold.
Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.
Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1
Existing medical AI benchmarks lack process visibility, atomic skill evaluation, and integrated hallucination detection. We introduce MedBench v5, a redesigned benchmark for clinical multimodal models (language, vision-language, and agent systems) that moves from static QA to dynamic, process-oriented evaluation. MedBench v5 features: (1) a dual-dimensional framework combining Clinical Cognitive Responsiveness (13 sub-dimensions) and Medical Atomic Skills (4 agent environments), covering 63 tasks; (2) three switchable information-flow stressors (omission, contradiction, evidence delay) for factorized degradation analysis; (3) a dynamic process audit protocol with five reasoning nodes that produces model-specific failure fingerprints; (4) hallucination propagation monitoring across initiation, propagation, anchoring, and contradiction interaction-capturing silent hallucination. Experiments on frontier models show that strong overall task performance does not guarantee process stability: stressors mainly disrupt contradiction detection, diagnosis updating, hallucination propagation, and contradiction-based self-correction, while final evidence grounding can remain superficially stable. MedBench v5 provides a unified infrastructure for capability profiling, controllable stress testing, process auditing, and hallucination trajectory analysis in clinical AI evaluation.
Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging. However, existing medical AI benchmarks are fragmented: some support multi-turn dialogues but lack images, while others provide multimodal inputs but focus on single-turn QA tasks. To address this gap, we introduce IMCBench, an image-grounded, multi-turn medical conversation benchmark that pairs real, publicly available clinical images with synthetic patient profiles to simulate realistic patient-clinician interactions. Each conversation is evaluated across three clinical dimensions: safety, accuracy, and appropriate use of uncertainty in diagnosis. We benchmark eight multimodal frontier models across four model families (Claude, GPT, Nova, and Llama), scoring each on a 1-5 scale using LLM-as-Jury scoring calibrated against expert clinician annotations. Our results show that Claude Opus 4.6 achieves the highest overall score (3.61), followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29), though no model dominates all dimensions and safety degrades for both malignant and rare conditions (Δ = -0.27 each). Ablation studies further reveal that both visual input and EHR context contribute to safe guidance (safety drops of 0.18 and 0.23 on average when each is removed), with stronger models leveraging visual features more effectively. Together, these findings demonstrate that accurate clinical description does not guarantee safe patient guidance, motivating the need for multi-dimensional evaluation frameworks in medical AI.
Maria Xenochristou, Ashutosh Joshi, Korosh Vatanparvar +10