cs.CVMar 20, 2026
SaveMedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI
Organizations: Gaash Research Lab, National Institute of Technology Srinagar, India · e& Group, UAE · Mohammad Bin Zayed University of Artificial Intelligence (MBZUAI), UAE · King Abdullah University of Science and Technology (KAUST), Saudi Arabia
Abstract
Despite the rapid progress of Multimodal Large Language Models (MLLMs), their ability to perform reliable visual grounding in high-stakes clinical software environments remains underexplored. Existing GUI benchmarks largely focus on isolated, single-step grounding queries, overlooking the sequential, workflow-driven reasoning required in real-world medical interfaces, where tasks evolve across independent steps and dynamic interface states. We introduce MedSPOT, a workflow-aware sequential grounding benchmark for clinical GUI environments. Unlike prior benchmarks that treat grounding as a standalone prediction task, MedSPOT models procedural interaction as a sequence of structured spatial decisions. The benchmark comprises 216 task-driven videos with 597 annotated keyframes, in which each task consists of 2 to 3 interdependent grounding steps within realistic medical workflows. This design captures interface hierarchies, contextual dependencies, and fine-grained spatial precision under evolving conditions. To evaluate procedural robustness, we propose a strict sequential evaluation protocol that terminates task assessment upon the first incorrect grounding prediction, explicitly measuring error propagation in multi-step workflows. We further introduce a comprehensive failure taxonomy, including edge bias, small-target errors, no prediction, near miss, far miss, and toolbar confusion, to enable systematic diagnosis of model behavior in clinical GUI settings. By shifting evaluation from isolated grounding to workflow-aware sequential reasoning, MedSPOT establishes a realistic and safety-critical benchmark for assessing multimodal models in medical software environments. Code and data are available at: https://github.com/Tajamul21/MedSPOT.
Figures & tables
| Benchmark | Size | Multi-Step | Step Eval. | Sequential | Failure | Medical |
| Tasks | Protocol | Taxonomy | Software | |||
| ScreenSpot [ 7 ] | 1,272 | × | × | × | × | × |
| ScreenSpot-Pro [ 20 ] | 1,581 | × | ✓ | × | × | × |
| OSWorld-G [ 48 ] | 564 | ✓ | × | × | × | × |
| OmniACT [ 18 ] | 9,802 | ✓ | ✓ | × | × | × |
| AssistGUI [ 11 ] | 100 | ✓ | ✓ | × | × | × |
Table 1: Comparison of MedSPOT against existing GUI grounding benchmarks. Green cells indicate the presence of a capability, red cells indicate absence. MedSPOT is the only benchmark that jointly evaluates workflow-aware sequential grounding, early-termination protocol, structured failure taxonomy, and medical software environments.
Figure 1: Comparison of interface complexity. (a) Medical GUI environments are densely structured, hierarchically organized, and semantically rich with domain-specific terminology. (b) General GUI environments typically involve simpler layouts and short-horizon navigation tasks.
Figure 2: Overview of the MedSPOT dataset construction and sequential evaluation pipeline.
Figure 3: Overview of the MedSPOT annotation pipeline. GUI interaction videos are recorded from real clinical software, segmented into causally consistent decision frames, and annotated with step-level instructions, semantic targets, bounding boxes, and action types to construct sequential grounding tasks.
Figure 4: MedSPOT dataset statistics illustrating overall scale, task category distribution, and per-software coverage across ten clinical platforms.
| Model Name | Params | SHR(%) | S1A(%) | WPS | TCA(%) |
| Closed Source Models | |||||
| GPT 4o mini [ 28 ] | - | 4.91 | 5.14 | 0.051 | 0.0 |
| GPT 5 [ 35 ] | - | 16.4 | 16.5 | 0.185 | 2.8 |
| Open Source Models | |||||
| Llama 3.2 Vision-Instruct [ 13 ] | 11B | 0.0 | 0.0 | 0.0 | 0.0 |
| Qwen2-VL-Instruct [ 41 ] | 7B | 1.83 | 1.87 | 0.02 | 0.0 |
Table 2: Comparison of state-of-the-art MLLMs on MedSPOT-Bench under the strict sequential evaluation protocol. We report Step Hit Rate ( SHR ), Step-1 Accuracy ( S1A ), Weighted Prefix Score ( WPS ), and Task Completion Accuracy ( TCA ), where higher values indicate stronger spatial precision and sustained workflow consistency. Best results are highlighted in bold green and second-best results are underlined in blue.
Figure 5: Comparison of representative models across Step-1 Accuracy ( S1A ), Step Hit Rate ( SHR ), and Task Completion Accuracy ( TCA ). The consistent gap between S1A and TCA highlights compounding sequential errors under strict workflow constraints.
| Model Name | Params | Edge Bias | Far Miss | Toolbar Confusion | Near Miss | No Prediction | Small Target |
| Open Source Models | |||||||
| Llama 3.2 Vision-Instruct [ 13 ] | 11B | 49 | 7 | 4 | 0 | 134 | 19 |
| Qwen2-VL-Instruct [ 41 ] | 7B | 129 | 26 | 28 | 0 | 0 | 27 |
| DeepSeek-VL2 [ 47 ] | 16B | 30 | 152 | 0 | 2 | 0 | 30 |
| Gemma 3 [ 37 ] | 27B | 34 | 44 | 110 | 0 | 0 | 25 |
| Mistral-3-Instruct [ 21 ] | 8B | 109 | 64 | 13 | 2 | 0 | 26 |
Table 3: Failure taxonomy distribution on MedSPOT-Bench. Open-source models are shaded in blue and closed-source models in red. Worst failure counts are highlighted in bold red , while lowest values are underlined blue .
Figure 6: Per-software performance analysis. (a) Average TCA across models, ranking clinical applications by difficulty. (b) Per-model TCA breakdown across ten medical software platforms, highlighting interface-dependent performance variability.
Figure 7: Ablation studies on evaluation design and spatial stratification. (a) Sequential evaluation significantly lowers task-level scores due to error compounding. (b) Small UI elements consistently exhibit lower accuracy across all threshold configurations.
Figure 8: Qualitative comparison on the task "Delete the selected study from the browser menu" (DICOMscope). GUI-Actor completes all steps, while other models fail due to toolbar confusion or near-miss predictions, resulting in early termination. Green box = ground truth; blue = correct prediction; red = incorrect prediction. Further examples are in the Supplementary material H .
| Appendix Section | Description |
| Section A | MedSPOT Dataset Card |
| Section B | Inference Protocol |
| Section C | Statistical Robustness |
| Section D | Data Provenance & Ethics |
| Section E | Failure Taxonomy Definitions |
| Section F | Insights |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Property | Value |
| Tasks | 216 task-driven interaction sequences |
| Annotated Keyframes | 597 |
| Software Platforms | 10 |
| Average Steps per Task | 2–3 |
| Imaging Modalities | CT, MRI, PET, X-Ray, Ultrasound |
| Interface Categories | |
Table 4: MedSPOT dataset composition.
| Software | Type | Tasks | Modalities | License |
| Orthanc [ 17 ] | Server | 10 | CT, MRI, Ultrasound, X-ray, PET | GPL |
| ITK-SNAP [ 52 ] | Segmentation | 20 | CT, MRI, CBCT, NIfTI | GPL |
| Weasis [ 33 ] | Viewer | 30 | CT, MRI, Ultrasound, X-ray, PET | EPL 2.0 |
| Bluelight [ 6 ] | Web Viewer | 12 | CT, MRI, Ultrasound, X-ray | MIT |
| MITK [ 43 ] | Viewer / Seg. | 25 | CT, MRI, Ultrasound, PET | BSD |
| RadiAnt [ 25 ] | Viewer | 27 | CT, MRI, US, X-ray, PET | Freeware |
Table 5: Dataset curation and benchmark software platforms included in MedSPOT .
Figure 9: Window-level adjustment workflow. This example illustrates a three-step interaction for modifying image windowing parameters. (1) The model must first locate and click the c/w icon in the viewer toolbar to open the window-level adjustment dialog. (2) Once the dialog appears, the model must confirm the selected center and width parameters by clicking the OK button. (3) Finally, the model applies the changes by clicking the Apply button. This example demonstrates a multi-stage GUI workflow in which each step reveals a new interface state.
Figure 10: File loading workflow. The task proceeds through three sequential actions. (1) The model must first click the Open File icon located in the viewer toolbar, which launches the system file explorer. (2) Within the file explorer interface, the model must identify and select the desired DICOM file from the list of available files. (3) Finally, the model confirms the selection by clicking the Open button to load the image into the viewer. This example highlights cross-interface grounding between application windows.
Figure 11: Software update notification workflow. This task involves navigating a menu hierarchy. (1) The model first identifies and clicks the Help menu in the top menu bar. (2) From the dropdown options that appear, it selects Check for Software Updates . (3) After the update status dialog appears, the model confirms the notification by clicking the OK button. This example illustrates hierarchical menu navigation within a GUI.
Figure 12: Lookup table (LUT) selection workflow. This task demonstrates modification of visualization settings in a medical imaging viewer. (1) The model first opens the LUT selection menu by clicking the LUT icon in the viewer toolbar. (2) From the available color mapping options in the dropdown palette, the model selects the Fire lookup table, which updates the rendering of the medical image.
Figure 13: Example annotation structure in MedSPOT. Each annotated step contains the GUI screenshot, the associated natural language instruction, a semantic description of the target GUI element, and the ground-truth bounding box identifying the element to be interacted with. Bounding boxes are represented using normalized coordinates, enabling consistent annotation across screenshots with different resolutions.
| Model | Checkpoint | Precision | Coord. Format | K | Temp |
| GUI-Actor | microsoft/GUI-Actor-7B-Qwen2.5-VL | BF16 | Norm [0,1] | 5 | 0 |
| UI-TARS | bytedance/UI-TARS-7B-SFT | BF16 | Absolute px | 1 | 0.3 |
| OS-Atlas | OS-Copilot/OS-Atlas-Base-7B | BF16 | [0,1000] | 1 | 0 |
| CogAgent | CogAgent-9B | BF16 | [0,1000] | 1 | 0 |
| AGUVIS | AGUVIS-7B | BF16 | Norm [0,1] | 1 | 0 |
| SeeClick | SeeClick | BF16 | Norm [0,1] | 1 | 0 |
Table 6: Per-model inference configuration used for evaluation on MedSPOT .
| Output Format | Models | Normalization Strategy |
| Normalized | AGUVIS, UGround | No normalization required |
| range | OS-Atlas, CogAgent, SeeClick, Qwen3-VL | Divide coordinates by |
| Absolute pixels | GPT-4o-mini, GPT-5, Gemma-3, LLaMA | Divide by image width and height |
| Native pixels | Qwen2-VL, Qwen2.5-VL, UI-TARS | Divide by and |
| Reasoning trace outputs | UI-TARS, Qwen2-VL, Qwen2.5-VL, Qwen3-VL | Apply think-block stripping before regex parsing |
Table 7: Model-specific coordinate extraction and normalization procedures used in evaluation.
| Parameter | Value | Description |
| Tolerance | px patch overlap | Applied only to GUI-Actor; other models require the point to lie inside |
| Top-K predictions | (GUI-Actor), (others) | GUI-Actor outputs multiple candidate coordinates |
| Retries | (UI-TARS), (others) | UI-TARS retries once with temperature on parse failure |
| Maximum tokens | Applied uniformly across all models |
Table 8: Shared inference hyperparameters used across all evaluated models.
| Model | TCA (%) | 95% CI (TCA) | SHR (%) | 95% CI (SHR) |
| GUI-Actor | 43.46 | [36.92, 50.00] | 49.58 | [42.88, 56.24] |
| Qwen3-VL | 35.05 | [28.97, 41.13] | 46.59 | [40.61, 52.68] |
| UI-TARS | 30.84 | [24.30, 37.38] | 41.93 | [35.89, 48.68] |
| OS-Atlas | 26.64 | [20.56, 32.71] | 55.01 | [48.93, 60.76] |
| AGUVIS | 26.64 | [21.03, 32.71] | 54.76 | [48.75, 60.63] |
| CogAgent | 15.42 | [10.75, 20.56] | 37.80 | [30.48, 44.85] |
Table 9: Bootstrapped 95% confidence intervals for TCA and SHR across all models ( tasks, 1000 bootstrap resamples).
Figure 14: Example of a No Prediction (NP) failure. Although the target element “Add DICOM Data” is visible in the interface, the model output does not contain a valid coordinate pair, resulting in an empty prediction set . This reflects a decoding or instruction-following failure rather than a spatial localization error.
Figure 15: Example of a Small Target (ST) failure. The actionable interface element occupies a very small portion of the GUI, requiring extremely precise localization. Even when the instruction is understood, predicted coordinates may fall outside the target due to the resolution limits of the vision encoder.
Figure 16: Example of a Near Miss (NM) failure. The predicted coordinate lies close to the correct UI element but outside the ground-truth bounding box. This reflects correct semantic grounding with insufficient localization precision.
Figure 17: Example of an Edge Bias (EB) failure. The model predicts a coordinate near the image boundary rather than the correct interface element, indicating a systematic bias in the spatial prediction distribution.
Figure 18: Example of a Toolbar Confusion (TC) failure. The predicted coordinate lies in the global toolbar region rather than the task-relevant element within the workspace, indicating a contextual grounding error.
Figure 19: Example of a Far Miss (FM) failure. The predicted coordinate is far from the correct UI element and corresponds to an unrelated interface control, indicating a breakdown in instruction understanding and visual grounding.
Figure 20: Scale and task distribution of MedSPOT .
Figure 21: Example of a slider-based interaction requiring drag input.
Figure 22: Example of multiple visually distinct controls leading to the same operation.
Figure 23: Example illustrating early termination in sequential evaluation.
Figure 24: Example of session-state changes affecting GUI layout and accessibility of target elements.
Figure 25: Easy Interface (ITK-SNAP): “Check for software updates”. The interface contains a small number of clearly separated toolbar elements. GUI-Actor, UI-TARS, and CogAgent correctly ground all three steps of the task, while GPT-5 fails only at Step 2. The minimal visual clutter and clear spatial separation of controls facilitate reliable grounding.
Figure 26: Hard Interface (RadiAnt): “Toggle DICOM tags in image overlay”. The dense toolbar layout causes all models to fail at Step 1. Predictions collapse toward the top boundary of the interface, producing edge-bias and toolbar-confusion failures.
Figure 27: Unsolved Task (Ginkgo CADx): “Add a text annotation to the image”. All evaluated models fail at Step 1. The target element is a small icon embedded within a dense toolbar, making it both visually ambiguous and difficult to localize precisely.
| Pair | Mean IoU | Mean GIoU | AR (%) | |||
| Value | 95% CI | Value | 95% CI | 0.50 | 0.75 | |
| A1 vs. A2 | 0.909 | [0.899, 0.919] | 0.898 | [0.887, 0.910] | 100.0 | 99.3 |
| A1 vs. A3 | 0.842 | [0.799, 0.885] | 0.832 | [0.790, 0.874] | 100.0 | 97.8 |
| A2 vs. A3 | 0.869 | [0.858, 0.880] | 0.864 | [0.852, 0.876] | 100.0 | 98.6 |
| Overall | 0.874 | [0.840, 0.899] | 0.865 | [0.829, 0.892] | 100.0 | 98.6 |
Table 10: Pairwise Bounding Box Inter-Annotator Agreement. Mean IoU and GIoU are reported with 95% bootstrap confidence intervals (1,000 resamples, seed = 42). Agreement Rate (AR) is the proportion of frames where IoU exceeds the stated threshold.
More in this topic
GUI Grounding
Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.
See all 57 papers