Interactive dashboards require users to reveal and connect evidence across stateful interactions. Although graphical user interface (GUI) agents could automate this process, existing dashboard benchmarks primarily report final answers or task success. They provide limited insight into whether failures arise from maintaining the analytical process, selecting actions, or grounding visual targets. We introduce DashAct, a progressive diagnostic benchmark that evaluates the same annotated dashboard interactions under systematically varied contextual and spatial support. DashAct contains 348 human-verified interaction trajectories with milestone dependencies and hierarchical target annotations. Its three stages evaluate end-to-end execution, verified-context action prediction, and target-conditioned grounding under four visual conditions. Aligned action and spatial annotations connect outcomes across these settings, providing diagnostic evidence for interpreting online failures. Experiments reveal persistent difficulties in action prediction and fine-grained grounding, alongside model-dependent responses to contextual and spatial support. A Qwen-based adaptation study further improves target localization and online Milestone Coverage, illustrating how DashAct can guide practical agent improvements.
Figures & tables
Figure 1: Overview of DashAct and its progressive diagnostic cascade. DashAct represents each dashboard task with a human-verified interaction trajectory and milestone-level process annotations. The same task process is evaluated under progressively stronger support, from end-to-end task execution to verified-context action prediction and target-specified local grounding.
Stage 1: End-to-End
Stage 2: Verified Context
Stage 3: Local
Model
Milestone Coverage
All-Milestone Completion
Answer Accuracy
Action Type Accuracy
Target Hit Rate
Action Success
Target Hit Rate
Closed-source models
Gemini 3 Flash Preview
62.7
42.9
42.6
63.5
37.1
31.3
58.4
GPT-5
29.9
14.9
9.6
62.2
6.0
6.6
32.5
Claude Sonnet 5
25.4
10.6
8.1
64.7
2.3
3.0
3.8
Open-source models
Table 1: Main results on DashAct (%) for the three evaluation stages: end-to-end execution, verified-context action prediction, and local grounding. Bold shows the best value in each column, and underlined shows the second-best.
Figure 2: Stage 1 Milestone Coverage (left) and Answer Accuracy (right), grouped by required milestone count. The black dashed line denotes the average across evaluated models within each milestone-count group.
Model
View Hit
Target Hit
Target Hit ∣ View
Closed-source models
Gemini 3 Flash Preview
67.6
37.1
54.9
GPT-5
55.5
6.0
10.8
Claude Sonnet 5
33.7
2.3
6.8
Open-source models
OpAgent-32B
47.3
29.3
61.9
Table 2: Stage 2 spatial localization performance (%). Target Hit ∣ View denotes target hit rate conditioned on the predicted point falling within the correct view.
Model
Overall
Controls
Marks
Closed-source models
Gemini 3 Flash Preview
58.4
72.6
48.7
GPT-5
32.5
44.3
24.4
Claude Sonnet 5
3.8
6.2
2.1
Open-source models
OpAgent-32B
27.2
39.2
14.0
Table 3: Stage 3 local grounding accuracy (%), reported overall and by dashboard target type: interface controls and visualization marks. Bold and underlining indicate the best and second-best results in each column, respectively.
Figure 3: Milestone-level recovery profiles under progressively stronger support. Each stacked bar groups comparable milestones by the earliest condition under which they succeed; milestones failing under all conditions are unresolved.