WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analytics
Authors: Yutong Chen, Zhike Tang, Zhihao Mai, Zhihao Shuai, Danli Luo, Jing Xu, Weikai Yang
Organizations: Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Nanyang Technological University, Singapore · School of Electronics and Information Technology (School of Microelectronics), Sun Yat-sen University, Guangzhou, China
Visual analytics (VA) enables sensemaking through interactive visualization, but effective analysis often requires experts to translate high-level intents into long sequences of interface operations and iteratively interpret visual feedback. We study whether modern vision-language models (VLMs) can take on this role as autonomous VA operators that observe the interface, plan multi-step exploration, execute interactions, and adapt based on intermediate visual feedback. To support systematic development and evaluation, we first introduce WidgetVA, a widget-centric agentic VA framework that standardizes interactive components as structured widgets with unified action (e.g., filter and zoom) and perception-query (e.g., selection summaries) APIs. This standardization supports two modes of system construction: wrapping an existing VA system to make it agent-operable without rebuilding it, and composing a new system from widgets as modular building blocks. To help agents coordinate across widgets rather than plan each interaction from scratch, each widget further packages reusable analytical workflows, giving agents more than a bare set of callable functions to plan over. Building on this framework, we present WidgetVABench, a benchmark of single- and multi-widget VA tasks that require agents to perform multi-step interactions to uncover evidence and produce verifiable results. Each task also provides fine-grained reference annotations so that WidgetVABench can score Answer, Reference Trace Similarity, and State separately rather than collapsing agent performance into one success score. Experiments across multiple VLMs show that our framework provides an effective scaffold for agentic VA, while the diagnostic measures expose persistent limitations for future work. The WidgetVA framework and WidgetVABench have been released in https://github.com/Hiverwin/widgetva.
Figures & tables
Figure 1: In WidgetVA , interactive views are exposed through a unified widget so that humans and agents can explore and analyze together in the same workspace. WidgetVA supports wrapping an existing VA page to make it agent-operable or composing a new system from widgets, and it provides reusable workflows to support multi-step planning. We also construct WidgetVABench to evaluate agent performance and exposes persistent limitations for future work.
Figure 2: Overview of our integrated analysis environment . (A) The Data Panel initializes an analysis session from uploaded tabular data or a supported visualization specification. (B) The Analysis Panel supports mixed-initiative analysis by allowing users to record useful insights and inspect the agent’s iterative analysis process. (C) The Visualization Workspace renders the current visualization state. (D) The Interaction Trace Panel records the analysis as a branch-aware provenance graph and links each analysis step to its corresponding iteration card and workspace state.
Type
Target
Path
Characteristic
Example
TK–PK
Known
Known
Specified analysis
Select the largest and smallest wind-farm regions, compare their average vibration values and proportions of high-vibration turbines, and identify the region that performs worse on both indicators.
TK–PU
Known
Unknown
Goal-directed analysis
Between the largest and smallest wind-farm regions, which requires a more urgent maintenance review? Support the answer with evidence.
TU–PK
Unknown
Known
Guided discovery
Inspect region size, vibration distributions, and anomalous turbine records in sequence, and report any noteworthy pattern discovered.
TU–PU
Unknown
Unknown
Open-ended exploration
Explore the wind-turbine dashboard and identify a phenomenon that warrants further investigation.
Table I: Task taxonomy based on target and path specification.
TK-PK
TK-PU
TU-PK
TU-PU
Setting
Answer
Trace
State
Answer
Trace
State
Answer
Trace
State
Answer
Trace
State
Model (Avg. A)
Single-widget tasks
S0
52.5
N/A
N/A
56.4
N/A
N/A
35.2
N/A
N/A
35.9
N/A
N/A
GPT (45.0)
54.2
N/A
N/A
56.7
N/A
N/A
37.3
N/A
N/A
34.5
N/A
N/A
Gemini (45.7)
33.3
N/A
N/A
31.6
N/A
N/A
16.2
N/A
N/A
23.0
N/A
N/A
Qwen (26.0)
S1
70.8
68.1
67.2
63.7
49.1
39.0
64.5
58.3
55.5
46.9
37.8
34.3
GPT (61.4)
Table II: Experimental results for single- and multi-widget tasks. N/A indicates that Trace and State are not applicable under S0 .
Figure 3: Scenario 2: finding an atmospheric filming window on an existing Vega-Lite page. (A) WidgetVA is connected directly to the existing visualization and receives the user’s filming goal (R3). (B) Cross-widget filtering and temporal brushing identify the relevant atmospheric conditions and progressively narrow the candidate period to September (R1). (C) Comparing early and late September reveals an atmosphere–reliability trade-off: late September provides more fog and drizzle but also substantially more rain. (D) The agent revisits the saved September states and drills down into rainy-day records (R2), verifying the higher disruption risk in late September and supporting Sept. 1–15 as the recommended filming window.
Vision-Language Models (VLMs) frequently misread values, hallucinate details, and confuse overlapping elements in charts. Current approaches rely solely on pixel interpretation, creating a Pixel-Only Bottleneck: agents treat interactive charts as static images, losing access to the structured specification that encodes exact values. We introduce Introspective and Interactive Visual Grounding (IVG), a framework that combines (1) spec-grounded introspection, which queries the underlying specification for deterministic evidence, with (2) view-grounded interaction, which manipulates the view to resolve visual ambiguity. To enable evaluation without VLM bias, we present iPlotBench, a benchmark of 500 interactive Plotly figures with 6,706 binary questions and ground-truth specifications. Experiments show that introspection improves data reconstruction fidelity, while the combination with interaction achieves the highest QA accuracy (0.81), with +6.7 % gains on overlapping geometries. We further demonstrate IVG in deployed agents that explore data autonomously and collaborate with human users in real time.
Sandboxed computer environments support multi-step reasoning with tools, executable programs, and persistent files, yet their extension from language models to vision-language models (VLMs) introduces a distinct state-management problem. Visual reasoning produces intermediate image-valued evidence---crops, masks, overlays, zoomed regions, and analytic renderings---that must remain addressable without accumulating unboundedly in multimodal context. We introduce VLM-in-Sandbox, a training-free framework for agentic multimodal reasoning in controlled computer environments. Its Visual Workspace registers generated artifacts in an image ledger, maintains a bounded active visual context, and lets the model explicitly promote selected evidence for subsequent inspection. This separates visual evidence generation, performed by sandbox tools, from visual evidence management. Across seven benchmarks and four base VLMs, VLM-in-Sandbox achieves the highest sample-weighted average accuracy among Vanilla VLM, Append-only Sandbox, and the proposed method. A compiler-matched 2×2 study on 1,260 examples further separates model-directed visibility from bounded retention: VLM-in-Sandbox reaches 66.27% accuracy with 18.6% fewer total tokens than the automatic, retain-all control. Over all 6,350 submitted GPT-4.1-mini examples, it produces 302 rescues and 142 regressions relative to Original Append-only. A local vLLM study with prefix caching confirms that the smaller request workload also reduces uncached tokens, time to first token, and end-to-end latency. These results identify explicit visual evidence state as a central abstraction for sandboxed VLM agents.
Hexiong Yang, Mingrui Chen, Jie Cao +1
NLPR&MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Science, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1
Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including 95.8% on V* and 35.3% on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler