cs.HCSep 4, 2026

MIVAIS: A Study Environment for Multi-Agent Mixed-Initiative Visual Analytics Applications

Authors: Tobias Stähle, Simon Schneider, Rita Sevastjanova, Mennatallah El-Assady

Organizations: ETH Zürich, Zürich, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Mixed-initiative Visual Analytics (VA) systems empower human users by interleaving human intuition with software agents and their machine intelligence. However, the development and rigorous evaluation of such systems remain constrained by engineering overhead. Developers must, e.g., implement complex, low-level state synchronization to manage asynchronous agent behaviors, while researchers struggle to capture the multimodal provenance required to study and evaluate human-AI collaboration. We present MIVAIS, a dual-layered research platform designed to abstract the structural complexities of mixed-initiative VA. First, it contributes a computational Infrastructure that standardizes human-software agent interaction, state synchronization, and communication between the agents. Second, it provides a declarative Study Environment that automatically logs multimodal human-AI telemetry - including application/system state, screen capture, audio, and additional sensor data - enabling seamless, in-situ user studies and post-session analysis. We technically validate our infrastructure by replicating three state-of-the-art systems (Podium, Voyager 2, and ProactiveVA). Furthermore, we evaluate the framework's expressiveness and efficiency through expert case studies with HCI and VA researchers, demonstrating how MIVAIS effectively lowers the barrier to prototyping and evaluating intelligent, co-adaptive interfaces.

Figures & tables

Figure 1. Overview of the MIVAIS framework. The MIVAIS Infrastructure (left) supports the implementation of modular, mixed-initiative Visual Analytics (VA) systems where software and human agents collaborate. The MIVAIS Study Environment (right) enables researchers to easily configure experiments, collect multi-modal data during participant sessions, and perform comprehensive post-analysis. Overview of the \appNameframework. The \appNameInfrastructure (left) supports the implementation of modular, mixed-initiative Visual Analytics (VA) systems where software and human agents collaborate. The \appNameStudy Environment (right) enables researchers to easily configure experiments, collect multi-modal data during participant sessions, and perform comprehensive post-analysis.
Figure 2. Architecture and data flow of a VA system built on the MIVAIS Infrastructure. Human and software agents interact with the application’s World State and each other, strictly mediated by a central Permission Guard. The backend handles state management and comprehensive audit logging, while an optional Study Gateway facilitates seamless integration with the broader Study Environment. In a system built on the \appNameInfrastructure, agents observe and act on the World State through the frontend or directly through the backend. Agents can send and receive messages through Communication Channels. The Permission Guard validates interactions with the World State and inter-agent communication based on the permission defined in the Agent Registry. The visual interface (frontend) and the backend are connected via a WebSocket communication that updates the state across the \appNameContext Provider (frontend) and World State in the backend. At runtime, Audit Logs are captured by collecting provenance and communication data from the agents and recording World State and permission updates. If the VA system is used with the Study Environment, the Study Gateway connects with the Study Environment and can be used to initiate a study session by setting agent roles, permissions, and initial World State. Further is collects the logs and can automatically validate participant tasks based on interactions or state updates.
Figure 3. In MIVAIS , a study and its tasks are defined in DSL YAML files. This example shows an excerpt from the Study Definition and one of the Task Definitions for a study on the reimplemented ProactiveVA system (with additional configuration details elided). The right part shows the deployed study and task, as well as the resulting study interface provided to participants. In \appName, a study and its tasks are defined in DSL \texttt{YAML} files. This example shows an excerpt from the Study Definition and one of the Task Definitions for a study on the reimplemented ProactiveVA system. The right part shows the deployed study and task, as well as the resulting study interface provided to participants.
Figure 4. The visual interface of MIVAIS for post-study analysis. The dashboard provides researchers with three primary analytical views: (I.) a Study Overview summarizing aggregate metrics and session progress, (II.) a Task Result Overview for evaluating specific task outcomes (e.g., questionnaires), and (III.) a comprehensive Replay View that synchronizes screen recordings, interaction timelines, and transcripts for deep inspection of individual sessions. The visual interface of MIVAIS for post-study analysis. The dashboard provides researchers with three primary analytical views: (I) a Study Overview summarizing aggregate metrics and session progress, (II) a Task Result Overview for evaluating specific task outcomes (e.g., questionnaires), and (III) a comprehensive Replay View that synchronizes screen recordings, interaction timelines, and transcripts for deep inspection of individual sessions.
Figure 5. Case study results implemented by the experts. UserRAG ( P1P_{1} ) and TopoAlign ( P2P_{2} ) are shown in deployed study mode, while PleaSQLarify ( P3P_{3} ) is shown in the Replay View of a study session.

Explore similar work

CardsList

  1. BONSAI: A Mixed-Initiative Workspace for Human-AI Co-Development of Visual Analytics Applications

    Apr 21, 2026Thilo Spinner, Matthias Miller, Fabian Sperrle-Roth +1Multi-Agent CollaborationHuman-in-the-Loop AI

    Thilo Spinner · Matthias Miller · Fabian Sperrle-Roth +1

  2. HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization

    Jun 25, 2026Kuangshi Ai, Patrick Phuoc Do, Chaoli WangScientific VisualizationData Visualization

    Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556, USA

  3. WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analytics

    Sep 21, 2026Yutong Chen, Zhike Tang, Zhihao Mai +4Tool-Using Vision-Language AgentsVisual Analytics

    Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Nanyang Technological University, Singapore · School of Electronics and Information Technology (School of Microelectronics), Sun Yat-sen University, Guangzhou, China