cs.HCAug 17, 2026
SaveMUSE: An Interactive Meta-Agent for Understanding and Steering LLM-powered Data Science Systems
Organizations: Purdue University West Lafayette, Indiana, USA · Microsoft Research Redmond, Washington, USA
Abstract
Recent advances in large language models have enabled a new class of agentic data science systems that allow users to complete complex data science workflows through natural language. Although these systems can significantly reduce manual effort, it remains difficult to diagnose their behavior and steer the reasoning process when failures or unexpected outputs occur. We present MUSE, an interactive meta-agent that enhances user understanding and control of agentic data science systems by (1) dynamically restructuring low-level execution traces into multiple semantic levels that support navigation from high-level overviews to low-level implementation details; (2) enabling users to reference specific workflow steps in context to ask grounded questions, provide feedback, and revise problematic steps without manually locating relevant execution history; and (3) supporting mixed-initiative steering by surfacing suspicious steps for inspection, scaffolding the repair process, and translating user repair intent into contextualized instructions for the underlying agent. In a between-subjects study (n = 15), MUSE improved task efficiency and increased users' confidence in understanding and steering agentic data science workflows.
Figures & tables
Figure 1. Left: An underlying data science agent that produces unstructured execution logs, including code edits, tool calls, and runtime outputs. Middle: MUSE serves as a meta-agent that transforms these raw logs into structured logs with multiple semantic levels and monitors the process to detect suspicious behavior and surface warnings at runtime. Right: Users interact with the structured logs to adjust semantic granularity, inspect surfaced issues, specify what to verify through scaffolded, in-context verification plans, and steer the agent in context by providing feedback and revising specific workflow steps.
| PID | Gender | Background | AI Tool Usage | Degree |
|---|---|---|---|---|
| P1 | Male | Engineering | Cursor | PhD |
| P2 | Male | Engineering | OpenCode | PhD |
| P3 | Male | Engineering | Cursor | PhD |
| P4 | Male | Engineering | Claude Code, Cursor | PhD |
| P5 | Male | Engineering | Claude Code, Codex | PhD |
| P6 | Female | Statistics | ChatGPT, Gemini | PhD |
Table 1. Demographics and background of the formative study participants.
Figure 2. MUSE dynamically converts the underlying DS agent’s raw execution log stream—including reasoning traces, tool invocations, code, artifacts, and errors—into semantic chunks using explicit step-completion markers.
Figure 3. MUSE provides an interactive interface for understanding and steering agentic data science systems. \smallA⃝ Users can adjust semantic levels (L1–L5) to navigate between high-level summaries and detailed execution traces. \smallB⃝ MUSE surfaces localized warnings on workflow steps to highlight potentially problematic behaviors. \smallC⃝ Users can drag and drop a step into the chatbox. \smallD⃝ Users ask follow-up questions in context without needing to reconstruct execution details. \smallE⃝ Users can highlight outputs (e.g., metrics) to trigger an in-situ verification process.
Figure 4. MUSE surfaces warnings, allowing users to inspect issues, request explanations, or revise the workflow.
Figure 5. MUSE asks follow-up questions and builds a verification script to validate the result.
| TID | Dataset | Difficulty | Task Goal | Key Features | Target |
|---|---|---|---|---|---|
| T1 | Food Delivery Time | Easy | Predict delivery time for reliable ETA estimation. | Distance, weather, traffic, vehicle type, etc. | Delivery time |
| T2 | Demand Forecasting | Easy | Forecast product demand for inventory planning. | Date, store, item | Sales |
| T3 | Telco Customer Churn | Medium | Identify customers likely to cancel subscriptions. | Tenure, Internet service, contract, monthly charges, etc. | Churn (Yes/No) |
| T4 | Customer Lifetime Value | Medium | Predict future customer revenue from historical transactions. | Quantity, unit price, customer ID, invoice date, etc. | Future revenue |
| T5 | MovieLens 100k | Hard | Predict user preferences for unseen movies. | User ratings, movie metadata, genre indicators, demographics | Rating (1–5) |
| T6 | Online News Popularity | Hard | Predict article popularity before publication. | Content statistics, keyword metrics, topic features, sentiment, etc. | Shares |
Table 2. User study tasks.
| ID | Condition | Background | Expert | Gender |
|---|---|---|---|---|
| R1 | A | Business Analytics | Yes | Male |
| R2 | A | Computer Science | Yes | Male |
| R3 | A | Finance | No | Male |
| R4 | A | Architecture | No | Male |
| R5 | A | Finance | No | Male |
| L1 | B | Aeronautics & Astronautics | No | Male |
Table 3. Participant assignment, background, self-reported agent expertise, and gender.
| Raw Logs | DiLLS | MUSE | ||||||||||||||||
| R1 | R2 | R3 | R4 | R5 | Avg | L1 | L2 | L3 | L4 | L5 | Avg | M1 | M2 | M3 | M4 | M5 | Avg | |
| Task 1 | ||||||||||||||||||
| Task ID | T1 | T1 | T2 | T3 | T3 | T1 | T1 | T2 | T4 | T5 | T1 | T2 | T2 | T3 | T4 | |||
| Model | Opus | Haiku | Opus | Opus | Opus | Opus | Opus | Opus | Opus | Opus | Haiku | Opus | Haiku | Opus | Opus | |||
| Time (min) | 26 | 21 | 30 | 30 | 30 | 27.4 | 21 | 10 | 30 | 16 | 26 | 20.6 | 14 | 17 | 16 | 25 | 30 | 20.4 |
| Attempts | 1 | 1 | 1 | 1 | 1 | 1.0 | 1 | 1 | 1 | 2 | 1 | 1.2 | 1 | 1 | 1 | 1 | 2 | 1.2 |
Table 4. Per-participant assignments and outcomes by condition. S = success, F = failure.
Figure 6. User confidence in the final data science outcomes under conditions A, B, and C.
Figure 7. User responses to the NASA TLX questionnaire (*: based on the ANOVA test).
Figure 8. 7-point Likert scale evaluations (1 = strongly disagree, 7 = strongly agree) of five comparison statements: S1 (understanding), S2 (error localization), S3 (verification), S4 (trust), and S5 (control).
Figure 9. Representative user event timelines across conditions in normalized time.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10. High-level control loop of the skill-based data science pipeline. The orchestrator executes or dispatches a stage, emits a completion signal, and then invokes the progress checker to determine the next action.
Figure 11. Relationship between MUSE and ds-pipeline . ds-pipeline advances the staged workflow and writes artifacts to the session sandbox, while MUSE reads those artifacts to answer user questions about progress, warnings, and results. When the interaction requires revising or continuing execution, MUSE hands control back to ds-pipeline .
| CONTEXT |
| You are given one semantic chunk from an AI-driven data workflow, including its execution log, script context, and report context. |
| OBJECTIVE |
| Produce a structured representation of this chunk that can support five levels of semantic detail in the user interface, from stage-level overview to implementation trace. |
| GUIDELINES |
| 1. Use the chunk to infer what was accomplished in this workflow step. |
| 2. Summarize the step in natural language before exposing detailed actions. |
Table 5. Prompt template used for semantic chunk translation.
| CONTEXT |
| You are given one semantic step from an AI-driven data analysis workflow. Each step contains execution evidence, including the raw execution log, the current step’s script context, and the report context. |
| OBJECTIVE |
| Determine whether the current step exhibits suspicious behavior that should be surfaced to the user as a warning. |
| GUIDELINES |
| 1. Detect warnings only when they are supported by concrete evidence in the provided execution trace. |
| 2. Use the script context as the primary source for detecting data-related issues. |
Table 6. The prompt used for warning detection in MUSE .
| CONTEXT |
| You are a QA assistant for a data science pipeline application. A separate execution agent runs the pipeline and writes artifacts into a session workspace. Your role is to answer the user’s questions about the current pipeline session without modifying files or rerunning work. |
| OBJECTIVE |
| Answer the user’s question about the selected workflow step or artifact from the available session context. |
| GUIDELINES |
| 1. Answer based only on the current session context and any read-only inspection you perform. |
| 2. Prefer progress artifacts and stage reports when they conflict with setup summaries. |
Table 7. Prompt template used by MUSE for contextualized step referencing.
| CONTEXT |
| You are continuing an existing workflow run. A previously executed step has been flagged as suspicious and should be revised in context. |
| OBJECTIVE |
| Translate the user’s high-level revision intent into a contextualized handoff request for the underlying DS agent. |
| GUIDELINES |
| 1. Treat the selected step as the restart point for revision. |
| 2. Use the warning context to understand why the step is being revised. |
Table 8. Template used to construct contextualized revision handoff requests.
| CONTEXT |
| You are helping a user verify a file or selected output from an AI-driven data workflow. The user has highlighted a specific file or text span and wants to check whether the result is correct or well supported. |
| OBJECTIVE |
| Generate a small set of concrete verification options grounded in the selected file or content. |
| GUIDELINES |
| 1. Base each verification option on the selected file or text span. |
| 2. Focus on actionable checks related to correctness, data quality, consistency, or workflow integrity. |
Table 9. Prompt template used to generate scaffolded verification options.
| CONTEXT |
| You are verifying a user-selected output from an existing workflow session. The workflow artifacts are stored in a session-scoped sandbox, and your role is to inspect them without modifying canonical pipeline outputs. |
| OBJECTIVE |
| Carry out the selected verification request using the current session context and determine whether the result is supported by the available evidence. |
| GUIDELINES |
| 1. Inspect only files and artifacts relevant to the selected output. |
| 2. Use the current session sandbox as the primary evidence source. |
Table 10. Prompt template used for verification execution.
| Detailed Warning Heuristics |
|---|
| Data-level Warnings |
| Aggressive Data Removal. This warning is triggered when the script removes rows or columns too broadly, destructively, or without sufficient justification, such that important information may be discarded and the remaining dataset may become biased or unrepresentative. |
| Overwriting Raw Data. This warning applies when transformed or cleaned outputs are written back to the original data source, so that the raw input is no longer preserved for reproducibility, auditing, or recovery. |
| Hard-coded Thresholds or Magic Numbers. This warning captures the use of fixed cutoffs, constants, binning boundaries, or manually chosen numeric rules that are embedded in code without visible empirical, domain, or methodological justification. |
| Suspicious Joins or Merges. This warning is raised when datasets are merged in ways that may duplicate observations, drop records, or misalign entities, especially when join keys, join types, or cardinality checks are not clearly validated. |
| Lost Categorical Semantics. This warning occurs when categorical variables are converted into in-memory categorical codes or labels, and then saved to flat outputs such as CSV without preserving the metadata needed to recover their original semantic meaning. |
Table 11. Detailed descriptions of warning heuristics in MUSE .
Figure 12. An example of the multi-level semantic representation for a data cleaning stage.
More in this topic
Human-AI Interaction
Human-AI interaction studies how people and AI systems communicate, collaborate, make decisions, and shape one another over time.
See all 330 papers