Data journalism, the practice of using data analysis to surface newsworthy stories, depends increasingly on the ability of reporters and investigative journalists to uncover trends, disparities, and accountability narratives. In practice, exploring large structured datasets remains slow and brittle: journalists must navigate hundreds of variables across many datasets over years, understand data coding conventions, and write non-trivial analysis code while hypotheses evolve. Although LLMs are often touted as "ask in English, get SQL/answers," real newsroom workflows expose recurring failures, e.g., schema mismatches and drift, misread domain semantics and units, and silent assumptions. We present DataWeave, a system that addresses these needs by combining conversational interaction, schema grounding, analytical planning, and executable query generation to support exploratory analysis over structured data. Rather than treating LLMs as autonomous answer engines, DataWeave frames them as interactive partners whose outputs can be inspected, corrected, and steered as hypotheses shift. We present a case study with professional journalists using our system to analyze the U.S. Department of Education's Integrated Postsecondary Education Data System (IPEDS), a high-stakes public dataset with substantial domain semantics and frequent schema updates. We also report how deployment experience and iterative refinement shaped the current DataWeave architecture and its analytical workflow. Our findings distill design principles and deployment lessons for trustworthy human-LLM collaboration in structured data analysis.
Figures & tables
Figure 1: Illustration of the DataWeave interaction workflow. A user poses a natural-language query over higher-education data, from which DataWeave constructs inspectable analytical artifacts including analytical assumptions, executable queries, and results (tables or visualizations). These artifacts are surfaced to the user to support verification, correction, and iterative refinement.
Figure 2: DataWeave architecture. User objectives are interpreted and clarified, grounded through objective-conditioned schema retrieval, translated into analytical plans and read-only SQL, and executed with bounded recovery. Queries, results, and errors are preserved as analytical traces to support transparency and reproducibility.
Design Principle
What this means
How this is realized in DataWeave
Permission Over Forgiveness
Ambiguous analytical intent should be resolved before unsupported assumptions propagate into the analysis.
DataWeave requests clarification when key information such as scope, terminology, or temporal context is ambiguous, while preserving the user’s analytical objective.
Executable Queries as Inspectable Analytical Artifacts
Executable queries and intermediate results should remain inspectable rather than being hidden behind natural-language answers.
DataWeave grounds analytical responses in executable SQL and preserves queries, returned results, and provenance as part of the analytical trace.
Objective-Conditioned Schema Grounding
LLM reasoning should use compact schema and semantic context relevant to the current analytical objective.
DataWeave retrieves relevant tables, variables, descriptions, and coded-value semantics on demand rather than exposing the complete schema to the model.
Error Exposure Over Silent Robustness
Analytical systems should favor explicit error exposure and recoverable failure states over silent corrections that may alter analytical intent.
DataWeave treats execution errors as informative signals rather than exceptional conditions to be hidden. Failures are surfaced directly to the user, and any automated recovery is deliberately bounded and constrained.
Guided Agent Delegation
LLMs may coordinate analytical steps dynamically, while user intent and interpretation remain under human control.
DataWeave allows the agent to sequence schema discovery, planning, query formulation, and recovery while using clarification and inspectable traces to preserve user oversight.
Table 1: Condensed view of our design principles, what they mean, and how DataWeave realizes them.
Metric
Value
Scale
Users
18
Sessions
298
Analytical steps
797
Avg. steps/session
2.67
Session Depth
Table 2: Summary of deployment usage.
Metric
Mean
Median
Columns inferred per step
8.94
6
Columns used per step
6.03
5
Column overlap
5.38
5
Overlap ratio (used covered by inferred)
0.86
1.00
Overlap ratio (inferred utilized)
0.64
0.67
Used-only columns (not inferred)
0.65
0
Table 3: Alignment between planner-inferred columns and columns actually used in execution (deployment logs). Higher overlap indicates stronger schema grounding.
Figure 3: Distribution of analytical patterns across real-world analytical steps. (a) Each step is assigned one dominant analytical intent (Data quality validation with n=1 not shown). (b) Steps may also receive multiple analytical dimension tags (ADT) capturing additional analytical properties.
Figure 4: Most frequently used columns across real-world analytical steps. Core identifiers (e.g., unitid , instnm ) and enrollment-related variables dominate usage, reflecting consistent grounding in entity resolution and enrollment analysis.
Figure 5: Distribution of planner-inferred versus executed columns per analytical step. The planner typically proposes a broader column set, while execution uses a smaller subset. This pattern indicates conservative schema grounding followed by selective use during execution.
Figure 6: Column footprint across step progression within sessions. Thin lines show individual sessions and the bold line shows the median. Later steps often involve broader column usage, reflecting refinement and deeper investigation.
Figure 7: Answer correctness across the evaluated DataWeave configurations. The earlier prototype baseline is evaluated on a 100-question subset covering the five most recent years, while the agentic configurations are evaluated on the full 400-question benchmark. Error bars show Wilson 95% confidence intervals.
Figure 8: Operational behavior of the evaluated configurations. Tool-call statistics are available for the agentic configurations; the earlier prototype harness does not record tool calls. Latency is reported for all evaluated runs.
Figure 9: Retrospective case study aligning DataWeave analyses with a 2021 Chronicle of Higher Education article on graduate enrollment trends. The figure shows selected steps in a reconstructed analytical workflow, where DataWeave queries reproduce reported statistics. Arrows link computed outputs to corresponding claims and tables in the article, illustrating how conversational analysis can support journalistic verification and story development.
Multimodal data analysis, which answers questions over relational tables, text, and images, has attracted growing attention in the data management community. Large language models (LLMs) enable such analysis in natural language by generating analysis plans over relational and semantic operators. However, LLM-generated plans are error-prone: a plan may silently compute something other than what was asked, fail during execution, or return a result that misses the question. This paper presents WeaveData, a multimodal data analysis system with self-critiquing and self-evolving LLM plans. First, WeaveData generates a typed logical plan for each question and critiques it step by step before execution, and it checks the executed result against the question afterwards. Second, WeaveData evolves a plan that fails or misses the question: it diagnoses the failure with the actual data, reuses the results that remain valid, and accumulates planning experience for later questions. Third, WeaveData grounds planning in a metadata knowledge graph of all modalities, clarifies ambiguous questions with the user, and backs every model judgment with evidence in an interactive notebook. We demonstrate WeaveData on two public multimodal datasets.
Min Jia, Shihao Zhou, Jun-Peng Zhu +9
Northwest A&F University · PingCAP · East China Normal University +1
LLM-based data-analysis tools are increasingly used to help users analyze messy spreadsheets and workbooks, from answering questions over uploaded files to generating code, summaries, and visualizations. These systems are often evaluated by the correctness of their final downstream answers. However, reliable data analysis also depends on an earlier step: understanding what the dataset contains before solving the requested task. For complex workbooks, this Data Exploration step includes identifying the logical tables behind physical sheets, interpreting column semantics, recovering keys and relationships, and detecting quality issues. In current tools and benchmarks, this step is usually left implicit, creating a gap between downstream task performance and the dataset understanding needed for reliable, human-checkable analysis. Our key contribution is to identify this overlooked gap, make Data Exploration a first-class evaluation target, and show through downstream experiments that stronger Data Exploration support improves task performance. To evaluate dataset understanding directly, we introduce two benchmark settings: a real multi-sheet workbook benchmark based on a Vitamin D study dataset, and an extension of DSBench with schema-fixed Data Exploration artifacts. In both settings, systems are evaluated by the quality of a structured artifact capturing tables, columns, semantic roles, relationships, and profiling signals. Our results show that strong LLMs and data-analysis agents still miss important logical structure even when they read spreadsheet content. Furthermore, explicit Data Exploration support often improves downstream correctness, suggesting it should be treated as a first-class, inspectable stage in LLM data-analysis workflows and a natural human-in-the-loop checkpoint where domain experts can review and correct the artifact before downstream analysis proceeds.
Real-world data analysis is a multi-step process over heterogeneous inputs rather than merely producing a final answer. A practical system should autonomously organize multi-step workflows, execute generated code in a sandboxed and controllable environment, and remain inspectable through visible action traces and intermediate artifacts. Existing LLM-based analysis tools, however, often emphasize isolated subtasks, leaving limited support for complete execution-grounded workflows. We present DA-Studio (Data Analysis Studio), an interactive web-based demo system for end-to-end data analysis that is autonomous, sandboxed, and inspectable. DA-Studio integrates an action-structured analysis backend, a sandboxed execution workspace, and a browser interface for task setup, streamed action traces, artifact preview, code editing and rerunning, and report export. Through iterative action generation, code execution, and feedback incorporation, it incrementally constructs executable analysis steps from raw files and natural-language requests while exposing intermediate results and artifacts throughout the process.