DataWeave: Deploying Human-LLM Analytics for Exploratory Structured Data Analysis
Organizations: Virginia Tech, VA, USA. · The Chronicle of Higher Education, Washington, DC, USA
Abstract
Data journalism, the practice of using data analysis to surface newsworthy stories, depends increasingly on the ability of reporters and investigative journalists to uncover trends, disparities, and accountability narratives. In practice, exploring large structured datasets remains slow and brittle: journalists must navigate hundreds of variables across many datasets over years, understand data coding conventions, and write non-trivial analysis code while hypotheses evolve. Although LLMs are often touted as "ask in English, get SQL/answers," real newsroom workflows expose recurring failures, e.g., schema mismatches and drift, misread domain semantics and units, and silent assumptions. We present DataWeave, a system that addresses these needs by combining conversational interaction, schema grounding, analytical planning, and executable query generation to support exploratory analysis over structured data. Rather than treating LLMs as autonomous answer engines, DataWeave frames them as interactive partners whose outputs can be inspected, corrected, and steered as hypotheses shift. We present a case study with professional journalists using our system to analyze the U.S. Department of Education's Integrated Postsecondary Education Data System (IPEDS), a high-stakes public dataset with substantial domain semantics and frequent schema updates. We also report how deployment experience and iterative refinement shaped the current DataWeave architecture and its analytical workflow. Our findings distill design principles and deployment lessons for trustworthy human-LLM collaboration in structured data analysis.
Figures & tables
| Design Principle | What this means | How this is realized in DataWeave |
|---|---|---|
| Permission Over Forgiveness | Ambiguous analytical intent should be resolved before unsupported assumptions propagate into the analysis. | DataWeave requests clarification when key information such as scope, terminology, or temporal context is ambiguous, while preserving the user’s analytical objective. |
| Executable Queries as Inspectable Analytical Artifacts | Executable queries and intermediate results should remain inspectable rather than being hidden behind natural-language answers. | DataWeave grounds analytical responses in executable SQL and preserves queries, returned results, and provenance as part of the analytical trace. |
| Objective-Conditioned Schema Grounding | LLM reasoning should use compact schema and semantic context relevant to the current analytical objective. | DataWeave retrieves relevant tables, variables, descriptions, and coded-value semantics on demand rather than exposing the complete schema to the model. |
| Error Exposure Over Silent Robustness | Analytical systems should favor explicit error exposure and recoverable failure states over silent corrections that may alter analytical intent. | DataWeave treats execution errors as informative signals rather than exceptional conditions to be hidden. Failures are surfaced directly to the user, and any automated recovery is deliberately bounded and constrained. |
| Guided Agent Delegation | LLMs may coordinate analytical steps dynamically, while user intent and interpretation remain under human control. | DataWeave allows the agent to sequence schema discovery, planning, query formulation, and recovery while using clarification and inspectable traces to preserve user oversight. |
| Metric | Value |
|---|---|
| Scale | |
| Users | 18 |
| Sessions | 298 |
| Analytical steps | 797 |
| Avg. steps/session | 2.67 |
| Session Depth | |
| Metric | Mean | Median |
|---|---|---|
| Columns inferred per step | 8.94 | 6 |
| Columns used per step | 6.03 | 5 |
| Column overlap | 5.38 | 5 |
| Overlap ratio (used covered by inferred) | 0.86 | 1.00 |
| Overlap ratio (inferred utilized) | 0.64 | 0.67 |
| Used-only columns (not inferred) | 0.65 | 0 |
Data Analysis Agents
Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.
See all 106 papers