Fail Loudly: An Auditable Runtime for Agentic Data Analysis

Hanxu Yan; Langxuan Deng; Zhengle Wang; Yibo Wang; Chunwei Liu

cs.AISep 26, 2026

Fail Loudly: An Auditable Runtime for Agentic Data Analysis

Authors: Hanxu Yan, Langxuan Deng, Zhengle Wang, Yibo Wang, Chunwei Liu

Organizations: Independent Researcher · Purdue University

Abstract

Large language models (LLMs) have enabled data-science agents to automate multi-step analyses over heterogeneous files. However, incorrect choices regarding data sources, scope, or statistical definitions often lead to silent errors: computations execute successfully but produce plausible yet incorrect outputs that fail to answer the intended question. To mitigate this, we present RADAR, an auditable runtime that makes an agent's analytical choices inspectable and supports their revision through execution feedback. RADAR operates through three core mechanisms. First, an evidence-preserving exploration module retrieves task-relevant content while retaining source locations and observation coverage. Next, the runtime uses typed operators to record the agent's declared inputs, operation arguments, and resulting observations. Finally, runtime validation checks proposed operations against these observations. When a conflict is detected, the runtime rejects the operation or provides diagnostic feedback, allowing the agent to revise its choices before errors propagate. This design enables agents to fail loudly while leaving semantic interpretation to the LLM. On KramaBench, RADAR achieves overall scores of 0.723 with full source retrieval and 0.747 with gold sources supplied, corresponding to relative gains of 35.9% and 28.8% over the strongest baselines. Beyond KramaBench, RADAR achieves relative performance gains of 14.0% on DA-Code and 59.3% on DABStep, demonstrating its applicability across diverse agentic data-analysis workflows.

Figures & tables

Figure 1: Silent errors and runtime interventions by RADAR. (A) Among 580 zero-scored DS-STAR runs across three benchmarks, 405 (69.8%) return normal-looking but incorrect outputs; we categorize their primary error types alongside other failures. (B) A baseline combines two Top-5 rankings without a pairing key, producing a well-formed 25-row table with incorrect row correspondence. On the same task, RADAR rejects an unkeyed merge before execution, prompting the agent to pair entries by rank. (C) A baseline silently loses one year after a successful inner join, while RADAR records the agent’s inspection output that reveals the missing year and guides repair.

Figure 2: Overview of RADAR. Source Map construction supports query-aware source selection and inspection. The agent submits typed operator batches to the runtime, which applies validation and optimization, records runtime observations in shared state, and returns feedback for revision.

Operator type	Main arguments	Function and runtime observations
Discover	query , excluded	Retrieve sources by semantic query or path pattern; return paths.
Bash	command	Inspect files with shell commands; record output and exit status.
Scan	file	Load a table; record provenance, schema, row count, and preview.
UnionScan	files , source_column	Union by column name; record source labels and table metadata.
Probe	table , columns	Inspect columns; report value frequencies, ranges, and null rates.
Query	sql , inputs , output	Run SQL; record result schema, row count, and visible records.

Table 1: Typed operators, their main arguments, and runtime observations.

		Domains							Resource use
System	Models	Archaeology	Astronomy	Biomedical	Environment	Legal	Wildfire	Overall	Cost ($)	Latency (s)
Full Input (source retrieval from the complete data lake)
Human		0.583	0.700	1.000	0.769	0.867	0.662	0.768	–	–
DS-Guru	DeepSeek V4-Flash	0.000	0.083	0.000	0.150	0.200	0.305	0.158	1.27	49
smolagents DR		0.417	0.250	0.556	0.533	0.533	0.746	0.532	6.38	636
AutoGen		0.417	0.250	0.111	0.333	0.500	0.380	0.372	3.16	183

Table 2: End-to-end scores and resource use on KramaBench across backbone models. Overall is the task-count-weighted mean. The Human row is from the KramaBench expert study. Bold marks the best automated system within each input setting and backbone. GPT-5.6 Luna uses high reasoning effort; GPT-4.1-mini uses the 2025-04-14 snapshot.

System	Data Wrangling	ML	EDA	Easy	Medium	Hard	Overall	Completion
DS-Guru	16.5	34.6	16.5	27.5	18.5	17.3	20.1	70.8
smolagents DR	29.2	44.8	39.6	53.8	36.6	28.7	38.5	91.6
AutoGen	31.4	41.8	35.2	46.8	33.7	30.7	35.8	86.6
Data Interpreter	21.0	24.5	32.6	36.8	26.9	25.3	28.6	84.8
DA-Agent	33.9	51.5	44.3	55.4	42.1	36.3	43.6	90.0
DS-STAR	33.6	58.8	46.6	61.4	42.6	42.2	46.4	96.8

Table 3: DA-Code mean scores (%) on 500 tasks. EDA aggregates 79 Data Insights, 73 Data Manipulation, 70 Statistical Analysis, and 78 Visualization tasks, weighted by task count. Completion represents the proportion of tasks for which the model produces results.

Figure 3: Non-delivery and silent errors across systems on each benchmark.

Runtime Enforcement for AI Agents

Runtime enforcement for AI agents checks and constrains actions as they execute, helping prevent unsafe or unauthorized effects.

See all 276 papers

Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis
Apr 27, 2026Zhisong Qiu, Shuofei Qiao, Kewei Xu +4Process Reward Models RL for LLM Agents
Zhejiang University Ant Group Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph Hangzhou, China · Zhejiang University Hangzhou, China · Ant Group Hangzhou, China +2
AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis
Aug 18, 2026Yangtian Liu, Yan Miao, Shuhan Liu +4AI Agent Monitoring Data Analysis Agents
State Key Lab of CAD&CG, Zhejiang University, Hangzhou, Zhejiang, China · Department of Computer Science and Engineering, Yonsei University, Seoul, Republic of Korea · School of Software Technology, Zhejiang University, Ningbo, Zhejiang, China
AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems
Feb 18, 2026Hadar Mulian, Sergey Zeltyn, Ido Levy +3LLM Agent Reliability LLM Agents
IBM Research, Israel

Apr 27, 2026J/K move · Enter open · S save

Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

Zhisong Qiu, Shuofei Qiao, Kewei Xu, Yuqi Zhu+3

Zhejiang University Ant Group Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph Hangzhou, China · Zhejiang University Hangzhou, China · Ant Group Hangzhou, China+2

Process Reward Models RL for LLM Agents Data Analysis Agents

Open paper AskPDF arXiv

Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs) within static domains such as mathematics. However, their potential in dynamic data analysis tasks remains underexplored. In this work, we first present a empirical study revealing that general-domain PRMs struggle to supervise data analysis agents. Specifically, they fail to detect silent errors, logical flaws that yield incorrect results without triggering interpreter exceptions, and erroneously penalize exploratory actions, mistaking necessary trial-and-error exploration for grounding failures. To bridge this gap, we introduce DataPRM, a novel environment-aware generative process reward model that (1) can serve as an active verifier, autonomously interacting with the environment to probe intermediate execution states and uncover silent errors, and (2) employs a reflection-aware ternary reward strategy that distinguishes between correctable grounding errors and irrecoverable mistakes. We design a scalable pipeline to construct over 8K high-quality training instances for DataPRM via diversity-driven trajectory generation and knowledge-augmented step-level annotation. Experimental results demonstrate that DataPRM improves downstream policy LLMs by 7.21% on ScienceAgentBench and 11.28% on DABStep using Best-of-N inference. Notably, with only 4B parameters, DataPRM outperforms strong baselines, and exhibits robust generalizability across diverse Test-Time Scaling strategies. Furthermore, integrating DataPRM into Reinforcement Learning yields substantial gains over outcome-reward baselines, achieving 78.73% on DABench and 64.84% on TableBench, validating the effectiveness of process reward supervision. Code is available at https://github.com/zjunlp/DataMind.

Abstract

Runtime Enforcement for AI Agents

Explore similar work

Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis

AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems