Authors: Zachary T. Rewolinski, Austin V. Zane, Hao Huang, Chandan Singh, Chenglong Wang, Jianfeng Gao, Bin Yu
Organizations: Dept. of Statistics, UC Berkeley · Dept. of EECS, UC Berkeley · Microsoft Research · Dept. of Statistics, Dept. of EECS, Center for Computational Biology, UC Berkeley
Agentic data science (ADS) pipelines have grown rapidly in both capability and adoption, with systems such as OpenAI Codex now able to directly analyze datasets and produce answers to statistical questions. However, these systems can reach falsely optimistic conclusions that are difficult for users to detect. To address this, we propose a pair of lightweight sanity checks grounded in the Predictability-Computability-Stability (PCS) framework for veridical data science. These checks use reasonable perturbations to screen whether an agent can reliably distinguish signal from noise, acting as a falsifiability constraint that can expose affirmative conclusions as unsupported. Together, the two checks characterize the trustworthiness of an ADS output, e.g. whether it has found stable signal, is responding to noise, or is sensitive to incidental aspects of the input. We validate the approach on synthetic data with controlled signal-to-noise ratios, confirming that the sanity checks track ground-truth signal strength. We then demonstrate the checks on 11 real-world datasets using OpenAI Codex, characterizing the trustworthiness of each conclusion and finding that in 6 of the datasets an affirmative conclusion is not well-supported, even though a single ADS run may support one. We further analyze failure modes of ADS systems and find that ADS self-reported confidence is poorly calibrated to the empirical stability of its conclusions.
Figures & tables
Figure 1: Pipeline for computing the PCS sanity checks. The user supplies a dataset and question. The pipeline then generates perturbed variants of these under both null-defining and PCS perturbations. The coding agent analyzes each independently, producing null and alternative response distributions. A bootstrap test & overlap coefficient then inform whether the agent found a data-grounded, stable positive conclusion.
Perturbation
Type
Description
Failure Mode Tested
Shuffle Feature Values
Null-Defining
Shuffles each column independently to break associations in the data.
Incorrect data analysis conclusions.
Add Non-Signal Features
PCS
Appends features irrelevant to & uncorrelated with the task of interest.
Distraction by irrelevant covariates.
Anonymize Feature Names
PCS
Replaces all column names with generic identifiers (e.g., feature1 , feature2 ).
Reliance on semantic variable names from pre-training data.
Shuffle Feature Names
PCS
Applies a random permutation of existing column names while leaving data values and feature descriptions in metadata unchanged.
Robustness to misleading variable names.
Positive Leading Statement
PCS
Prepends the question with a strong prior belief that the answer is “Yes.”
Sycophancy toward user-stated beliefs.
Negative Leading Statement
PCS
Prepends the question with a strong prior belief that the answer is “No.”
Sycophancy toward user-stated beliefs.
Table 1: Perturbations used in the PCS sanity checks. The null-defining distribution breaks associations between the features and the outcome. The PCS perturbations should not affect an experienced data scientist or agentic data science system, although some perturbations (e.g. positive leading statement ) may mislead human data scientists due to confirmation bias.
Overlap check passed ✓
Overlap check failed ✗
Yes check passed ✓
The positive conclusion is stable.
The positive conclusion may not be grounded in the data.
Yes check failed ✗
There is some positive signal, but not enough for a “Yes”.
No evidence supporting a positive conclusion.
Table 2: Interpretations of the four PCS sanity-check regimes.
Figure 2: The PCS sanity checks behave as expected. As more signal is introduced to the data, the agent’s responses become more positive, leading to less similarity with the null.
Figure 3: Sanity check results on BLADE datasets. (a) KDEs of the null and alternative response distributions for each dataset. (b) Each dataset’s OVL and alternative response mean, colored by category (Table 2 ). Vertical and horizontal dashed lines mark the OVL decision boundary and the scale midpoint, respectively. (c) Agreement between subsampled and full-sample classifications as a function of the number of alternative runs (Appendix A.5 ).
Figure 4: Multi-faceted analysis of agentic instability.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Blocked
Unblocked
Δ
affairs
0.045
0.040
− 0.005
amtl
0.140
0.122
− 0.018
boxes
0.041
0.038
− 0.003
caschools
0.057
0.050
− 0.007
crofoot
0.037
0.021
− 0.016
hurricane
0.040
0.032
− 0.008
Appendix
Table A1: Empirical rejection rates ( α^ ) at α=0.05 for the bootstrap mean test. R=1,000 replicates per dataset, B=10,000 bootstrap resamples per test. Δ is unblocked minus blocked.
Figure A1: QQ plots comparing observed p -value quantiles to theoretical Uniform(0,1) quantiles for the blocked (left) and unblocked (right) bootstrap tests, pooled across all 11 datasets and R=1,000 replicates. The blocked test tracks the diagonal closely. The unblocked test’s curve falls slightly above the diagonal at low quantiles, reflecting its mild conservatism.
Dataset Name
Question
Source Paper(s)
affairs
Does having children decrease (if at all) the engagement in extramarital affairs?
Fair (1978) ; Long and Freese (2006) ; Kleiber and Zeileis (2008)
amtl
Do modern humans ( Homo sapiens ) have higher frequencies of antemortem tooth loss (AMTL) compared to non-human primate genera ( Pan, Pongo, Papio ), after accounting for the effects of age, sex, and tooth class?
Gilmore (2013) ; Konigsberg and Frankenberg (2013) ; McElreath (2018)
boxes
Do children’s reliance on social information and preference for majority cues vary across cultures and developmental stages?
Van Leeuwen et al. (2018) ; McElreath (2018)
caschools
Is a lower student-teacher ratio associated with higher academic performance?
Kleiber and Zeileis (2008) ; Stock and Watson (2020)
crofoot
Do relative group size and contest location influence the probability of a capuchin monkey group winning an intergroup contest?
Crofoot et al. (2008) ; McElreath (2018)
fish
How many fish on average do visitors takes per hour, when fishing?
McElreath (2018)
Appendix
Table A2: Summary of BLADE datasets and their corresponding questions.
Figure A2: We observe that datasets tend to move out of the “Failed both checks” quadrant as PVE increases, demonstrating that the two agentic stability checks corroborate the signal present in the data.
Figure A3: Empirical null and alternative distributions, with the alternative distribution being produced on a PVE-controlled dataset. Note the rightward shift of the alternative distribution as PVE increases.
Dataset
Null Mean (SD)
Alt Mean (SD)
p -Value
OVL
Checks Passed
teachingratings
7.18 (5.72)
69.94 (3.54)
0.0001
0.000
Both
amtl
26.50 (16.51)
93.85 (6.56)
0.0001
0.019
Both
panda_nuts
30.38 (10.92)
70.04 (6.70)
0.0001
0.069
Both
soccer
23.20 (5.44)
64.25 (9.82)
0.0001
0.052
Both
mortgage
29.83 (5.78)
53.46 (16.44)
0.0228
0.102
Both
caschools
17.26 (11.14)
55.81 (19.89)
0.0016
0.280
Yes Check
Appendix
Table A3: Results of the sanity check on eleven BLADE datasets.
Figure A4: Classification agreement vs. sample size under random subsampling (both distributions reduced). Most datasets stabilize within 5–10 runs. caschools and mortgage require substantially more.
Figure A5: Classification agreement vs. alternative sample size (full null retained). The pattern is similar to Fig. A4 , confirming that the alternative distribution is the primary driver of instability at small sample sizes.
Figure A6: Component-level agreement under random subsampling. Left: Bootstrap test (significant or not). Right: Overlap coefficient (low or high).
Figure A7: Component-level agreement under alt-only subsampling. The decomposition mirrors Fig. A6 , with the same bottleneck checks driving instability for the borderline datasets.
Figure A8: Empirical null and alternative distributions, with the “null” distribution being produced on datasets with a very small signal-to-noise ratio.
Figure A9: Within-perturbation volatility. Each cell shows the mean ∣delta∣ across all within-perturbation run pairs for a given perturbation and pipeline step, averaged over ten datasets. High values indicate that the perturbation makes the pipeline’s output unpredictable relative to the unperturbed reference.
Figure A10: Baseline-vs-perturbation divergence. Each cell shows the mean ∣delta∣ for comparisons between an unperturbed run and a perturbed run. High values indicate that perturbed outputs differ from unperturbed outputs by more than unperturbed outputs differ from each other.
Figure A11: The three independent raters tend to agree on the scores for each human-performed data analysis. The raters were asked to put the conclusion of each team’s analysis on the same Likert scale that the agent was prompted with.
Figure A12: Comparison of individual rater values shows general agreement.
Large language models have significantly lowered the barrier to writing code. Naturally, this has also led to the development of data science agents that promise automating the design and execution of data science from task formulation to model development. Existing evaluations of agentic data science (agentic DS) assess downstream predictive performance and code correctness, providing preliminary insight in agentic DS capabilities. We argue that these evaluations overlook the fundamental principles of data science: methodological data science- and domain knowledge. As such, only assessing downstream performance masks methodological failures such as data leakage and inappropriate validation, limiting the development of robust agentic DS systems. In this paper, we evaluate agentic DS beyond accuracy from three different angles: 1) a large-scale analysis of failure modes in common agentic DS frameworks and models across 220 data science tasks, 2) a reusable and extensible test suite with curated data science tasks that test for six specific failures in a controlled manner, and 3) a qualitative assessment of agentic DS pipelines against expert solutions on Kaggle competitions. We find that the majority of agentic solutions contain at least one failure mode despite the complexity of agentic harness or model capacity. Moreover, we identify a positive correlation between the occurrence of data leakage failures and predictive performance, confirming the necessity to evaluate agents beyond accuracy. Our qualitative analysis shows that DS agents generally lack problem solving creativity and domain knowledge, compared to competition-winning data scientists. We believe that principled failure mode analysis, as presented here, should be integral part of agentic DS evaluations, and that the insights we surface will be key in developing trustworthy agentic DS pipelines.
Xue Li, Hongqian Xia, Daniel Gomm +3
Centrum Wiskunde & Informatica, Amsterdam, Netherlands
Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26 to 39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. After freezing the scorer, we compare its judgments with human annotations on 200 trajectories, finding 96% task-success agreement. Human judgments support retry gains over Base and confirm adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.
We present a framework for evaluating data-science agents on live, continuously updated data using executable ground truth and format-agnostic factoid scoring. Consider this example query: "what were last week's audience sizes"---the reference answer changes as the underlying data changes, so static references become outdated and standard LLM-as-a-judge pipelines cannot verify responses against a fixed ground truth. Our central contribution, ground-truth-as-code, encodes each expected answer as an executable reference function that recomputes the answer directly from live data at evaluation time, ensuring the reference remains consistent with the system it describes. We combine this with a factoid-level, format-agnostic judge that decomposes both the agent's response and the computed ground truth into atomic claims and scores precision, recall, and accuracy over them, irrespective of the response format (prose, list, table, HTML, etc.). The approach is applicable to agents whose expected outputs can be expressed as executable data computations. We validate the framework through a human--LLM agreement study on an internally developed machine learning skill deployed in production, using a synthetic database constructed to reproduce production schemas and entity relationships. Relative to a natural-language ground-truth baseline, our method achieves a 29% improvement in the Matthews Correlation Coefficient (MCC)---a class-balanced measure of agreement between expert annotators and LLM-as-a-judge predictions---and a 16% reduction in token consumption per test case, while a self-directed baseline lacking explicit ground truth is anti-correlated with human judgment. Agents that perform multi-source data integration and computation over non-stationary data are routinely deployed in industry; we propose ground-truth-as-code as a practical methodology for their evaluation.