Pinpointing Pivot Points: The DECIDE Framework for Data Analysis Decision Importances

Zachary T. Rewolinski; Jan Simson; Frauke Kreuter; Christoph Kern; Bin Yu

stat.MLOct 28, 2026

Pinpointing Pivot Points: The DECIDE Framework for Data Analysis Decision Importances

Authors: Zachary T. Rewolinski, Jan Simson, Frauke Kreuter, Christoph Kern, Bin Yu

Organizations: UC Berkeley · LMU Munich · Munich Center for Machine Learning (MCML) · University of Mannheim

Abstract

Human judgment calls made throughout the data science life cycle can change the conclusions drawn from data. Independent analysts often make different choices when given the same data, leading to often unacceptable variations in results and contributing to the ongoing reproducibility crisis across scientific disciplines. The rapid adoption of agentic tools threatens to amplify this crisis. Several approaches have been developed to address this variation by following the same strategy: repeating an analysis under different combinations of analytical decisions and inspecting the distribution of outcomes. Examples include multiverse analysis, specification curve analysis, and vibration of effects. More broadly, the Predictability-Computability-Stability (PCS) framework assesses how documented judgment calls across the data science life cycle affect conclusions, with predictability serving as a reality-check that a model captures signal. Each of these approaches results in a "multiverse" object. While the distribution of outcomes reflects the overall variability of conclusions, estimating the influence of each individual decision requires additional tools. A common heuristic is to train a model to predict the outcome from these decisions and treat its coefficients or feature importances as proxies for each decision's influence. Although this strategy may work in certain cases, it assumes the model is correctly specified and accurately predicts the outcome, which we show is not always the case. In this work, we introduce DECIDE, a model-free framework that quantifies each decision's importance from the actual outcomes across the plausible decision space, avoiding issues of predictability by design. We demonstrate the effectiveness of DECIDE by applying it to real-world multiverse analyses. We further provide what is, to our knowledge, the first formal definition of the multiverse.

Figures & tables

Panel of Figure 1 — Figure 1: A simple multiverse represented as a depth-3 tree, with each leaf node holding the result $f\left(\mathcal{U}\right)$ of one universe. Here, we can imagine $c_{1}$ being data cleaning (e.g., drop or impute missing values), $c_{2}$ being feature selection (e.g., include or exclude sensitive attributes), and $c_{3}$ being model choice (e.g., logistic regression or random forest). Taking $c_{2}$ as the decision of interest , each leaf node color marks one comparable set: the universes which agree on every decision but $c_{2}$ . Decide measures the spread $\Phi$ within each of these sets.

Panel of Figure 2 — Figure 2: Surrogate regression models poorly approximate the multiverse outcomes they are used to explain. Left: observed vs. predicted distributions of universe outcomes. Right: residuals vs. fitted values. Top: on fertility , the predicted distribution has a different shape than the observed outcomes. Bottom: on hurricane , the surrogate misses the long right tail of observed outcomes and the residuals show heteroscedasticity, outliers, and curvature.

Figure 3: Full tree view of the Fertility / Durante multiverse. Its branch structure expands over its five decisions, making explicit the full set of possible universe paths.

Panel of Figure 4 — Figure 4: A multiverse consisting of three binary decisions. The first two levels ( $c_{1}$ and $c_{2}$ ) impact the output accuracies through their interaction. The last level ( $c_{3}$ ) influences the accuracies with its fixed effect.

			fANOVA		Random Forest
	$\textsc{Decide}_{\texttt{total}}$	Reg.	Main	Total	MDI	PFI
$c_{1}$	0.000625	0.000000	0.106792	0.355601	0.294151	0.000544
$c_{2}$	0.000625	0.000000	0.116994	0.354927	0.283771	0.000534
$c_{3}$	0.000400	0.040000	0.479171	0.627509	0.422078	0.000727

Table 1: Comparison of decision importance scores on the multiverse shown in Figure 4 . Only

\textsc{Decide}_{\texttt{total}}

correctly identifies that the interacting decisions (

c_{1},c_{2}

) have a higher importance than the main effect decision (

c_{3}

). The most important feature(s) for each approach are shown in bold.

Decision	Total	Main	Ratio	Total	Main	Reg.
	Decide			fANOVA
Relationship Status	0.089	0.078	88.3%	0.506	0.410	0.297
Fertile	0.087	0.056	64.0%	0.417	0.251	0.244
Menstrual Calculation	0.041	0.011	27.1%	0.193	0.055	0.152
Cycle Length	0.028	0.012	43.7%	0.132	0.050	0.152
Certainty	0.008	0.005	69.6%	0.054	0.022	0.146

Table 2: Importance estimates per decision, by method for the Fertility / Durante multiverse [ 52 , 11 ] .

Panel of Figure 5 — Figure 5: Ablation stability on the Fertility / Durante multiverse. Left: all pairs agree closely on the full multiverse ( $\rho\geq 0.90$ ); after ablation, disagreement is concentrated in pairs including regression as an estimator. Right: $\textsc{Decide}_{\texttt{total}}$ is the most stable estimator overall and most stable or tied for most stable in every ablation, regression is the least.

Panel of Figure 6 — Figure 6: Ablation stability on the Hurricane / Jung multiverse. Left: agreement splits along total vs. main effects: the two total-effect estimators agree almost perfectly with each other, as do the two main-effect estimators, while pairs across the two groups agree less, most visibly on the full multiverse. Right: $\textsc{Decide}_{\texttt{total}}$ and $\text{fANOVA}_{\texttt{total}}$ are the most stable on average; fixing ‘Model’ destabilizes all other estimators.

Decision	Total	Main	Ratio	Total	Main	Reg.
	Decide			fANOVA
Model	2.251	1.004	44.6%	0.540	0.247	2.004
Death Outliers	1.714	0.722	42.1%	0.427	0.175	1.006
Damage Transform	1.035	0.001	0.1%	0.231	0.000	0.059
Damage Outliers	0.964	0.157	16.3%	0.229	0.041	0.336
Covariates	0.771	0.057	7.4%	0.186	0.013	0.383

Table 3: Importance estimates per decision, by method for the Hurricane / Jung multiverse [ 12 , 53 ] .

Table 4: Importance estimates per decision, by method for the three multiverses in Orben & Przybylski [ 59 ] .

			Decide	fANOVA
Multiverse	$N_{\mathcal{M}}$	Reg.	Main + Total	Main	Total
Fertility / Durante	210	11 ms	144 ms	132 ms	345 ms
Hurricane / Jung	1,728	6 ms	220 ms	189 ms	7.3 s
YRBS	372	5 ms	91 ms	119 ms	3.2 s
MTF	40,966	49 ms	766 ms	4.4 s	–
MCS	20,004	84 ms	181 ms	13.9 s	–

Table 5: Median execution time per method and multiverse, over 3 runs.

N_{\mathcal{M}}

denotes the size of the multiverse. Dashes mark methods not evaluated for a given multiverse due to computational constraints. Decide includes several checks regarding data structure in its timing.

Decision	Group	$\textsc{Decide}_{\texttt{total}}$	Rank	$\textsc{Decide}_{\texttt{total}}$	Rank
		$\Lambda=$ Mean, $\Phi=$ Variance		$\Lambda=$ Max, $\Phi=$ Range
$c_{1}$	Routine	0.000900	1	0.060	8
$c_{2}$	Routine	0.000784	2	0.056	9
$c_{3}$	Routine	0.000676	3	0.052	10
$c_{4}$	Routine	0.000576	4	0.048	11
$c_{5}$	Routine	0.000484	5	0.044	12

Table 6: Decide importance scores on the synthetic multiverse. The catastrophic decision

c_{12}

ranks last under average-case instantiation but first under worst-case instantiation.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Figure B1: Visualizations of the three types of sparsity described in Section 3.6 . The decision of interest is denoted in orange . Colored arcs represent comparable sets.

Panel of Figure 14 — Figure B2: Method stability for the three adolescent well-being multiverses of Orben and Przybylski [ 59 ] . Top: inter- and intra-method stability under ablation for YRBS. Bottom: inter-method agreement on the full multiverse for MTF and MCS, which vary only three decisions and are therefore not ablated. YRBS: every estimator stays highly stable; the most difficult ablation is fixing ‘X Variable’, where the main-effect estimators drop. MTF: Decide and fANOVA agree closely ( $0.93$ – $1.00$ ), while regression is only weakly correlated with either ( $0.14$ – $0.49$ ). MCS: methods disagree strongly; regression is almost perfectly anti-correlated with $\textsc{Decide}_{\texttt{main}}$ ( $-0.96$ ) and $\text{fANOVA}_{\texttt{main}}$ ( $-0.92$ ). Note that all three multiverses vary at most four judgment calls, so the individual cells are coarse (see the caveat in Section 5.4 ).

Decision	Description	No. of Opts.
Fertile	Assessment of fertility (high/low) using different windows as cutoffs	5
Menstrual Calculation	How the next menstrual onset is estimated	3
Relationship Status	How relationship status is coded, in particular the option “dating or involved with only one partner”	3
Cycle Length	Exclusion criteria for women based on cycle length	3
Certainty	Exclusion criteria for women based on certainty of start days of previous menstrual onsets	2

Appendix

Table C1: Decisions in the Fertility / Durante multiverse. These five decisions lead to

n_{fert}=210

universes (less than the theoretical

N_{fert}=270

universes due to constraints).

Decision	Description	No. of Opts.
Death Outliers	Whether to exclude the storms with the most deaths	3
Damage Outliers	Whether to exclude the storms with the most property damage	4
Femininity	Whether the ‘femininity’ of the hurricane name is encoded as continuous or binary	2
Model	Which model class to use.	2
Damage Transform	Whether to log-transform damage (i.e., measure with dollars or log-dollars)	2
Predictor Specification	Whether the effect of ‘femininity’ interacted with other features (and if so, which ones)	6

Appendix

Table C2: Decisions in the Hurricane / Jung multiverse. These seven decisions lead to

N_{hurr}=1{,}728

universes.

Decision	Description	No. of Opts.
X Variable	Operationalization of technology use	3
Y Variable	Operationalization of well-being	31
Controls	Which covariates to include	2
Combination	How well-being measures are combined	2

Appendix

Table C3: Decisions in the YRBS multiverse. These four decisions lead to

N_{yrbs}=372

universes.

Decision	Description	No. of Opts.
X Variable	Operationalization of technology use	13
Y Variable	Operationalization of well-being	4,095
Controls	Which covariates to include	2

Appendix

Table C4: Decisions in the MTF multiverse. These three decisions lead to

N_{MTF}=40{,}966

universes.

Decision	Description	No. of Opts.
X Variable	Operationalization of technology use	6
Y Variable	Operationalization of well-being	1,667
Controls	Which covariates to include	2
Respondent	Whether to use well-being measures by cohort members or caregivers	2

Appendix

Table C5: Decisions in the MCS multiverse. These four decisions lead to

N_{MCS}=20{,}004

universes. The authors selected and analyzed a subset of

n_{MCS}=20{,}004

universes out of a theoretical total of

N_{MCS}=603{,}979{,}752

defensible universes (or

>2.5

trillion if any combination of covariates were to be explored). ‘Respondent’ is determined completely by ‘Y Variable’, and is thus merged in accordance with the strategy described in Section 3.6 .

Sensitivity Analysis

Sensitivity analysis studies how changes in inputs, assumptions, or model parameters affect outputs, predictions, and decisions.

See all 40 papers

Sanity Checks for Agentic Data Science
Apr 13, 2026Zachary T. Rewolinski, Austin V. Zane, Hao Huang +4AI Agent Reliability Data Analysis Agents
Dept. of Statistics, UC Berkeley · Dept. of EECS, UC Berkeley · Microsoft Research +1
Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making
Oct 13, 2026Jing Liu, Yongxing Qi, Ping Yao +1LLM Decision-Making Structured Output Generation
Hangzhou Innovation Institute, Beihang University, Hangzhou, China · Unit, Beijing, China
SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores
Sep 5, 2026Bowen Liu, Shuo Nie, Bodong Du +1Scientific Agent Benchmarks LLM Agent Evaluation
HKUST · Alibaba Group

Apr 13, 2026J/K move · Enter open · S save

Sanity Checks for Agentic Data Science

Zachary T. Rewolinski, Austin V. Zane, Hao Huang, Chandan Singh+3

Dept. of Statistics, UC Berkeley · Dept. of EECS, UC Berkeley · Microsoft Research+1

AI Agent Reliability Data Analysis Agents Language Model Calibration

Open paper AskPDF arXiv

Agentic data science (ADS) pipelines have grown rapidly in both capability and adoption, with systems such as OpenAI Codex now able to directly analyze datasets and produce answers to statistical questions. However, these systems can reach falsely optimistic conclusions that are difficult for users to detect. To address this, we propose a pair of lightweight sanity checks grounded in the Predictability-Computability-Stability (PCS) framework for veridical data science. These checks use reasonable perturbations to screen whether an agent can reliably distinguish signal from noise, acting as a falsifiability constraint that can expose affirmative conclusions as unsupported. Together, the two checks characterize the trustworthiness of an ADS output, e.g. whether it has found stable signal, is responding to noise, or is sensitive to incidental aspects of the input. We validate the approach on synthetic data with controlled signal-to-noise ratios, confirming that the sanity checks track ground-truth signal strength. We then demonstrate the checks on 11 real-world datasets using OpenAI Codex, characterizing the trustworthiness of each conclusion and finding that in 6 of the datasets an affirmative conclusion is not well-supported, even though a single ADS run may support one. We further analyze failure modes of ADS systems and find that ADS self-reported confidence is poorly calibrated to the empirical stability of its conclusions.

Abstract

Sensitivity Analysis

Explore similar work

Sanity Checks for Agentic Data Science

Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making

SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores