Pinpointing Pivot Points: The DECIDE Framework for Data Analysis Decision Importances

Zachary T. Rewolinski; Jan Simson; Frauke Kreuter; Christoph Kern; Bin Yu

stat.MLOct 28, 2026

Pinpointing Pivot Points: The DECIDE Framework for Data Analysis Decision Importances

Authors: Zachary T. Rewolinski, Jan Simson, Frauke Kreuter, Christoph Kern, Bin Yu

Organizations: UC Berkeley · LMU Munich · Munich Center for Machine Learning (MCML) · University of Mannheim

Abstract

Human judgment calls made throughout the data science life cycle can change the conclusions drawn from data. Independent analysts often make different choices when given the same data, leading to often unacceptable variations in results and contributing to the ongoing reproducibility crisis across scientific disciplines. The rapid adoption of agentic tools threatens to amplify this crisis. Several approaches have been developed to address this variation by following the same strategy: repeating an analysis under different combinations of analytical decisions and inspecting the distribution of outcomes. Examples include multiverse analysis, specification curve analysis, and vibration of effects. More broadly, the Predictability-Computability-Stability (PCS) framework assesses how documented judgment calls across the data science life cycle affect conclusions, with predictability serving as a reality-check that a model captures signal. Each of these approaches results in a "multiverse" object. While the distribution of outcomes reflects the overall variability of conclusions, estimating the influence of each individual decision requires additional tools. A common heuristic is to train a model to predict the outcome from these decisions and treat its coefficients or feature importances as proxies for each decision's influence. Although this strategy may work in certain cases, it assumes the model is correctly specified and accurately predicts the outcome, which we show is not always the case. In this work, we introduce DECIDE, a model-free framework that quantifies each decision's importance from the actual outcomes across the plausible decision space, avoiding issues of predictability by design. We demonstrate the effectiveness of DECIDE by applying it to real-world multiverse analyses. We further provide what is, to our knowledge, the first formal definition of the multiverse.

Figures & tables

Panel of Figure 1 — Figure 1: A simple multiverse represented as a depth-3 tree, with each leaf node holding the result $f\left(\mathcal{U}\right)$ of one universe. Here, we can imagine $c_{1}$ being data cleaning (e.g., drop or impute missing values), $c_{2}$ being feature selection (e.g., include or exclude sensitive attributes), and $c_{3}$ being model choice (e.g., logistic regression or random forest). Taking $c_{2}$ as the decision of interest , each leaf node color marks one comparable set: the universes which agree on every decision but $c_{2}$ . Decide measures the spread $\Phi$ within each of these sets.

Panel of Figure 2 — Figure 2: Surrogate regression models poorly approximate the multiverse outcomes they are used to explain. Left: observed vs. predicted distributions of universe outcomes. Right: residuals vs. fitted values. Top: on fertility , the predicted distribution has a different shape than the observed outcomes. Bottom: on hurricane , the surrogate misses the long right tail of observed outcomes and the residuals show heteroscedasticity, outliers, and curvature.

Figure 3: Full tree view of the Fertility / Durante multiverse. Its branch structure expands over its five decisions, making explicit the full set of possible universe paths.

Panel of Figure 4 — Figure 4: A multiverse consisting of three binary decisions. The first two levels ( $c_{1}$ and $c_{2}$ ) impact the output accuracies through their interaction. The last level ( $c_{3}$ ) influences the accuracies with its fixed effect.

			fANOVA		Random Forest
	$\textsc{Decide}_{\texttt{total}}$	Reg.	Main	Total	MDI	PFI
$c_{1}$	0.000625	0.000000	0.106792	0.355601	0.294151	0.000544
$c_{2}$	0.000625	0.000000	0.116994	0.354927	0.283771	0.000534
$c_{3}$	0.000400	0.040000	0.479171	0.627509	0.422078	0.000727

Table 1: Comparison of decision importance scores on the multiverse shown in Figure 4 . Only

\textsc{Decide}_{\texttt{total}}

correctly identifies that the interacting decisions (

c_{1},c_{2}

) have a higher importance than the main effect decision (

c_{3}

). The most important feature(s) for each approach are shown in bold.

Decision	Total	Main	Ratio	Total	Main	Reg.
	Decide			fANOVA
Relationship Status	0.089	0.078	88.3%	0.506	0.410	0.297
Fertile	0.087	0.056	64.0%	0.417	0.251	0.244
Menstrual Calculation	0.041	0.011	27.1%	0.193	0.055	0.152
Cycle Length	0.028	0.012	43.7%	0.132	0.050	0.152
Certainty	0.008	0.005	69.6%	0.054	0.022	0.146

Table 2: Importance estimates per decision, by method for the Fertility / Durante multiverse [ 52 , 11 ] .

Panel of Figure 5 — Figure 5: Ablation stability on the Fertility / Durante multiverse. Left: all pairs agree closely on the full multiverse ( $\rho\geq 0.90$ ); after ablation, disagreement is concentrated in pairs including regression as an estimator. Right: $\textsc{Decide}_{\texttt{total}}$ is the most stable estimator overall and most stable or tied for most stable in every ablation, regression is the least.

Panel of Figure 6 — Figure 6: Ablation stability on the Hurricane / Jung multiverse. Left: agreement splits along total vs. main effects: the two total-effect estimators agree almost perfectly with each other, as do the two main-effect estimators, while pairs across the two groups agree less, most visibly on the full multiverse. Right: $\textsc{Decide}_{\texttt{total}}$ and $\text{fANOVA}_{\texttt{total}}$ are the most stable on average; fixing ‘Model’ destabilizes all other estimators.

Decision	Total	Main	Ratio	Total	Main	Reg.
	Decide			fANOVA
Model	2.251	1.004	44.6%	0.540	0.247	2.004
Death Outliers	1.714	0.722	42.1%	0.427	0.175	1.006
Damage Transform	1.035	0.001	0.1%	0.231	0.000	0.059
Damage Outliers	0.964	0.157	16.3%	0.229	0.041	0.336
Covariates	0.771	0.057	7.4%	0.186	0.013	0.383

Table 3: Importance estimates per decision, by method for the Hurricane / Jung multiverse [ 12 , 53 ] .

Table 4: Importance estimates per decision, by method for the three multiverses in Orben & Przybylski [ 59 ] .

			Decide	fANOVA
Multiverse	$N_{\mathcal{M}}$	Reg.	Main + Total	Main	Total
Fertility / Durante	210	11 ms	144 ms	132 ms	345 ms
Hurricane / Jung	1,728	6 ms	220 ms	189 ms	7.3 s
YRBS	372	5 ms	91 ms	119 ms	3.2 s
MTF	40,966	49 ms	766 ms	4.4 s	–
MCS	20,004	84 ms	181 ms	13.9 s	–

Table 5: Median execution time per method and multiverse, over 3 runs.

N_{\mathcal{M}}

denotes the size of the multiverse. Dashes mark methods not evaluated for a given multiverse due to computational constraints. Decide includes several checks regarding data structure in its timing.

Decision	Group	$\textsc{Decide}_{\texttt{total}}$	Rank	$\textsc{Decide}_{\texttt{total}}$	Rank
		$\Lambda=$ Mean, $\Phi=$ Variance		$\Lambda=$ Max, $\Phi=$ Range
$c_{1}$	Routine	0.000900	1	0.060	8
$c_{2}$	Routine	0.000784	2	0.056	9
$c_{3}$	Routine	0.000676	3	0.052	10
$c_{4}$	Routine	0.000576	4	0.048	11
$c_{5}$	Routine	0.000484	5	0.044	12

Table 6: Decide importance scores on the synthetic multiverse. The catastrophic decision

c_{12}

ranks last under average-case instantiation but first under worst-case instantiation.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Figure B1: Visualizations of the three types of sparsity described in Section 3.6 . The decision of interest is denoted in orange . Colored arcs represent comparable sets.

Panel of Figure 14 — Figure B2: Method stability for the three adolescent well-being multiverses of Orben and Przybylski [ 59 ] . Top: inter- and intra-method stability under ablation for YRBS. Bottom: inter-method agreement on the full multiverse for MTF and MCS, which vary only three decisions and are therefore not ablated. YRBS: every estimator stays highly stable; the most difficult ablation is fixing ‘X Variable’, where the main-effect estimators drop. MTF: Decide and fANOVA agree closely ( $0.93$ – $1.00$ ), while regression is only weakly correlated with either ( $0.14$ – $0.49$ ). MCS: methods disagree strongly; regression is almost perfectly anti-correlated with $\textsc{Decide}_{\texttt{main}}$ ( $-0.96$ ) and $\text{fANOVA}_{\texttt{main}}$ ( $-0.92$ ). Note that all three multiverses vary at most four judgment calls, so the individual cells are coarse (see the caveat in Section 5.4 ).

Decision	Description	No. of Opts.
Fertile	Assessment of fertility (high/low) using different windows as cutoffs	5
Menstrual Calculation	How the next menstrual onset is estimated	3
Relationship Status	How relationship status is coded, in particular the option “dating or involved with only one partner”	3
Cycle Length	Exclusion criteria for women based on cycle length	3
Certainty	Exclusion criteria for women based on certainty of start days of previous menstrual onsets	2

Appendix

Table C1: Decisions in the Fertility / Durante multiverse. These five decisions lead to

n_{fert}=210

universes (less than the theoretical

N_{fert}=270

universes due to constraints).

Decision	Description	No. of Opts.
Death Outliers	Whether to exclude the storms with the most deaths	3
Damage Outliers	Whether to exclude the storms with the most property damage	4
Femininity	Whether the ‘femininity’ of the hurricane name is encoded as continuous or binary	2
Model	Which model class to use.	2
Damage Transform	Whether to log-transform damage (i.e., measure with dollars or log-dollars)	2
Predictor Specification	Whether the effect of ‘femininity’ interacted with other features (and if so, which ones)	6

Appendix

Table C2: Decisions in the Hurricane / Jung multiverse. These seven decisions lead to

N_{hurr}=1{,}728

universes.

Decision	Description	No. of Opts.
X Variable	Operationalization of technology use	3
Y Variable	Operationalization of well-being	31
Controls	Which covariates to include	2
Combination	How well-being measures are combined	2

Appendix

Table C3: Decisions in the YRBS multiverse. These four decisions lead to

N_{yrbs}=372

universes.

Decision	Description	No. of Opts.
X Variable	Operationalization of technology use	13
Y Variable	Operationalization of well-being	4,095
Controls	Which covariates to include	2

Appendix

Table C4: Decisions in the MTF multiverse. These three decisions lead to

N_{MTF}=40{,}966

universes.

Decision	Description	No. of Opts.
X Variable	Operationalization of technology use	6
Y Variable	Operationalization of well-being	1,667
Controls	Which covariates to include	2
Respondent	Whether to use well-being measures by cohort members or caregivers	2

Appendix

Table C5: Decisions in the MCS multiverse. These four decisions lead to

N_{MCS}=20{,}004

universes. The authors selected and analyzed a subset of

n_{MCS}=20{,}004

universes out of a theoretical total of

N_{MCS}=603{,}979{,}752

defensible universes (or

>2.5

trillion if any combination of covariates were to be explored). ‘Respondent’ is determined completely by ‘Y Variable’, and is thus merged in accordance with the strategy described in Section 3.6 .

Sensitivity Analysis

Sensitivity analysis studies how changes in inputs, assumptions, or model parameters affect outputs, predictions, and decisions.

See all 40 papers

Apr 13, 2026

Sanity Checks for Agentic Data Science

Agentic data science (ADS) pipelines have grown rapidly in both capability and adoption, with systems such as OpenAI Codex now able to directly analyze datasets and produce answers to statistical questions. However, these systems can reach falsely optimistic conclusions that are difficult for users to detect. To address this, we propose a pair of lightweight sanity checks grounded in the Predictability-Computability-Stability (PCS) framework for veridical data science. These checks use reasonable perturbations to screen whether an agent can reliably distinguish signal from noise, acting as a falsifiability constraint that can expose affirmative conclusions as unsupported. Together, the two checks characterize the trustworthiness of an ADS output, e.g. whether it has found stable signal, is responding to noise, or is sensitive to incidental aspects of the input. We validate the approach on synthetic data with controlled signal-to-noise ratios, confirming that the sanity checks track ground-truth signal strength. We then demonstrate the checks on 11 real-world datasets using OpenAI Codex, characterizing the trustworthiness of each conclusion and finding that in 6 of the datasets an affirmative conclusion is not well-supported, even though a single ADS run may support one. We further analyze failure modes of ADS systems and find that ADS self-reported confidence is poorly calibrated to the empirical stability of its conclusions.

Zachary T. Rewolinski, Austin V. Zane, Hao Huang +4

Dept. of Statistics, UC Berkeley · Dept. of EECS, UC Berkeley · Microsoft Research +1

AI Agent Reliability Data Analysis Agents Language Model Calibration

Ask PDF

Oct 13, 2026

Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making

What binds large language models on auditable, hypothesis-driven decision tasks, where every candidate can be checked against the evidence? On the public TCMEval-SDT benchmark for Traditional Chinese Medicine syndrome differentiation (200 cases), the binding constraint is output discipline and reasoning structure, not model capability alone: two years of model scaling move zero-shot scores by only -0.03 to +0.08, while output-discipline failures (non-terminating reasoning, format non-compliance, large run-to-run spread) persist even in frontier models. We test this with a model-agnostic agent framework formalizing decision making as a five-stage closed loop, enforced by a candidate x evidence matrix falsification protocol: every candidate is judged against every evidence item, at least half of the candidates must be falsified before a conclusion is licensed, and survivors are emitted in a machine-verifiable single-line format. Across five base models, the effect is sign-conditional on zero-shot output discipline: it rescues models whose discipline fails (DeepSeek-V4-Flash: +0.267/+0.271, entirely termination rescue), reduces a high-variance model's run-to-run standard deviation from 0.218 to 0.007 (Kimi K3), and significantly costs already-disciplined models (-0.087 to -0.120 on budget-matched Qwen3-4B/32B). Ablations attribute the cost to the candidate-bounded format layer and show the effect is independent of injected knowledge content (protocol-knowledge decoupling). The protocol also transfers to misinformation-robust mathematical reasoning, detecting injected false premises at 89.8% recall with a 6.8% false-positive rate (MCC 0.54; a flag-everything baseline scores 0) and restoring accuracy to the no-misinformation ceiling. Where zero-shot output discipline is weak, the reasoning protocol sets the floor; base-model capability always sets the ceiling.

Jing Liu, Yongxing Qi, Ping Yao +1

Hangzhou Innovation Institute, Beihang University, Hangzhou, China · Unit, Beijing, China

LLM Decision-Making Structured Output Generation Language Model Robustness LLM Auditing Clinical Reasoning

Ask PDF

Sep 5, 2026

SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores

Scientific coding agents produce interdependent code, results, figures, and claims, yet evaluating final outputs alone does not establish whether their conclusions are scientifically supported. We formulate evidence-grounded multimodal scientific analysis, requiring agents to produce executable analyses and claims supported by results and visualizations from the same run. We introduce SciRIGOR, an evaluation framework and benchmark comprising 100 cases from scientific articles across six domains and 17 subfields. The framework reconstructs typed evidence graphs, separates artifact fidelity from relational validity, and scores complete claim-support paths while localizing the earliest unsupported relation. Source-grounded alternative paths accommodate scientifically equivalent analyses and visualizations. We evaluate 11 agent/model configurations. On full-benchmark runs, claims agree with faithful and unfaithful results at nearly identical rates (91.8% versus 91.0%). Yet no system exceeds 62.6% on the soft evidence-chain score or 18.0% strict whole-chain success. These findings show that internal coherence does not establish scientific correctness: evaluation must verify support along the complete data-to-claim path.

Bowen Liu, Shuo Nie, Bodong Du +1

HKUST · Alibaba Group

Scientific Agent Benchmarks LLM Agent Evaluation Evidence-Grounded Reasoning Scientific Claim Verification Scientific Reproducibility

Ask PDF

Abstract

Sensitivity Analysis

Explore similar work

Sanity Checks for Agentic Data Science

Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making

SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores